LoRA

LoRA (Low-Rank Adaptation) freezes the pre-trained weights of a base model and injects trainable low-rank update matrices into the layers of the Transformer architecture. Training targets those small matrices, not the full parameter set. On GPT-3 175B the original paper measured trainable parameters falling by roughly a factor of 10,000 and the GPU memory requirement by a factor of three, with quality on par with full fine-tuning and no extra inference latency.

Since the base weights are never modified, one base model can carry several small, portable adapters for different tasks or styles. An adapter is a fraction of the size of the base, which makes it easy to store and share. In PEFT a LoraConfig describes the adapter and get_peft_model() wraps the base in a trainable PeftModel.

Rank and alpha

The number of trainable parameters is set mainly by the rank r and the shape of the original weight matrix: a lower rank means smaller update matrices. lora_alpha is a scaling factor, and the effective multiplier is alpha divided by the rank, not alpha alone; the rank-stabilised variant divides by the square root of r instead and is enabled with use_rslora.

target_modules selects which modules receive an adapter; Transformer models usually take the attention blocks. The diffusers example script targets the UNet modules to_q, to_k, to_v and to_out.0 and sets lora_alpha equal to the rank. PEFT starts one matrix with Kaiming-uniform values and the other at zero, so the adapter begins as an identity transform.

Loading and merging

During training the two matrices live separately. Loaded separately, base and adapter are both resident, which is flexible but costs memory and can add latency. Merging folds the adapter weights into the base: merge_and_unload() returns a standalone model with no adapter layers. To undo it later, merge_adapter() and unmerge_adapter() keep the PeftModel, and add_weighted_adapter() combines several LoRAs under a weighting scheme.

On the diffusers side, load_lora_weights() loads the adapter, set_adapters() activates it and sets a scale, and fuse_lora() with unload_lora_weights() fuses the weights before torch.compile. Hotswapping replaces an already loaded adapter in place, but it is unsupported for LoRAs that target the text encoder.

Where it is used

The original paper develops the method for language models, while the PEFT documentation states that it applies to any dense layer, which is why it works for diffusion models too; the diffusers training scripts use LoRA for DreamBooth, SDXL and Wuerstchen. In practice that means teaching a style, a character or an object next to text-to-image generation. We run local models under LM Studio with Llama class bases, and Forge runs the image base the adapters load into.

What to watch

• An adapter is tied to the base architecture and to the configured target_modules, so one trained for another base cannot be dropped in unchanged, and a rank or alpha mismatch can silently degrade quality.
• Quality depends on the training data: the diffusers dataset guide notes that the image and caption pairs and the accuracy of the captions determine the result.
• The licence of the base model extends to the adapter weights. The FLUX.1 [dev] licence counts LoRA among the components of the model and applies its restrictions to any derivative created from it.

Further reading

• LoRA: Low-Rank Adaptation of Large Language Models (arXiv)
• PEFT: LoRA conceptual guide
• Diffusers: LoRA training

At CyberElectro LoRA is how our own style and language adapters are made: the base models stay shared while task-specific knowledge lives in small, version-controlled adapter files.