cmedina/agenx-lora-training
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add prepare_model_for_kbit_training and update LoRA r=64 alpha=128

Browse Source
This commit is contained in:
Christian Medina
2026-07-01 07:48:44 -04:00
parent 357c9fa781
commit 2fa5173c56
2 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
training/configs/ornith-35b-lora.yaml
View File

@@ -9,8 +9,8 @@ tokenizer_type: LlamaTokenizer
# No need for BitsAndBytes configuration # No need for BitsAndBytes configuration
# LoRA Configuration # LoRA Configuration
lora_r: 16 lora_r: 64
lora_alpha: 32 lora_alpha: 128
lora_dropout: 0.05 lora_dropout: 0.05
target_modules: target_modules:
- q_proj - q_proj

5
training/scripts/train.py
View File

@@ -52,6 +52,11 @@ def train(config_path):
) )
print("Model loaded with QLoRA (4-bit).") print("Model loaded with QLoRA (4-bit).")
# Prepare model for k-bit training
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)
print("Model prepared for k-bit training.")
# Add LoRA # Add LoRA
lora_config = LoraConfig( lora_config = LoraConfig(
r=config["lora_r"], r=config["lora_r"],