cmedina/agenx-lora-training
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: switch from DeepSpeed ZeRO-3 to FSDP for QLoRA compatibility

Browse Source
This commit is contained in:
Christian Medina
2026-07-01 22:33:06 -04:00
parent a655040db2
commit e64515e3ca
2 changed files with 14 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
train-on-this-server.sh
View File

@@ -18,9 +18,9 @@ pip install --upgrade pip
echo "Installing training dependencies..."
pip install transformers datasets trl peft accelerate bitsandbytes deepspeed
# Run training with DeepSpeed ZeRO-3
echo "Starting training with DeepSpeed ZeRO-3..."
# Run training with FSDP
echo "Starting training with FSDP..."
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
torchrun --nproc_per_node=2 train.py --config training/configs/ornith-35b-lora.yaml
accelerate launch --multi_gpu --num_processes 2 train.py --config training/configs/ornith-35b-lora.yaml
echo "Training completed!"