Christian Medina
|
0456d55c8a
|
fix: disable gradient checkpointing (device mismatch with distributed MoE)
|
2026-07-02 17:29:19 -04:00 |
|
Christian Medina
|
d9c3781242
|
fix: quote 'no' in eval_strategy (YAML boolean issue)
|
2026-07-02 17:09:40 -04:00 |
|
Christian Medina
|
565ddd416e
|
fix: disable eval (no test split in dataset)
|
2026-07-02 16:49:34 -04:00 |
|
Christian Medina
|
728073ef6a
|
fix: use 'output' column (not 'text')
|
2026-07-02 16:14:53 -04:00 |
|
Christian Medina
|
0752c03b83
|
fix: use absolute path for dataset (no ~ expansion)
|
2026-07-02 16:04:21 -04:00 |
|
Christian Medina
|
b8214f64cb
|
fix: use 4-bit model, remove DeepSpeed, fix dataset path
|
2026-07-02 15:23:26 -04:00 |
|
Christian Medina
|
8747111671
|
fix: update config comment to reflect Ornith-35B
|
2026-07-02 00:11:10 -04:00 |
|
Christian Medina
|
a655040db2
|
feat: use DeepSpeed ZeRO-3 for proper model sharding across GPUs
|
2026-07-01 22:08:37 -04:00 |
|
Christian Medina
|
815115e97a
|
feat: add BnB 4-bit quantization strategy + use bf16 model
|
2026-07-01 21:14:31 -04:00 |
|
Christian Medina
|
cc353b3dea
|
refactor: restructure project - scripts at root level
|
2026-07-01 16:39:54 -04:00 |
|
Christian Medina
|
293f9caf65
|
fix: use correct repo_root path for dataset files
|
2026-07-01 16:35:47 -04:00 |
|
Christian Medina
|
7e13680e27
|
additional fixes
|
2026-07-01 16:05:52 -04:00 |
|
Christian Medina
|
088521094e
|
feat: add 4 loading strategies (no BnB for already-quantized model)
|
2026-07-01 15:57:42 -04:00 |
|
Christian Medina
|
3089f27901
|
fix: load 4-bit model AS-IS without BnB (already quantized)
|
2026-07-01 15:56:00 -04:00 |
|
Christian Medina
|
1197235b4d
|
fix: skip prepare_model_for_kbit_training to avoid OOM
|
2026-07-01 15:18:48 -04:00 |
|
Christian Medina
|
d9fd079686
|
fix: add accelerate CPU offload to bf16 strategies
|
2026-07-01 15:14:00 -04:00 |
|
Christian Medina
|
4ed80ad3a3
|
feat: restore all 6 loading strategies
|
2026-07-01 15:11:45 -04:00 |
|
Christian Medina
|
313b44381f
|
fix: remove DeepSpeed, use torchrun without distributed training
|
2026-07-01 15:10:54 -04:00 |
|
Christian Medina
|
7de54746a2
|
feat: add 6 loading variants with different offload strategies
|
2026-07-01 14:45:29 -04:00 |
|
Christian Medina
|
22d1bfd573
|
fix: use 4-bit model first, then bf16 with CPU offload
|
2026-07-01 14:43:30 -04:00 |
|
Christian Medina
|
3fa2cfe4cc
|
fix: use DeepSpeed ZeRO-3 directly (skip CPU loading)
|
2026-07-01 14:41:48 -04:00 |
|
Christian Medina
|
3760aaf316
|
fix: rename output dir to ornith-35b-lora
|
2026-07-01 14:19:43 -04:00 |
|
Christian Medina
|
ce758f7eb5
|
feat: improve loading strategies with low_cpu_mem_usage and proper DeepSpeed config
|
2026-07-01 14:00:41 -04:00 |
|
Christian Medina
|
2d13e63c25
|
feat: add 4 loading strategies with automatic fallback
|
2026-07-01 13:55:23 -04:00 |
|
Christian Medina
|
b300235be0
|
fix: load 4-bit model AS-IS without additional quantization
|
2026-07-01 13:53:43 -04:00 |
|
Christian Medina
|
e967bd74f1
|
fix: skip quantization, use DeepSpeed ZeRO-3 CPU offload directly
|
2026-07-01 13:53:06 -04:00 |
|
Christian Medina
|
7539e39a93
|
fix: use accelerate load_checkpoint_and_dispatch for proper CPU offload
|
2026-07-01 13:39:51 -04:00 |
|
Christian Medina
|
528e7002ef
|
feat: add 8-bit CPU offload fallback between 4-bit and bf16
|
2026-07-01 13:32:04 -04:00 |
|
Christian Medina
|
59ddfb45ed
|
feat: add fallback to bf16 with CPU offload if 4-bit fails
|
2026-07-01 13:29:23 -04:00 |
|
Christian Medina
|
b41d99f956
|
fix: remove broken quantization_config before loading
|
2026-07-01 13:12:57 -04:00 |
|
Christian Medina
|
1228a1f38b
|
fix: load model first then quantize with BnB
|
2026-07-01 13:11:14 -04:00 |
|
Christian Medina
|
14042dbb44
|
fix: use pre-quantized 4bit model
|
2026-07-01 12:35:19 -04:00 |
|
Christian Medina
|
f784d06f0a
|
fix: load QLoRA directly to GPU (skip CPU)
|
2026-07-01 12:33:21 -04:00 |
|
Christian Medina
|
9471d5cc77
|
fix: cast learning_rate to float (YAML parses 2e-4 as string)
|
2026-07-01 11:19:00 -04:00 |
|
Christian Medina
|
2fa52cd9d4
|
fix: load QLoRA model to CPU first, DeepSpeed distributes
|
2026-07-01 09:37:20 -04:00 |
|
Christian Medina
|
2fa5173c56
|
feat: add prepare_model_for_kbit_training and update LoRA r=64 alpha=128
|
2026-07-01 07:48:44 -04:00 |
|
Christian Medina
|
357c9fa781
|
fix: use standard QLoRA pattern with BitsAndBytesConfig
|
2026-07-01 07:47:42 -04:00 |
|
Christian Medina
|
da5eb3abed
|
feat: implement QLoRA with 4-bit BitsAndBytes quantization
|
2026-07-01 07:45:33 -04:00 |
|
Christian Medina
|
f0ee6bc9a2
|
fix: torch_dtype -> dtype (fix deprecation warning)
|
2026-07-01 07:38:36 -04:00 |
|
Christian Medina
|
f651ca030c
|
fix: convert warmup_ratio to warmup_steps (fix deprecation warning)
|
2026-07-01 07:37:47 -04:00 |
|
Christian Medina
|
f47944ee75
|
fix: remove quantization_config after loading to bypass SFTTrainer check
|
2026-07-01 07:36:38 -04:00 |
|
Christian Medina
|
86b401c12f
|
fix: convert FP8 to bf16 for PEFT compatibility
|
2026-07-01 07:30:10 -04:00 |
|
Christian Medina
|
6f631f00ae
|
fix: use FP8 model with 32GB CPU offload
|
2026-07-01 07:29:32 -04:00 |
|
Christian Medina
|
9d92531c55
|
fix: use Ornith-1.0-35B bf16 base model (PEFT compatible)
|
2026-07-01 07:28:21 -04:00 |
|
Christian Medina
|
05ace2d38f
|
fix: load NVFP4 as bf16 by overriding quantization config
|
2026-07-01 07:27:13 -04:00 |
|
Christian Medina
|
be7b589c6b
|
fix: use BitsAndBytes INT4 for PEFT compatibility
|
2026-07-01 00:34:34 -04:00 |
|
Christian Medina
|
d7301d37dd
|
fix: use float16 dtype to preserve NVFP4 quantization
|
2026-07-01 00:08:42 -04:00 |
|
Christian Medina
|
af09d47c6f
|
fix: load model to CPU first, let DeepSpeed distribute
|
2026-07-01 00:03:24 -04:00 |
|
Christian Medina
|
905c2ba30d
|
fix: preserve NVFP4 quantization, don't force bf16
|
2026-06-30 23:50:27 -04:00 |
|
Christian Medina
|
114e25fcfb
|
fix: use device_map=auto to distribute model across GPUs
|
2026-06-30 23:45:57 -04:00 |
|