Data-efficient fine-tuning and profiling optimization: software training practices

Data-efficient fine-tuning techniques — practical examples for software training

Data-efficient fine-tuning reduces GPU-hour costs through correct sample selection, using LoRA/PEFT and 8-bit optimization combinations. Concrete tools: Hugging Face Transformers, PEFT, bitsandbytes. The example below uses a custom training loop with LoRA + 8-bit Adam (bitsandbytes) instead of the HF Trainer. To run this code install 'pip install transformers datasets accelerate bitsandbytes peft'.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model, LoraConfig
import bitsandbytes as bnb

model = AutoModelForCausalLM.from_pretrained('model-name', load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('model-name')

lora_config = LoraConfig(r=8, lora_alpha=32, target_modules=['q_proj','v_proj'], bias='none')
model = get_peft_model(model, lora_config)

optimizer = bnb.optim.Adam8bit(model.parameters(), lr=1e-4)

# basit eğitim adımı
for batch in dataloader:
    optimizer.zero_grad()
    with torch.cuda.amp.autocast():
        outputs = model(**batch)
        loss = outputs.loss
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

This example reduces memory usage via 'load_in_8bit=True' because some optimizer state and parameters are stored in 8-bit; the bitsandbytes mechanism quantizes parameters and optimizer state and uses specialized fused kernels to lower memory and memory-access cost. Experiment: run a float32 baseline and record samples/sec and GPU memory peak, then run 8-bit + LoRA and compute percent gain (example: (new/old - 1) * 100).

Profiling and bottleneck isolation — torch.profiler and NVIDIA nsys

Profiling process: (1) take a deterministic baseline, (2) change a single variable, (3) re-profile. Tools: torch.profiler (PyTorch), NVIDIA Nsight Systems (nsys) and Nsight Compute (ncu). The torch.profiler example below captures CUDA kernel durations and CPU op times; when saving the profile use record_shapes and profile_memory (requires CUPTI).

import torch
from torch.profiler import profile, record_function, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
             record_shapes=True, profile_memory=True, with_stack=True) as prof:
    with record_function('model_infer'):
        outputs = model(input_ids)

print(prof.key_averages().table(sort_by='cuda_time_total', row_limit=20))
prof.export_chrome_trace('trace.json')

Notes on measurements: torch.profiler does not require explicit CUDA synchronization to read CUDA times correctly, but for timeline analysis inspect export_chrome_trace with Chrome tracing or TensorBoard. For a broader system view use nsys: 'nsys profile --trace=cuda,cudnn,osrt -o train_report python train.py --train-args'. Nsight outputs reveal long-running CPU bottlenecks (GIL, data loader CPU cost), I/O delays (dataset.decode) and small kernel launch overheads. Example solutions: (a) increase DataLoader num_workers or use DALI for preprocessing; (b) input packing or bucketing to reduce padding; (c) enable Tensor Core usage with autocast + GradScaler. After changes, re-measure with torch.profiler / nsys and compare throughput (samples/s), GPU utilization and peak memory.

Distributed training and memory optimization — Deepspeed and FSDP configurations

To reduce duplication of optimizer/gradient/parameter state for large models use Deepspeed ZeRO or PyTorch FSDP. Tools: deepspeed, torch.distributed, nccl. The Deepspeed config example below enables ZeRO stage 2 with optimizer offload and overlap_comm optimizations; save this JSON as 'ds_config.json' and run 'deepspeed --num_gpus 8 train.py --deepspeed_config ds_config.json'.

{
  "train_batch_size": 512,
  "gradient_accumulation_steps": 8,
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true
  },
  "fp16": {"enabled": true}
}

Technical rationale: with ZeRO stage 2 parameters are not fully replicated on each process; offloading optimizer state to CPU/NVMe frees GPU memory. Communication latency may increase; overlap_comm=true helps overlap communication with computation to improve effective throughput. Measurement: use 'nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 2' and compare Deepspeed run outputs to baseline for throughput and OOM events. Edge case: high world_size combined with a very small micro_batch can trigger an allgather explosion; increase gradient_accumulation_steps or use FSDP reduce_scatter=True as a remedy.

Input preparation, padding and batching strategies — cost-reduction examples

Padding and batch arrangement are hidden cost sources in many LLM training pipelines. Tools: Hugging Face datasets and DataCollatorWithPadding, sentence packing techniques. The example below shows dynamic padding and bucketing; install with 'pip install datasets transformers'.

from datasets import load_dataset
from transformers import AutoTokenizer, DataCollatorWithPadding

dataset = load_dataset('your_dataset')
tokenizer = AutoTokenizer.from_pretrained('model-name')

def preprocess(example):
    return tokenizer(example['text'], truncation=True)

dataset = dataset.map(preprocess, batched=True)

# dynamic padding collator
collator = DataCollatorWithPadding(tokenizer=tokenizer, padding='longest')

# bucketing example (group by length bins)
def length_bucket(example):
    return len(example['input_ids']) // 64

dataset = dataset.sort('input_ids', reverse=False)
# Daha sofistike: shuffle within buckets + pack sequences to reduce padding

Practical reason: padding-inflated FLOPs cause the model to compute on padding tokens; packing increases effective tokens/sec. Measurement approach: compare cost per token for the same epoch (e.g. total FLOPs or training time divided by processed tokens). Additional note: when using variable sequence lengths, setting cudnn.benchmark=True can optimize kernel selection for some input sizes; however if input sizes are highly variable benchmark overhead may hurt performance. Detect this edge case by toggling torch.backends.cudnn.benchmark and measuring performance.

Frequently Asked Questions

Which metrics should I always collect with torch.profiler during software training?

With torch.profiler record at minimum: CPU and CUDA activities (ProfilerActivity.CPU, ProfilerActivity.CUDA), record_shapes=True, profile_memory=True and with_stack=True. This combination provides per-operator CUDA time, memory allocations and call stacks. Example: 'with profile(...): ...; prof.key_averages().table(sort_by="cuda_time_total")'. Also run 'nsys profile --trace=cuda,cudnn -o out python train.py' to see the bigger picture.

What steps should I follow to quickly isolate memory errors in a training workload?

Step 1: build a reproducer on a single GPU with a small batch. Step 2: take live measurements with 'torch.cuda.memory_summary()' and 'nvidia-smi --query-gpu=memory.used --format=csv -l 1'. Step 3: try ZeRO/FSDP/8-bit combinations; for example change Deepspeed config 'offload_optimizer' or 'stage':2. Step 4: re-measure and check OOM frequency. Edge case: if DataLoader uses excessive prefetch or has structural leaks, try disabling workers one by one to find the culprit.

Between Deepspeed ZeRO and PyTorch FSDP which is more beneficial for a software training workload?

Both shard memory; the choice depends on workload. Deepspeed ZeRO (especially with offload) suits GPU-memory-constrained setups using NVMe/CPU offload; FSDP offers tighter PyTorch integration and communication-optimized strategies like reduce_scatter/reshard. Compare with identical hyperparameters across both configs measuring throughput (samples/s), peak_memory and network traffic (nccl_stats). Example test: try Deepspeed with 'zero_optimization.stage=2' and FSDP with 'flatten_parameters=True, cpu_offload=False' and compare results.