Profile-driven LLM training pipeline optimization for software training

Metric design and profiling targets — in the context of software training

For each training run pick at least three metrics: step time (median ms/step), GPU memory peak (MiB) and samples/second (throughput). Use concrete tools to collect these three measurements: PyTorch profiler for CUDA/CPU timing, NVIDIA Nsight Systems (nsys) for API/PCIe loads and nvidia-smi --query-gpu=memory.used --format=csv -l 1 to record memory windows. Example profile command:

nsys profile --output=run_trace python -m torch.distributed.run --nproc_per_node=4 train.py

Why: PyTorch profiler shows CPU/CUDA overlap at kernel level; nsys captures I/O and DMA wait time at the API/driver level, letting you distinguish whether step latency is GPU compute or host-GPU transfer.

To make profiling experiments reproducible run at least 3 consecutive warmup+measured runs before/after each change and report the median. Example shell measurement:

for i in 1 2 3; do python -m torch.distributed.run --nproc_per_node=4 train.py --profile; done

Measurement rationale: a single run is noisy, the median reduces noise and percent change (e.g. -28% step time) becomes interpretable.

Data pipeline isolation and I/O optimization

Measure I/O with Linux tools to isolate data-sourced delays: iostat -x 1, pidstat -d 1 -p and syscall counts with bpftrace. Example bpftrace command:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @[comm] = count(); }' -p $(pgrep -f train.py)

If this data shows an I/O-bound bottleneck, use WebDataset tar-shard streaming or NVIDIA DALI. WebDataset example:

from webdataset import WebDataset, split_by_worker
dataset = WebDataset('data/shards-%06d.tar').shuffle(1000).decode('pil')
loader = torch.utils.data.DataLoader(dataset, batch_size=32, num_workers=8, pin_memory=True)

Why: tar-shard streaming increases disk I/O parallelism and reduces small-file overhead; DALI moves decoding to the GPU and reduces CPU->GPU transfers.

Practical step: microbenchmark your dataset pipeline (e.g. dataloader iterator throughput) and bind that to the training step. Measurement snippet:

import time
it = iter(loader)
start = time.time()
for i in range(100):
    next(it)
elapsed = time.time() - start
print('samples/s', 32*100/elapsed)

If samples/s is below the throughput required by training, change num_workers, prefetch_factor, sharding or persistent_workers and re-measure.

Memory management and distributed optimization — ZeRO, 8-bit optimizer and grad-accum

To reduce GPU memory pressure apply two concrete approaches together: ZeRO (DeepSpeed or FairScale) for optimizer/state sharding and offload, plus 8-bit optimizers (bitsandbytes). Short DeepSpeed config example:

{
  "zero_optimization": {"stage": 2, "offload_optimizer": {"device": "cpu"}},
  "train_batch_size": 128
}

Launch command:

deepspeed --num_nodes 1 --num_gpus 4 train.py --deepspeed

Why: ZeRO stage 2/3 reduces GPU memory by partitioning optimizer state and gradients; offload moves state from GPU RAM to CPU, enabling training of larger models.

bitsandbytes 8-bit optimizer usage and PyTorch integration:

from bitsandbytes.optim import Adam8bit
opt = Adam8bit(model.parameters(), lr=1e-4)
# veya Hugging Face accelerate ile
from accelerate import Accelerator
acc = Accelerator()
model, opt, train_dataloader = acc.prepare(model, opt, train_dataloader)

Measurable outcome: the ZeRO+8bit combination can reduce GPU memory usage by ~40-70% at the same batch size; measure with nvidia-smi and compare before/after peak memory:

nvidia-smi --query-gpu=memory.used --format=csv -f mem_before.csv
# değişiklik sonrası
nvidia-smi --query-gpu=memory.used --format=csv -f mem_after.csv

Applied performance profiling example: identifying bottlenecks with torch.profiler

Sample with PyTorch profiler to extract CPU/CUDA overlap and kernel costs. The minimal example below profiles one step of the training loop:

from torch.profiler import profile, ProfilerActivity, record_function
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
             schedule=torch.profiler.schedule(wait=1, warmup=1, active=3),
             record_shapes=True, with_stack=True) as prof:
    for step, batch in enumerate(loader):
        with record_function('model_forward'):
            out = model(batch['input'])
        if step >= 5:
            break
print(prof.key_averages().table(sort_by='cuda_time_total', row_limit=10))
prof.export_chrome_trace('trace.json')

Analysis step: open the Chrome trace or TensorBoard Profiler UI with export_chrome_trace; you'll see GPU compute overlapping host transfers, kernel waits or memcpy delays clearly.

Set up an automated profiling pipeline for comparisons and regression detection: run base and head with the same seed and 3 runs, collect profiles and compare medians. CI snippet:

python tools/run_profile.py --config /configs/base.json --out base_trace
python tools/run_profile.py --config /configs/head.json --out head_trace
python tools/compare_profiles.py base_trace head_trace --metrics step_time,gpu_memory

Why: automated profile comparison brings performance regressions into code review and removes misleading single-run results.

Edge cases, fault isolation and continuous monitoring

A common edge case is intermittent OOMs during training; causes are often batch I/O differences or token length variance. Concrete approach: collect a per-batch token length histogram and cap the 99th percentile. Code example:

lengths = [len(tok) for tok in batch['input_ids']]
import numpy as np
p99 = np.percentile(lengths, 99)
if p99 > 1024:
    # overflow risk mitigation
    batch['input_ids'] = [ids[:1024] for ids in batch['input_ids']]

Also capture GPU memory dump after an OOM:

python -c 'import torch; print(torch.cuda.memory_summary())' > mem_after_oom.txt

That dump shows which tensors are being kept larger than expected.

To isolate NaN/inf errors add a gradient sanity check in the training loop. Concrete code:

for name, param in model.named_parameters():
    if torch.isnan(param).any() or torch.isinf(param).any():
        torch.save({'model': model.state_dict(), 'step': step}, 'nan_checkpoint.pt')
        raise RuntimeError('NaN detected in ' + name)

For determinism and reproducibility set seeds and try torch.backends.cudnn.deterministic or torch.use_deterministic_algorithms(True); these can affect performance but increase the chance of reproducing the issue.

Frequently Asked Questions

How to start LLM training profiling — what are the basic steps for software training?

Concrete starter steps: (1) baseline measurements: 3 runs, median step_time and GPU memory peak; (2) capture CPU/CUDA timings with PyTorch profiler:

from torch.profiler import profile, ProfilerActivity
# örnek komut yukarıdaki section'da

(3) record I/O/PCIe waits with nsys:

nsys profile --output=trace python -m torch.distributed.run --nproc_per_node=4 train.py

Then compare medians at each step.

If the data pipeline is a bottleneck during software training, which concrete measurements should be taken?

First microbenchmark dataloader throughput (samples/s) and measure disk/scheduler waits with iostat and pidstat:

iostat -x 1
pidstat -d 1 -p $(pgrep -f train.py)

If disk I/O is high use WebDataset tar-sharding or DALI and adjust num_workers/prefetch_factor and re-measure.

How does using distributed ZeRO and an 8-bit optimizer together affect software training costs?

Measure the concrete impact by recording peak memory with nvidia-smi at the same batch size, then run again with ZeRO+offload or ZeRO+bitsandbytes and report percent reduction. Example commands:

deepspeed --num_gpus 4 train.py --deepspeed
# bitsandbytes optimizer örneği kod yukarıda

Why: ZeRO partitions optimizer state to lower GPU memory, 8-bit optimizers compress optimizer state further; together they let you increase batch size or shift to lower-cost GPUs for large models.