Profiling, measurement and concrete optimization steps - practical for software training

Getting started: stable load, metrics and basic tools - software training

The foundation of any performance effort is a repeatable load scenario and defined metrics. Tools you will use: wrk or hey (HTTP load), perf + FlameGraph (native CPU), py-spy or scalene (Python), clinic.js (Node). Measurement flow: 1) create a baseline workload, 2) profile, 3) make changes, 4) re-measure and report the difference. Example baseline measurement commands:

wrk -t4 -c200 -d30s http://localhost:8080/path
# Linux native CPU sampling -> flamegraph
perf record -F 99 -g -- ./bin/myserver --config prod.yaml
perf script | ./stackcollapse-perf.pl > out.folded
./flamegraph.pl out.folded > baseline.svg

At this step be sure to use the exact same workload and dataset; record metrics such as p95, p99, throughput and RSS.

CPU hotspot isolation: Python example, py-spy and code change

Example problem: compiling a regex and copying data unnecessarily on every request inside an HTTP handler increases CPU cost. Measurement tool: take a sampling profile with py-spy and generate an SVG flamegraph, and compare with cProfile. To take a baseline:

py-spy record -o baseline.svg -- python server.py --port 8080
# Alternative cProfile run
python -m cProfile -o baseline.prof server.py

Bad example (compiling regex on every request):

import re
from flask import Flask, request
app = Flask(__name__)
@app.route('/u')
def u():
    pattern = re.compile(r"[A-Z]{3}")
    m = pattern.search(request.args.get('s', ''))
    return m.group(0) if m else 'no'

Fix: move regex compilation to module level and cache the computation:

import re
from functools import lru_cache
PAT = re.compile(r"[A-Z]{3}")
@lru_cache(maxsize=10000)
def normalize(s):
    # örnek ağır dönüşüm
    return s.strip().lower()
@app.route('/u')
def u():
    s = normalize(request.args.get('s', ''))
    m = PAT.search(s)
    return m.group(0) if m else 'no'

Why this helps: regex compilation creates CPU and memory-allocation cost; lru_cache amortizes Python function call cost for repeated inputs. After the change, re-measure with py-spy and compare baseline.svg with the new SVG; ideally the sample rate for the hotspot will drop noticeably.

I/O and concurrency: backpressure, batching and profiling with clinic.js for Node

Solutions for I/O bottlenecks usually involve concurrency control and batching. Measurement tool for Node: clinic.js (clinic doctor + flame), also 0x. Clinic command examples:

clinic doctor -- node server.js --port 3000
# to inspect results
clinic flame -- node server.js

Example change - limited concurrency promise pool:

function pLimit(concurrency){
  const queue = []
  let active = 0
  const next = () => {
    if(active >= concurrency || queue.length === 0) return
    active++
    const {fn, resolve, reject} = queue.shift()
    fn().then(v => { resolve(v); active--; next(); }).catch(e => { reject(e); active--; next(); })
  }
  return (fn) => new Promise((res, rej) => { queue.push({fn, resolve: res, reject: rej}); next(); })
}
// kullanım
const limit = pLimit(50)
await Promise.all(tasks.map(t => limit(() => fetchUpstream(t))))

Batching example: issuing selects/inserts in batches of 100 instead of one-by-one reduces the number of I/O round trips. Measure: compare latency p95/p99 and throughput with clinic; also measure Node event loop lag using process.hrtime.

Memory leaks and heap profiling: Go pprof and Java async-profiler

Provide example applications for memory traces on Go and Java. Go: add a pprof handler to the app and take a heap profile with go tool pprof. Commands:

// main.go içinde
import _ "net/http/pprof"
import "net/http"
func init(){
  go func(){ http.ListenAndServe(":6060", nil) }()
}
# Çalıştırıp profil alma
curl http://localhost:6060/debug/pprof/heap > heap.prof
go tool pprof -http=:8081 ./bin/myapp heap.prof

For Java, use async-profiler for allocation sampling:

# async-profiler dizinindeyken
./profiler.sh -d 30 -e alloc -f heap.svg 
# CPU sampled flame
./profiler.sh -d 30 -e cpu -f cpu.svg 

Leak detection: find the types and stacks holding the most memory in heap.svg or the pprof UI; a typical fix is to replace long-lived global maps with a slab pool and add object reuse. Measurement logic: compare RSS and heap in-use under the same load between baseline and patched versions; for Go record the output of go tool pprof -top.

CI integration: automated perf regression, artifacts and acceptance criteria

When you add performance tests to CI the goal is deterministic regression alerts. GitHub Actions example: run a benchmark, compute p95/p99, and fail the job if degradation is worse than 10%. Simple step example:

# .github/workflows/perf.yml snippet
- name: Run wrk baseline
  run: wrk -t2 -c50 -d20s http://localhost:8080/endpoint > wrk.baseline.txt
- name: Parse p95
  run: grep 'Latency' wrk.baseline.txt | awk '{print $3}' > p95.baseline
# Yeni commit sonrası
- name: Run wrk current
  run: wrk -t2 -c50 -d20s http://localhost:8080/endpoint > wrk.current.txt
- name: Compare
  run: |
    b=$(cat p95.baseline)
    c=$(grep 'Latency' wrk.current.txt | awk '{print $3}')
    awk -v b=$b -v c=$c 'BEGIN{ if(c/b > 1.1) exit 1 }'

Also upload profile artifacts (flamegraph.svg, heap.prof) as CI artifacts and document performance changes in the PR template. Why: reduce manual submission errors and environmental differences by using the same container image and a fixed database snapshot.

Frequently Asked Questions

How do I start Python profiling for software training?

Define your target metrics (p95, throughput, RSS). Use py-spy for a baseline: py-spy record -o baseline.svg -- python app.py. Then change the code you want to optimize and re-measure with py-spy. Compare the SVGs visually and validate with cProfile: python -m cProfile -o out.prof app.py; inspect details with snakeviz out.prof.

How do I set up performance regression tests in CI for software training?

Run a fixed load in the CI job using wrk, record p95 and throughput, then compare. Example: wrk -t2 -c50 -d20s http://localhost:8080/endpoint > wrk.out. To set a threshold, take the historical average p95 from past artifacts; add a shell check that fails the job if the new value exceeds that threshold by 10%.

How do I validate native CPU hotspots during software training?

For Linux, perf + FlameGraph is the fastest way. Steps: perf record -F99 -g -- ./bin/app; perf script | ./stackcollapse-perf.pl > out.folded; ./flamegraph.pl out.folded > flame.svg. The widest blocks on the flamegraph indicate the code paths consuming the most CPU; optimize those stacks and compare again with the same commands.