3. Latency & Load Testing Metrics — Deep-Dive Reference

3.1 Never trust the average

Latency distributions are heavy-tailed. Averaging hides the exact thing you care about: how bad is the worst experience a real user hits?

Example: 1000 requests
  990 requests: 20ms
  10 requests:  2000ms   (GC pause, lock contention, slow DB query)

  Average  = (990*20 + 10*2000) / 1000 = 39.8ms   <- looks fine!
  p50      = 20ms
  p99      = 2000ms                                 <- the real story

Always report p50, p90, p95, p99, p99.9, plus max. For systems with SLAs, p99.9 or p99.99 is often the contractually meaningful number (1 in 1,000 or 1 in 10,000 users).

3.2 Tail latency amplification (fan-out)

If a single user request fans out to 20 backend calls, and each backend call has a p99 of 100ms (i.e., 1% chance of being slow), the probability that at least one of the 20 calls is slow is:

P(at least one slow) = 1 - (1 - 0.01)^20 ≈ 18%

So a "p99 of 100ms" backend turns into an effective p50-ish experience of slowness for a fanned-out caller. This is why high-fan-out architectures (search, feed aggregation, microservices with deep call graphs) must budget latency far more conservatively per hop than the end-to-end SLA suggests. Mitigations: hedged requests, timeouts + retries with jitter, request-scoped deadlines.

3.3 Little's Law — the load-testing Rosetta Stone

L = λ * W

L = average number of requests in the system (concurrency)
λ = average arrival rate (throughput, req/s)
W = average time a request spends in the system (latency)

This single equation tells you, given any two of {concurrency, throughput, latency}, the third. It's how you sanity-check load test results:

Example: your tool reports throughput = 500 req/s, p50 latency = 200ms.
Expected concurrency in flight: L = 500 * 0.2 = 100 concurrent requests.

If your load generator was configured for 100 concurrent workers, and it produced
exactly ~500 req/s at ~200ms latency, the numbers are self-consistent — good.
If concurrency was configured at 1000 but throughput stayed at 500 req/s, your
server has hit a **concurrency ceiling** (thread pool, connection pool, DB pool
exhaustion) — requests are queueing somewhere, and the 200ms you're measuring is
now including queue wait time, not just service time.

3.4 Open-loop vs. closed-loop load generators

This is the most-misunderstood concept in load testing.

• Closed-loop: a fixed number of virtual users, each waits for a response before sending the next request (think of N people repeatedly hitting refresh). As the server slows down, the arrival rate drops to compensate — the tool self-throttles and hides how bad things really are. Most naive load-test scripts are accidentally closed-loop.
• Open-loop: requests arrive at a fixed rate (e.g., Poisson-distributed at 1000 req/s) regardless of how fast the server responds. This matches real-world traffic — real users don't stop arriving because your server is slow. Open-loop testing reveals queueing collapse that closed-loop testing conceals.

Rule of thumb: for capacity planning and finding breaking points, use open-loop (rate-based) load generation — e.g., vegeta, k6 with arrival-rate executors, wrk2. For simulating a fixed pool of real users browsing a site, closed-loop (ab, plain wrk, Locust in its default mode) is fine.

3.5 Latency vs. response time vs. RTT — precise definitions

These three terms get used interchangeably in casual conversation but mean different things, and mixing them up produces reports that don't compare apples-to-apples:

When a load report says "latency: 240ms," always ask: is this response time including network, or server-reported service time? A load generator sitting in the same AZ as the server and one running from a user's mobile device on the other side of the planet will report wildly different "latency" for an identical backend — because most of the number is §2's network fundamentals, not the application.

3.6 Average (mean) — what it's actually good for

mean = (sum of all latencies) / (count of requests)

The mean is not useless — it's the right metric when you care about aggregate cost, not the worst individual experience: total CPU-seconds consumed, total time budget across a batch job, or cost-per-request billing models. It is the wrong metric for user-experience SLAs precisely because it's sensitive to outliers in one direction only (a few very slow requests drag it up) while being insensitive to how many users actually experienced that slowness. Never report a mean alone — always pair it with at least p50 and p99 so a reader can tell whether the distribution is tight or heavy-tailed (§3.1's worked example: mean 39.8ms with p99 of 2000ms is a very different system than a mean of 39.8ms with p99 of 45ms, even though the mean is identical).

3.7 Median (p50) vs. percentiles — computation methods matter

The median is "the middle value" — but how you find it, and how you find p90/p95/p99/p99.9 above it, is not one universally agreed algorithm. Two common methods, applied to 10 sorted latency samples (ms): 12, 15, 18, 20, 22, 25, 30, 45, 90, 250:

Nearest-rank (used by many load-testing tools' simple implementations):

rank = ceil(p/100 * N)
p50 -> rank = ceil(0.50*10) = 5  -> 5th value = 22ms
p90 -> rank = ceil(0.90*10) = 9  -> 9th value = 90ms

Linear interpolation (numpy's default np.percentile, many stats libraries):

index = (N-1) * p/100
p50 -> index = 9*0.50 = 4.5 -> interpolate between 5th(22) and 6th(25) = 23.5ms
p90 -> index = 9*0.90 = 8.1 -> interpolate between 9th(90) and 10th(250) = 106ms

Notice p90 differs by 90ms vs. 106ms — a ~18% difference — purely from methodology, with identical raw data. Never compare a p99 from one tool against a p99 from another tool without confirming they use the same computation method, and never compare against a hand-rolled percentile function without checking which convention it follows.

At scale, sorting every raw sample is expensive (O(n log n) time, O(n) memory for a multi-hour test generating millions of samples). Production-grade tools use streaming approximate-percentile structures instead:

• HdrHistogram (used internally by wrk2, gatling, and available as a library for custom tools — see §9.4's use of it in Rust): buckets values into logarithmically-sized ranges, giving a fixed relative error (e.g., 3 significant figures ⇒ ~0.1% error) across a huge dynamic range (microseconds to minutes) in O(1) memory and O(1) recording time per sample, regardless of how many samples you record.
• t-digest: a different approximate-quantile sketch, popular in monitoring/APM backends (Datadog, Prometheus's summary type historically), optimized for accuracy at the extreme tails (p99.9+) at the cost of coarser accuracy near the median.

Demonstrating the two methods directly in Python:

import numpy as np

samples = [12, 15, 18, 20, 22, 25, 30, 45, 90, 250]

def nearest_rank_percentile(data, p):
    data = sorted(data)
    n = len(data)
    rank = max(1, int(np.ceil(p / 100 * n)))
    return data[rank - 1]

print("nearest-rank p90:", nearest_rank_percentile(samples, 90))       # 90
print("linear-interp p90:", np.percentile(samples, 90))                 # 106.0

3.8 Standard deviation, variance, and coefficient of variation

variance   = mean((x_i - mean)^2)
stddev     = sqrt(variance)
coeff. of variation (CV) = stddev / mean

Standard deviation quantifies spread around the mean, but on the heavy-tailed distributions typical of latency data it's dominated by the same outliers that make the mean misleading (§3.6) — it is rarely reported on its own for latency, but the coefficient of variation (stddev normalized by the mean) is a useful single number for comparing the relative stability of two systems with different absolute latencies: a CV near 0 means very consistent latency, a CV above ~1 signals a heavy tail or bimodal distribution worth investigating with a full histogram rather than summary statistics alone.

3.9 Jitter — variation between consecutive requests

Percentiles describe the overall shape of a distribution but say nothing about order — two systems can have identical p50/p95/p99 while one delivers steady, predictable latency and the other swings wildly request-to-request. That swinginess is jitter, and it matters independently of percentiles for anything sensitive to consistency rather than just worst-case bound: VoIP, video calls, multiplayer gaming, financial market-data feeds, and any UI that does incremental/streaming rendering.

Same p50 (50ms), wildly different jitter:

System A (low jitter):  48, 51, 49, 52, 50, 49, 51, 50ms   — smooth, predictable
System B (high jitter): 5, 95, 10, 88, 3, 99, 15, 85ms     — same mean, unusable for real-time media

Two common ways to quantify it:

1. Simple successive-difference jitter (easy to compute from a load test's raw latency series):

jitter = mean(|latency[i] - latency[i-1]|) for consecutive requests

2. RFC 3550 (RTP) smoothed jitter estimate — the formula VoIP/video systems actually use internally, an exponentially-weighted moving average of the inter-packet spacing difference between sender and receiver:

J(i) = J(i-1) + (|D(i-1,i)| - J(i-1)) / 16

where D(i-1,i) is the difference in relative transit time between two consecutive packets. The /16 smoothing constant (a 1/16 gain, per the RFC) keeps the estimate stable against single outliers while still tracking real trend changes.

// rtp_jitter.c — RFC 3550 style jitter estimator over a stream of timestamps.
// send_ts[]/recv_ts[] are in the same clock units (e.g., RTP timestamp units).
#include <stdio.h>
#include <stdlib.h>
#include <math.h>

double compute_rfc3550_jitter(const double *send_ts, const double *recv_ts, int n) {
    double jitter = 0.0;
    double prev_transit = recv_ts[0] - send_ts[0];

    for (int i = 1; i < n; i++) {
        double transit = recv_ts[i] - send_ts[i];
        double d = fabs(transit - prev_transit);
        jitter += (d - jitter) / 16.0;   // RFC 3550 sec 6.4.1
        prev_transit = transit;
    }
    return jitter;
}

int main(void) {
    // simulated send times (steady 20ms cadence) vs. arrival times (network jitter)
    double send[] = {0, 20, 40, 60, 80, 100, 120, 140};
    double recv[] = {5, 30, 42, 90, 83, 140, 121, 200};
    int n = sizeof(send) / sizeof(send[0]);

    double j = compute_rfc3550_jitter(send, recv, n);
    printf("estimated jitter: %.3f (same units as timestamps)\n", j);
    return 0;
}

gcc -O2 -o rtp_jitter rtp_jitter.c -lm && ./rtp_jitter

Simulating jitter for a load test: tc netem's second delay argument is exactly this — delay 100ms 20ms distribution normal (§8.1) adds 100ms latency with 20ms of jitter drawn from a normal distribution, letting you reproduce System A/B above on a real network path and verify your client's jitter buffering/retry logic under each condition.

3.10 Fan-out — the full math

§3.2 introduced the headline formula; here's the complete picture for both fan-out shapes.

Parallel fan-out (breadth) — N independent calls issued concurrently, caller waits for all N (a typical "gather" / Promise.all pattern):

P(at least one of N calls exceeds threshold T) = 1 - (1 - p)^N

where p is the single-call probability of exceeding T (e.g., p = 0.01 for a p99 threshold).

Beyond the failure-threshold framing, fan-out breadth also drags up the expected latency even when nothing "fails" — the caller's total time is the maximum of N samples (an order statistic), not the mean of one. For latency distributions with an exponential-like tail, the expected maximum of N iid samples grows roughly with scale * ln(N) — i.e., fan-out breadth costs you tail latency logarithmically, for free, even under perfectly nominal conditions. This is why search/aggregation systems that fan out to dozens or hundreds of shards invest heavily in hedged requests (fire a duplicate request to a second replica if the first hasn't returned within, say, the p50 latency, and take whichever answers first) and in tail-tolerant timeouts + cancellation rather than trying to make every single shard uniformly fast.

Sequential fan-out (depth) — a chain of M dependent hops, each waiting for the previous:

total_latency ≈ sum(latency_i) for i in 1..M          (means simply add)
total_variance ≈ sum(variance_i) for i in 1..M         (variances add for independent hops)
total_stddev = sqrt(total_variance)

Because variances (not standard deviations) add, the relative jitter of a long chain shrinks somewhat as depth grows (central limit theorem smoothing), but the absolute latency grows linearly with every hop — which is why deep microservice call chains (A → B → C → D → E) are budgeted hop-by-hop in strict latency budgets (each service typically gets no more than its allotted milliseconds of the end-to-end SLA), rather than hoping it works out.

Most real systems are a mix of both shapes (a request fans out breadth-wise to several services, each of which internally makes a sequential chain of calls to its own dependencies) — model your load test's synthetic backend delays (e.g., via Toxiproxy, §8.2) to match the actual shape of your call graph, not a flat single-hop assumption.

3.11 Throughput, RPS, QPS, TPS, and goodput — same-sounding, different things

These terms are used inconsistently across tools and teams; always state units explicitly in a report rather than assuming "throughput" means the same thing to every reader.

A concrete trap: a service under retry storms (§6.1, §6.3) can show rising raw RPS/throughput at the load balancer while goodput is flat or falling — the extra "throughput" is wasted work from clients retrying failed or timed-out calls. Always track success-only throughput (goodput) alongside raw request rate; a load test dashboard that only shows total RPS can make a degrading system look like it's handling more traffic than it actually is.

Comparing numbers across tools: "500 TPS" from a JMeter test plan where each transaction contains 5 HTTP samplers is 2500 RPS at the HTTP layer — not directly comparable to "500 RPS" reported by wrk against a single endpoint. Always normalize to requests-per-second at a specific, named endpoint before comparing results across tools or teams.

3.12 Concurrency, saturation, and queue depth

• Concurrency / VUs (virtual users): the closed-loop knob most tools expose (-c in wrk/hey, --users in Locust/Goose). Not the same as requests-in-flight if virtual users include think-time/sleep between actions — a VU that spends 90% of its time sleeping contributes far less to requests-in-flight than a VU hammering back-to-back. Little's Law (§3.3) gives you the real requests-in-flight number: L = λ * W, independent of how many VUs are configured.
• Saturation: the fraction of a resource's capacity currently in use, and — more importantly — whether work is queueing for it (USE method, §11.2). Saturation is a leading indicator: queue depth on a thread pool or DB connection pool rises before p99 latency visibly degrades, because the first queued requests still complete reasonably fast — watch queue depth / pool-utilization metrics directly rather than waiting for the latency graph to confirm what's already happening.
• Queue / wait time: response_time − service_time (§3.5) — the portion of latency spent waiting rather than being processed. A load test that shows rising p99 with flat service-time-per-request (visible via APM/tracing) but rising queue time is diagnosing a concurrency-limit problem (thread pool, connection pool, semaphore) rather than a per-request performance problem — the fix is capacity/pooling, not code optimization.

3.13 Error rate and Apdex

error_rate = failed_requests / total_requests

Always segment error rate by status-code class (4xx client errors mixed into the same bucket as 5xx server errors hides real degradation) and watch it alongside latency — a service that starts returning fast 503s under overload (load shedding working as designed, §5.3) will show a latency improvement and an error-rate spike simultaneously; reading either metric alone gives the wrong story.

Apdex (Application Performance Index) condenses latency into a single satisfaction score using a target threshold T:

satisfied   = requests with response_time <= T
tolerating  = requests with T < response_time <= 4T
frustrated  = requests with response_time > 4T

Apdex = (satisfied + tolerating/2) / total_requests     // range: 0 (worst) to 1 (best)

Apdex trades away percentile granularity for a single trend-able number executives/dashboards can watch over time — useful as a high-level health indicator, but always keep the underlying percentiles available for actual debugging; Apdex alone can't tell you whether the slow tail is 2xT or 50xT.

3.14 Don't average percentiles — merging results across nodes/shards

A subtle but common mistake: if you run a distributed load test (Locust workers, k6 with multiple instances, or a fleet of servers each reporting their own p99), you cannot average the per-node p99 values and call the result the global p99 — percentiles are not linear, so mean(p99_node1, p99_node2, ..., p99_nodeN) is mathematically meaningless and typically underestimates the true global tail.

The correct approach is to merge the underlying histograms (not the derived percentiles) and compute the percentile once from the combined data — exactly what HdrHistogram is designed to support:

// merge_histograms.go — combine per-node HdrHistogram data into one global view
package main

import (
	"fmt"
	"github.com/HdrHistogram/hdrhistogram-go"
)

func main() {
	// each node records its own latencies independently during the test
	node1 := hdrhistogram.New(1, 60_000, 3) // 1us..60s range, 3 significant figures
	node2 := hdrhistogram.New(1, 60_000, 3)

	for _, v := range []int64{20, 22, 25, 30, 45} {
		node1.RecordValue(v)
	}
	for _, v := range []int64{18, 21, 90, 250, 26} {
		node2.RecordValue(v)
	}

	// WRONG: (node1.ValueAtQuantile(99) + node2.ValueAtQuantile(99)) / 2
	// RIGHT: merge raw histograms, then compute the percentile once
	global := hdrhistogram.New(1, 60_000, 3)
	global.Merge(node1)
	global.Merge(node2)

	fmt.Printf("node1 p99=%d node2 p99=%d (do NOT average these)\n",
		node1.ValueAtQuantile(99), node2.ValueAtQuantile(99))
	fmt.Printf("global p99 (correct, from merged histogram)=%d\n",
		global.ValueAtQuantile(99))
}

Locust's distributed mode and k6's cloud/operator output already do this merge internally before reporting aggregate percentiles — but any custom multi-node harness (§9) must merge raw histograms itself, or report per-node percentiles clearly labeled as such rather than a misleading blended average.

3.15 Worked example — reading a real report end-to-end

A typical vegeta report (§7.3, §10.1) output looks like this:

Requests      [total, rate, throughput]  30000, 500.02, 499.87
Duration      [total, attack, wait]      60.016s, 60s, 16.3ms
Latencies     [min, mean, 50, 90, 95, 99, max]  8.1ms, 24.3ms, 19.8ms, 38.1ms, 52.4ms, 187.6ms, 1.204s
Success       [ratio]                    99.62%
Status Codes  [code:count]               200:29886  503:114

Reading it with everything covered in this section:

• Requests rate (500.02) vs. throughput (499.87): rate is what you asked for (open-loop target); throughput is what was actually achieved (successful completions/sec, §3.11's "goodput" sense) — the tiny gap here (0.15 req/s) is healthy; a large gap means the server couldn't keep up with the requested rate.
• mean (24.3ms) vs. p50 (19.8ms): mean is pulled above the median (§3.6) — expected, and the gap is small, so the tail isn't yet extreme.
• p99 (187.6ms) vs. max (1.204s): a big jump from p99 to max tells you a very small number of requests (≤1%, potentially just 1-2 out of 30,000) hit something pathological (GC pause, lock, cold cache) — worth cross-referencing against server-side APM traces at that exact timestamp, not something to over-index on if it's truly one outlier.
• Success ratio (99.62%) and the 503 count (114): 114/30000 ≈ 0.38% error rate, consistent with the success ratio — if this were a flash-sale test (§6.1), 503s here might be correct load-shedding behavior rather than a bug; check what the 503s correlate with (a specific hot endpoint? a specific time window during the ramp?) before treating the error rate alone as pass/fail.
• What's missing from this report: jitter, saturation/queue-depth, and service-vs-wait-time breakdown — none of which vegeta captures on its own. Get those from server-side metrics (APM, Prometheus histograms, §11) captured during the same time window, and correlate by timestamp — a load tool's client-side report is only half the picture (§3.5).

Quick-reference: metrics glossary