Your average latency is a story about a system nobody uses
Mean response time describes a request that mostly does not exist. Tail latency describes the one your customer remembers — and at scale, almost every user hits the tail eventually.
By Hello Infotech
There is a particular kind of meeting where someone presents a dashboard showing an average response time of 40 ms, and someone else says that the application feels slow. Both are telling the truth.
The arithmetic of the tail
Consider a page that makes 20 backend calls to render. Suppose each call is fast
99% of the time and slow 1% of the time. The probability that all twenty are
fast is 0.99²⁰ ≈ 0.82.
Roughly one page load in five hits at least one slow call. The user does not experience the average of twenty calls — they experience the slowest one, because the page is not finished until the last response arrives.
This is why fan-out amplifies the tail. Add more parallel calls and the proportion of requests touching the slow path goes up, not down. A p99 that looks acceptable in isolation becomes a p80 experience once you compose enough of them.
Averages hide bimodality
An average assumes the underlying distribution has a meaningful centre. Real service latency rarely does. It is usually at least two distributions superimposed:
- Cache hit — a few hundred microseconds.
- Cache miss — tens of milliseconds.
A 90% hit rate produces an average that describes neither case. No request takes the average amount of time. Optimising against that number is optimising against a fiction.
The same shape appears wherever there is a fast path and a slow path: JIT warm versus cold, connection pooled versus newly established, small payload versus paginated.
Where the tail actually comes from
In practice, the recurring causes are few:
Queueing. As utilisation approaches capacity, queueing delay grows non-linearly. At 50% utilisation the queue is negligible; at 90% it dominates everything else. The tail is often the first visible symptom that you are running closer to the cliff than you thought.
Garbage collection and reclaim pauses. A stop-the-world pause hits whichever request is unlucky. It shows up nowhere in the mean and clearly in the p99.9.
Head-of-line blocking. One slow item in a shared queue delays everything behind it, regardless of how fast those items would otherwise have been.
Retries. A retry does not remove the original latency — it adds to it. A system that retries under load also adds load, which is how a latency problem becomes an availability problem.
Contention. Lock convoys, connection-pool exhaustion, a single hot partition. Always intermittent, always in the tail.
What to measure instead
Percentiles, at the percentile that matters. p50 for capacity planning, p99 for user experience, p99.9 for anyone whose requests fan out. Choose them deliberately.
Never average a percentile. The mean of per-instance p99 values is not the fleet p99, and there is no arithmetic that recovers it. Aggregate histograms and compute the percentile from the merged distribution.
Measure where the user is. Server-side timers exclude queueing before your handler, TLS negotiation, and the network. The number that matters is the one observed at the client.
Keep the histogram. A pre-computed percentile cannot be re-sliced later. A histogram can — by endpoint, by tenant, by region — which is exactly what you will want at 3am.
The uncomfortable conclusion
Tail latency is not an optimisation to schedule after the features. At any real scale it is the user experience, because with enough requests every user is eventually one of the unlucky ones.
The practical shift is small: put the p99 on the dashboard next to the average, and put the histogram behind it. Most teams find the conversation changes within a week.