Why your load test lies: open versus closed workload models

how a load test can report a healthy latency tail while the real one is much worse

Some performance regressions only show up in production. The load test passed and the dashboards looked fine, so the team shipped. Within twenty minutes of real traffic the latency climbed sharply. Nobody had introduced a bug. The load test was just answering a different question than everyone thought it was.

A note on units first. Latency is how long one request takes. We rarely care about the average; we care about the slow requests, so we talk in percentiles. The p99 latency is the response time that all but the slowest 1 percent of requests stay under (written p99). It describes the slow tail. The p50, also called the median, is the typical request: half are faster, half are slower. There is also p90 (all but the slowest 10 percent stay under it) and p99.9 (all but the slowest 0.1 percent), which reach further into the tail.

The question the test answered: "how fast does this service respond when I send a request, wait for the response, and only then send the next one?" The question everyone thought it answered: "how does this service behave when real users keep arriving regardless of how slow it is?"

These are not the same. The first is a closed-loop model: a fixed set of workers that each wait for a reply before sending again, so the worker count is the dial. The second is an open-loop model: requests arrive on a clock that does not care how slow the service is. The gap between those two models is where a closed-loop test hides its worst latencies.

The setup

Picture a recommendation service called dovetail that serves product cards. The team runs a nightly benchmark with a tool I will call loadwhip (any closed-loop tool with concurrency knobs works the same way). The script:

# nightly_bench.py
import asyncio, httpx, time

CONCURRENCY = 200
DURATION_S = 60
URL = "http://dovetail.internal/cards?user=42"

async def worker(client, results):
    end = time.monotonic() + DURATION_S
    while time.monotonic() < end:
        t0 = time.monotonic()
        r = await client.get(URL)
        results.append((time.monotonic() - t0) * 1000)

async def main():
    results = []
    async with httpx.AsyncClient(timeout=10) as client:
        await asyncio.gather(*[worker(client, results) for _ in range(CONCURRENCY)])
    results.sort()
    p50 = results[len(results) // 2]
    p99 = results[int(len(results) * 0.99)]
    print(f"requests={len(results)} p50={p50:.1f}ms p99={p99:.1f}ms")

asyncio.run(main())

One detail in the code: time.monotonic() is a clock that only ever moves forward and is not affected by the system clock being adjusted, so it is safe for measuring how long something took.

This reports something like:

requests=482103  p50=4.2ms  p99=12.1ms

A p99 of 12ms at 200 concurrent workers reads as healthy, so the team ships.

The next morning real traffic arrives. It is not 200 patient workers but an arrival process: a stream of requests whose timing is set by external demand, not by how fast the server is. A request shows up roughly every N microseconds, whether or not the previous one has finished. The same service, at the same total request rate, suddenly reports a p99 of 340ms. Only the test's model changed.

Closed loop versus open loop

The terminology comes from queueing theory, the branch of math that studies how lines form when work arrives faster than it can be served. It has been understood for roughly sixty years.

CLOSED LOOP (what most load testers do by default)
+---------+   request    +---------+
| worker  | -----------> | service |
| (waits) | <----------- |         |
+---------+   response   +---------+
   |  ^
   |  | "I will not send the next one
   v  |  until I see the response"

OPEN LOOP (what real users do)
+---------+   request    +---------+
| arrival | -----------> | service |
| process |              |         |
+---------+              +---------+
   |
   | new request fires on a schedule,
   v independent of any response

In the closed model, the load generator has a fixed number of virtual users, each one a strict request-then-response loop. If the service slows down, the workers slow with it and the load they send drops automatically. The system can never overload itself, because the test is back-pressured: backpressure is when a slow consumer signals upstream to slow down. Here a slow response makes a worker wait, and a waiting worker issues fewer requests.

In the open model, arrivals are independent of completions. If the service slows down, requests pile up and the queue grows. The latency of the request at the back of the queue includes the wait for everything in front of it. This is what real users do: your browser fires the request whether or not the previous user's has finished.

These two models measure different things. Service time is how long the server takes once it starts working on your request. Response time is service time plus queue wait, where queue wait is the time your request sits in line before the server picks it up. The closed model measures service time; the open model measures response time, which is what a user feels. For an idle system the queue wait is zero and the two coincide. For a loaded system they diverge by a wide margin, because queue wait is exactly the term the closed test refuses to grow.

Why p99 looks healthy in closed mode

Closed-loop testers self-throttle on slow responses, so the slow responses never interact with each other.

Imagine dovetail has a GC pause every few seconds that adds 200ms to about 0.5% of responses. GC stands for garbage collection: the runtime periodically halts request work to reclaim memory it is no longer using. A GC pause is one of those halts, during which nothing else runs. The two models handle the stall in opposite ways:

sequenceDiagram
    participant W as Closed worker
    participant A as Open arrivals
    participant S as Service stalled 200ms
    Note over S: GC pause begins
    W->>S: request, then waits 200ms
    Note over W: sends nothing during stall
    W-->>W: one 200ms sample
    A->>S: req at t=0
    A->>S: req at t=50
    A->>S: req at t=150
    Note over A,S: all queue behind the stall
    S-->>A: many slow samples, growing waits

The closed worker that hits the stall just waits. It records one bad sample, and the p99 barely moves. The p99 holds steady only because the generator silently drops the requests it would otherwise have sent during the stall, under-reporting the tail. In open mode at the same average rate, every request arriving during those 200ms inherits a wait that grows as the queue lengthens, so one 200ms event becomes many slow responses and p99 jumps. (The runtime mitigation, adaptive concurrency limits, is a separate story; see blog #14. The point here is that the test never surfaced the problem at all.)

This is what Gil Tene named "coordinated omission" in his 2013 talk: the generator and the service have implicitly coordinated to omit the consequences of slow events. The bad samples that should exist were never generated, because the very workers that would have generated them were stuck waiting on the stall.

What the corrected number looks like

When we rerun the dovetail benchmark with an open-model generator, we read the achieved throughput off the closed test and fire at that rate. The closed test completed 482,103 requests in 60 seconds, about 8,035 requests per second. Throughput is how many requests the service completes per second, written rps (requests per second), sometimes qps (queries per second); the two mean the same thing. The numbers below are illustrative for this fictional service, though the order-of-magnitude shape is consistent with published coordinated-omission demonstrations by Gil Tene and ScyllaDB:

Throughput matched because dovetail had enough capacity at 8,035 rps. Had the open-loop test targeted a higher rate than the service could sustain, achieved throughput would fall below target and the queue would grow for the whole run instead of settling. The p50 is barely different. The p99 is 28x worse, the p99.9 roughly 100x worse. The tail diverges far more than the median because a median request almost never lands during a stall, while a tail request arrived behind one and inherited the full queue. The closed test hid those requests.

The shape of the divergence is the diagnostic. If open and closed numbers differ a lot at the tail and little at the median, you have a queueing problem hiding behind a closed-loop test. If they differ at both, you also have a throughput ceiling.

How to actually configure an open-loop test

The major modern load testers all support open-loop generation. You have to ask for it explicitly, because the defaults are almost always closed-loop. A few I have used:

• wrk2, Gil Tene's fork of Will Glozer's wrk, written specifically to address coordinated omission. Pass -R <rate> and it fires at that rate regardless of response time, measuring latency from when the request should have been sent rather than when it actually went out.
• k6 has constant-arrival-rate and ramping-arrival-rate executors, which are open-loop. The default constant-vus executor models a fixed pool of looping virtual users, which is closed-loop in effect for single-request scenarios. (VUs is k6's abbreviation for virtual users.)
• vegeta is open-loop by default. vegeta attack -rate=8000/s -duration=60s produces an open-loop test with no extra config.
• Gatling has constantUsersPerSec and rampUsersPerSec. Its constantConcurrentUsers and rampConcurrentUsers are closed-loop (the injectClosed model); the open-model injection steps include atOnceUsers, rampUsers, constantUsersPerSec, and rampUsersPerSec (docs.gatling.io/concepts/injection/).

The same benchmark in k6 as an open-loop test:

// nightly_bench_open.js
import http from 'k6/http';

export const options = {
  scenarios: {
    real_traffic: {
      executor: 'constant-arrival-rate',
      rate: 8000,           // 8000 requests/sec, regardless of latency
      timeUnit: '1s',
      duration: '60s',
      preAllocatedVUs: 500,  // pool to draw from
      maxVUs: 4000,          // ceiling if service gets slow
    },
  },
  thresholds: {
    http_req_duration: ['p(99)<50'], // this is what we actually care about
  },
};

export default function () {
  http.get('http://dovetail.internal/cards?user=42');
}

A few things to notice. rate is in requests per second, not virtual users. preAllocatedVUs and maxVUs are the pool of virtual users the executor draws from when the service slows down. Set maxVUs high enough that the generator never becomes the bottleneck: at least rate * worst_acceptable_latency_seconds * 4. That is Little's law with a safety factor. Little's law is a conservation identity, not a tuning trick: the average number of requests in flight equals the arrival rate times the average time each spends in the system (L = rate * latency). A 3x latency spike triples the virtual users needed at a fixed rate, and the 4x cap covers that. Finally, the threshold is on response time, which now includes queue wait.

If you must stay in a closed-loop tool, wrk2 will at least correct the recorded latencies. When a response takes longer than the planned gap between requests, it back-fills synthetic samples for the requests that should have been sent during the stall, timing each from its scheduled send time rather than its actual one. If requests are scheduled every 1ms and a 200ms stall hits, the request scheduled for t=50ms but not sent until t=200ms is recorded as 150ms of wait plus service. wrk2 tracks both a corrected and an "uncorrected" histogram. (A histogram here is the distribution of measured latencies, bucketed by value.) See github.com/giltene/wrk2.

The diagnostic checklist

If you inherit a load test and want to know whether it is lying to you, check these in order:

1. Look at the tool invocation. Is it vus, concurrency, threads, or clients? Closed loop. Is it rate, rps, arrival-rate, or qps? Open loop.
2. Check whether the test reports a target rate or an achieved rate. Closed-loop tools usually report only achieved rate, since the rate is whatever the service let them have. Open-loop tools report both, and the gap between target and achieved is the first thing to look at.
3. Plot p50 against p99 over time. If they move together, you are measuring service time. If p99 spikes while p50 stays flat, you are measuring queue wait, the real signal.
4. A closed-loop worker is serial: it holds at most one slow request at a time, because it will not send the next until the current one returns. Over a run of length T with C workers, any single worker's slowest response cannot exceed roughly T/C. Compare the observed max to test_duration / concurrency (a 60s run at 200 workers gives 300ms). A max suspiciously near that ceiling means the test was concurrency-bound, not service-bound: the workers set the worst case, not the service.

When closed loop is actually correct

Closed-loop modeling is the right answer when your real workload is also closed-loop. An RPC chain (RPC stands for remote procedure call: one service calling another over the network as if it were a local function) where the calling service has a strict connection pool and applies its own backpressure is genuinely closed-loop. Batch pipelines processing fixed-size chunks are closed-loop. A message-queue worker with prefetch=N is closed-loop with N: in a message queue, consumers pull jobs and acknowledge each when done, and prefetch=N caps how many pulled-but-not-yet-acknowledged messages a consumer holds at once, so it self-limits exactly the way N looping workers do.

For these systems, a closed-loop test with concurrency = real production concurrency is right, and an open-loop test would overstate tail latency by simulating queue conditions that cannot occur.

The mistake is using closed-loop tests for inherently open-loop workloads: anything user-facing, anything triggered by external events, anything fronting a CDN (a content delivery network, the layer of edge servers between users and your service) or a mobile app. If your service is on the receiving end of an arrival process, you have to test it with one.

What the dovetail team actually did

In our illustrative scenario, after the open-loop rerun produced the 340ms p99, the investigation was short. The GC log showed periodic ~200ms pauses. The fix was a small tuning change: they switched the JVM flags and tuned the heap, and the open-loop p99 dropped by about an order of magnitude. The JVM is the Java Virtual Machine, the runtime that executes Java programs; the heap is the region of memory where the program's objects live and that garbage collection cleans up. The closed-loop p99 barely moved. The improvement was only visible because the open-loop test was now measuring queue wait.

The team also added a CI guard. CI stands for continuous integration: an automated step that runs on every change before it merges. The nightly benchmark now runs both models and fails the build if the open-loop p99 exceeds 50ms or if the gap between closed and open p99 exceeds 3x. The second check is the more interesting one: it catches new queueing problems early, even when the absolute number still looks fine.

Run your load test both ways at least once. If the numbers agree, no queue is hiding in the tail. If they disagree, the open-loop number is the one your users will live with.