Open-Loop or Closed-Loop: Which LLM Serving Benchmark Do You Need?

An LLM serving benchmark can look rigorous while answering the wrong question. A test that keeps a fixed number of requests active tells you how the system behaves under sustained concurrency. A test that sends requests according to an independent arrival schedule tells you whether the service can absorb demand without an expanding queue. These are not interchangeable experiments.

The practical answer is simple: use a closed-loop test when you want to measure capacity under controlled concurrency, and use an open-loop test when you want to study user-facing behavior under an external request rate. For a serious serving evaluation, run both and explain which question each result answers.

What changes between closed-loop and open-loop tests?

In a closed-loop benchmark, each simulated client waits for a response before submitting another request. The workload therefore slows down when the server slows down. Fixed-concurrency tools often use this model because it is repeatable and useful for finding how throughput changes as more requests compete for compute and memory.

This feedback loop is also the model’s main limitation. A saturated service completes requests more slowly, so the clients generate new work less frequently. The offered load automatically falls toward the service’s completion rate. That behavior can hide the queue growth that would occur if real arrivals continued independently.

In an open-loop benchmark, the generator schedules arrivals without waiting for earlier requests to finish. If demand exceeds sustainable capacity, the queue grows and latency deteriorates. This model is better suited to questions such as whether a service can meet a latency objective during a known traffic pattern. However, the generator itself must keep accurate time and avoid becoming the bottleneck.

Neither model is inherently more truthful. Each controls a different variable: closed-loop testing controls active concurrency, while open-loop testing controls offered arrival rate.

Start with the decision, not the metric

Choose the workload from the operational decision you need to make.

• Capacity exploration: Use stepped closed-loop concurrency to identify where throughput stops improving and latency begins rising sharply.
• Admission-control planning: Use open-loop arrivals to observe queue growth, rejection behavior, and recovery after demand exceeds sustainable service capacity.
• Interactive experience: Measure time to first token and inter-token latency separately. A favorable end-to-end average can conceal a slow initial response or uneven token delivery.
• Batch processing: Emphasize completed output tokens per unit of time, total completion time, and resource use, while still reporting the input and output length distribution.
• Mixed traffic: Separate short interactive prompts from long generation jobs. Otherwise, changes in the request mix can be mistaken for changes in serving efficiency.

A benchmark should also define its stopping condition. A short run may capture warm caches or startup effects rather than steady behavior. A long run can expose thermal throttling, memory pressure, queue accumulation, or worker instability. Report warm-up treatment, measurement duration, request count, and whether unfinished requests were included or discarded.

Why heterogeneous distributed inference needs extra controls

Distributed serving adds variables that a single-device benchmark can ignore. A result is interpretable only when it names the hardware, topology, model, precision, and test conditions. Hardware means more than an accelerator label: report relevant CPU, GPU or other processor roles, available memory, and which devices performed each stage. Topology should identify whether workers shared one physical host, used distinct computers on a local network, or communicated across a wider network.

Do not describe a result as multi-computer validation merely because several processes or devices appeared in the test. That claim requires distinct physical hosts. Likewise, do not claim WAN validation unless traffic actually crossed the stated wide-area topology.

Record the exact model identity, model revision when available, precision or quantization format, prompt-length distribution, requested output lengths, decoding settings, batching policy, concurrency or arrival process, network conditions, and software versions. In heterogeneous systems, also document route selection and whether the route stayed fixed during comparison. These controls prevent a scheduler change, workload change, and hardware change from being compressed into one unexplained number.

Mycellios is an experimental platform and physical runtime for connecting heterogeneous computers and mobile devices into routes for LLM inference. Its stated scope includes heterogeneous hardware planning and scheduling, voluntary desktop and browser participation, OpenAI-compatible inference access, and reproducible benchmarking and validation. Those capabilities define useful areas to evaluate; they do not, by themselves, establish a performance result. Any Mycellios benchmark still needs the complete conditions above.

Report distributions and overload behavior

Throughput and latency are coupled. Increasing batching or concurrency may improve total token throughput while delaying individual requests. Report both rather than presenting the better-looking metric alone.

For latency, include a distribution rather than only an average. At minimum, distinguish time to first token, inter-token latency or token cadence, and end-to-end request latency. For throughput, separate requests completed from input and output tokens processed. State whether queued, rejected, timed-out, cancelled, and failed requests contribute to the calculations.

An open-loop result is incomplete without the offered load. A closed-loop result is incomplete without the concurrency. In either case, show whether the run reached a stable state. If arrivals continued faster than completions, throughput may look flat while waiting time grows without bound. That is overload, not stable capacity.

Every figure should be labeled as measured, simulated, or theoretical. Measured figures come from the named physical test setup. Simulated figures depend on a model and its assumptions. Theoretical figures are bounds or calculations, not observed serving performance. Do not blend these categories in one comparison without explicit labels.

A defensible two-pass benchmark

1. Define the request corpus. Preserve prompt and output-length distributions, decoding settings, and model identity across both passes.
2. Run a closed-loop sweep. Increase concurrency in controlled steps and observe throughput, latency distributions, errors, resource pressure, and route stability.
3. Select open-loop arrival levels. Base them on the region found in the first pass, but identify them as test inputs rather than promised capacity.
4. Run long enough to observe queues. Track arrivals, starts, completions, rejections, cancellations, and outstanding requests over time.
5. Repeat and disclose variance. Keep conditions fixed, report run-to-run spread, and investigate outliers instead of silently removing them.
6. Publish the boundary. State what the experiment did not validate, especially physical-host count, network scope, workload diversity, and duration.

Finally, performance validation does not establish privacy. Transport encryption can protect data in transit, but it does not make activations private from a node that processes them. Any distributed benchmark involving external workers should describe its threat model and data handling separately from speed measurements.

The most useful benchmark is not the one with the largest throughput value. It is the one whose load model matches the decision, whose queue behavior is visible, and whose hardware and test conditions allow another engineer to understand what was actually measured.