A generation system can produce many tokens in parallel and still take longer to deliver a useful answer. Initialization, repeated denoising steps, verification, and retries all matter.
Speed claims need a defined measurement boundary.
Understand the alternative
Autoregressive generation produces a sequence through successive prediction steps. Diffusion language models explore a different generative process.
The LLaDA paper describes masked-token prediction through a reverse generation process and reports comparisons under its experimental setup.
That research supports studying the approach. It does not establish the archive’s projected thousands-of-words-per-second targets for an unbuilt local system.
Measure the whole request
Record time from an accepted input to a usable final output. Include model loading if the product routinely pays that cost, or report warm and cold conditions separately.
For an interactive application, also consider when the first useful content appears and whether intermediate output is stable enough to display.
Throughput in a batch and latency for one person are different product metrics.
Match the quality target
A faster method that produces more invalid outputs may require additional attempts. Include those attempts in the cost of an accepted answer.
Use the same task set and scoring policy across candidates. Record output length, constraints, and any post-processing.
A short incomplete response should not win a speed comparison against a complete solution.
Report hardware and concurrency
Memory, accelerator type, precision, batch size, and concurrent requests affect performance. A research-server number cannot be presented as a laptop guarantee.
Include resource use and failure rate under sustained load. A brief ideal run may not represent the service users encounter.
Keep the benchmark script and environment description with the result.
Include token editing in the comparison
The February 2026 LLaDA 2.1 paper adds token-to-token editing to masked-token generation and exposes different speed and quality operating modes. This makes the stopping and revision policy part of the system being measured.
The current LLaDA 2.2-flash model card goes further with block routing and insertion/deletion operations aimed at agent workflows. Its model-card claims are not an independent replication, and its technical-report status should not be confused with established results across every workload.
These systems require more than a tokens-per-second row. Record how many candidate tokens are produced, revised, removed, and finally delivered, and whether tools can consume partial output safely.
Measure time to an accepted result
Use a fixed set of prompts with both short structured outputs and longer reasoning or coding tasks. For each model and decoding policy, record startup, input processing, generation, revision, validation, and any repair attempts. Report the quality threshold before comparing speed.
For a tool request, measure time until a valid, authorized call is available and the tool’s outcome is accepted. A fast stream that repeatedly produces invalid arguments may lose its apparent advantage. For text, distinguish first visible output from text stable enough for the application to show without disruptive revision.
Keep hardware, concurrency, context length, output limits, and cache policy explicit. Compare distributions and failed requests, not just the fastest successful run. The proposed benchmark evaluates a complete serving policy; it does not convert a paper’s throughput claim into a guarantee for another deployment.
Avoid multiplying speculative gains
The original research notes combined proposed advances in tokenization, memory, generation, and hardware into ambitious performance targets.
Benefits measured in separate experiments do not necessarily multiply when combined. They may address the same bottleneck or introduce new overhead.
Integrate one change at a time and measure the complete system.
Choose the product tradeoff
The best method depends on whether the application values quick interaction, batch output, controllability, editing, or exact constraints.
A measured result with a modest gain is useful. A dramatic target can remain a research objective, clearly labeled as such.
The task is complete when the user receives an acceptable answer, not when a benchmark counter records an impressive rate.
Keep a good idea close.
Follow Signal Tower
