The archived experiment reported 96 correct answers for a QED-based pipeline and 95 for a Claude baseline. The most important part of that sentence is the denominator: one hundred selected problems.
MATH-500 contains 500 problems. Running a subset can be useful, but its result should be named as a subset result. It does not establish a full MATH-500 score.
Name the systems being compared
The draft describes an eleven-stage pipeline with multiple candidates, a reasoning cache, and verification. Its baseline used a single model call. That can be a legitimate comparison between two practical systems. It is not an isolated comparison of their underlying model weights.
Publish the selected problem IDs and selection method. Record every model actually called, including any external verifier or fallback. A pipeline should be called fully local only when its complete execution path supports that description.
The archived figures are reported observations. This rewrite does not claim a fresh benchmark run, a complete audit of the original result files, or a current ranking.
One answer does not settle the question
A one-answer difference in a hundred questions is small enough that individual grading decisions matter. The draft itself identifies answer-formatting failures, including equivalent fraction notation and interval extraction.
Keep separate counts for mathematically wrong answers, equivalent answers rejected by the parser, empty or truncated responses, and execution failures. Report an end-to-end score using a rule fixed in advance, then explain the error categories. Do not silently remove inconvenient failures after looking at the result.
For a paired comparison, show which exact questions each system alone solved. Aggregate totals conceal the disagreement pattern needed to assess the difference.
A deterministic parser has a narrow job
A parser can consistently extract the same string from the same output. That does not make model generation deterministic. Nor does it prove that the extracted expression is mathematically equivalent to the reference.
Test the scorer on equivalent notation and deliberately incorrect near matches before evaluating candidates. Preserve raw answers alongside normalized ones. If the task asks for a proof, an exact final number is only part of the requirement.
A grading correction should create a new result version and explain what changed.
Include the cost of getting an answer
The draft reports longer latency for the pipeline. That tradeoff belongs beside accuracy. Count all candidates, retries, verification calls, failed runs, and human review.
Owned hardware still consumes electricity and maintenance effort. API spending is only one line in the budget. A fair report can show both marginal cash expense and broader operating cost without pretending they are interchangeable.
Report paired uncertainty before ranking systems
Anthropic’s January 2026 agent-evaluation guidance distinguishes capability on at least one attempt from reliability across repeated attempts. State which question a new experiment asks before choosing its metric. A system given many candidates has a different opportunity to succeed from a single-call baseline.
For the archived 96-versus-95 result, the totals alone do not identify the paired disagreement. If both systems answered nearly the same questions correctly, the interpretation differs from a case with many offsetting wins and losses.
A new report should retain a row per problem and trial: correctness, failure class, total cost, and elapsed time for both systems. Estimate uncertainty in the paired difference using an appropriate paired method. When several trials share one problem, keep that grouping in the analysis rather than treating every response as an unrelated new problem.
Audit disputed scores without changing the rules mid-run
OpenAI’s July 2026 coding-benchmark audit shows why evaluators themselves deserve inspection. Its specific findings concern coding tasks, but they motivate a separate check here: can the math scorer distinguish equivalent answers from near misses?
Freeze the scorer before running the comparison. Retain raw output and normalized answers. If a genuine scoring defect is discovered later, publish both the original and corrected result versions and apply the correction symmetrically.
Keep problem selection separate from evaluation. Do not replace failed questions with easier ones or select the subset after inspecting outputs. None of these proposed methods changes the historical 96 and 95 counts. They define the evidence a fresh comparison would need before supporting a broader claim.
The useful next experiment
Freeze a representative held-out set. Run repeated trials with documented resource limits. Compare the complete pipeline with its simpler baseline and with versions that remove individual stages.
If the improvement persists, those ablations can identify what earned it. If it disappears, the experiment still did useful work: it prevented a promising pilot from turning into an unsupported production claim.
Keep a good idea close.
Follow Signal Tower
