The strongest part of the revised House ASI roadmap was its link between a criticism and a proposed repair. Hash-based retrieval needed a semantic alternative. Repeated candidates needed genuine diversity. A verifier that accepted its own assumptions needed independent checks.
The earlier roadmap was less disciplined. It treated ambitious schedules, multiplied efficiency factors, and an unverified competitive lead as established advantages. A more useful plan makes progress conditional on evidence.
Start with the smallest reproducible baseline
Package the current system so another person can run it against a fixed set of tasks. Record versions, prompts, dependencies, seeds where applicable, and resource limits. Preserve complete outputs and failures.
Run a simple model with the same task access. The pipeline must justify its additional complexity against that baseline. A component count is not a capability measurement.
Repair retrieval without discarding the evidence
Replace the hash-derived vectors with a retrieval method suited to semantic search, then compare both on representative queries.
One proposed correction in the archive went too far: deleting original text and letting an embedding carry the information. An embedding is not a reversible archive of the source. Preserve the source document, its provenance, and a stable identifier; use the embedding to help find it.
Test irrelevant queries, stale documents, contradictory sources, and permission boundaries. A system that retrieves something for every question is not necessarily retrieving useful evidence.
Separate experimental factors
If the proposal combines a new tokenizer with a new generation architecture, compare each factor independently and in combination. Keep data and compute accounting explicit.
Token-level perplexity is difficult to compare directly across different tokenizers because the units changed. Include measures normalized to a common representation and downstream tasks with consistent scoring.
Small pilot runs can expose broken assumptions. They do not establish that an advantage will grow with model size. Scaling remains a separate experiment.
Make gates capable of stopping the project
Each phase needs a measurable result, a budget, and a disposition when it fails. A missed threshold should trigger a decision, not a rewritten success story.
Examples include rejecting a memory module that adds latency without improving accepted answers, or keeping an autoregressive baseline when a diffusion variant does not meet the workload’s quality target.
Targets such as phone deployment, many-language support, or long-term autonomous operation require their own evidence. They should not inherit a pass from an unrelated math benchmark.
Update the tokenizer experiment without skipping ablations
The revised August 2026 paper Faster Superword Tokenization focuses on making cross-word tokenizer training more efficient through candidate aggregation and staged merging. Faster construction of a tokenizer is a different claim from faster model training or better downstream reasoning.
A useful pilot compares the existing tokenizer and the proposed one on the same source corpus. Measure tokenizer construction time, peak memory, encoded length, vocabulary cost, and downstream task quality. Evaluate unusual identifiers, code, numbers, and each intended language instead of relying only on average English compression.
If model training changes at the same time, retain a factorial comparison so the contribution of each change remains identifiable.
Put prompt optimization behind a fixed gate
GEPA, revised in February 2026, uses reflective feedback on execution trajectories to evolve prompts. It is an optimization method for the agent’s instructions, not evidence that the model’s weights learned from a local memory cache.
A roadmap can test such a method on a development task set while keeping the final evaluation sealed. Fix the allowed tools, budget, scoring rules, and data access before optimization. Record every candidate evaluated, including failed ones.
Stop when the budget is exhausted, improvement plateaus under the declared rule, or a required regression fails. Choose the candidate using development evidence, then run the held-out evaluation once for the reported decision. Repeatedly tuning against that result turns it into development data and requires a new test set.
The milestone is an improvement with a documented boundary, not simply another iteration that reports a higher internal score.
Treat novelty as a research claim
A literature search can say what was found within a stated scope and date. It rarely establishes that no one else has combined a set of ideas. Nor does an unusual combination guarantee a useful one.
Describe the hypothesis and publish the ablation results. Avoid claiming months of competitive advantage from the absence of a matching title in a transcript collection.
The next milestone is a result that survives reproduction. A roadmap earns confidence by making that result easier to inspect, including when the honest outcome is that the simpler system works better.
Keep a good idea close.
Follow Signal Tower
