(auto)²-research: SoTA on Karpathy's NanoChat Benchmark

If you were suddenly given a large team of grad students to lead, how would you organize them to do research effectively? This question has been at the top of mind for us at rekursiv.ai.

After our ARC-AGI and Sudoku campaigns, we gave the team a new challenge: improve a language model within a five-minute training budget on a single GPU. They reached 0.889824 mean bits per byte (BPB) on Karpathy’s NanoChat benchmark, as shown in the benchmark comparison.

NanoChat benchmark: rekursiv.ai's result averaged across 10 seeds, alongside reported reference scores. Each of our runs used five minutes of training. Lower BPB is better.

This campaign was the first field run of our improved infrastructure, developed specifically for AI scientists. The harness lets agents run experiments, check one another’s results, and pass findings to the next team. While the agents improved the model, we used their research records to improve the harness itself.

Across the search, the agents launched 6,164 experiments. They explored a wide variety of ideas, spanning improvements to training data, architecture, tokenizers, and custom kernels.

Why NanoChat?

NanoChat gave us a short experiment cycle with room to change almost every part of training. A faster kernel could fit more updates into the budget, while a different architecture could learn more from each update. A new tokenizer could change how much text fit into the model’s context. The team had to test each idea in the full training loop to find out whether it helped.

We organized the research into waves of two-hour campaigns. Each wave started with the previous teams’ findings, then explored new ideas in parallel. Figure 2a shows the experiments within each campaign window; Figure 2b shows progress over research time.

Figure 2a. Each point is a scored run within a two-hour campaign window. Colors follow successive research waves across fixed 120-minute intervals. Each time a wave finishes, a new campaign with a team of AI Scientists spawns. Red crosses mark failed or unscored terminal attempts.

Figure 2b. The same runs ordered by completion time. Early on, the AI Scientists launched many experiments to explore a large space of ideas. As time went on, the knowledge graph accumulated a repository of failed experiments, making it more difficult to make quick progress. A phase shift happened, and the agents decided to spend more time to think of more novel ideas before launching new experiments, as shown by the tapering off of experiments per unit time in the plot.

We discovered that, while the challenge did require B200 hardware for the final evaluation, we did not actually need it for the AI Scientist teams to do meaningful science. Therefore, we set out to use H200s as a proxy before transferring over to the more expensive hardware. We spent most of the search on a cluster of H200s over two days, then transferred successful recipes to a single B200 for another day while the H200s continued to run in parallel.

Table 2. A comparison in training time on an H200 required to reach the same training steps as a B200 on a selected experiment. We removed B200 specific operations so the same experiment can run on both sets of hardware.

The search run took about 1.75× longer on an H200 to reach a similar update count and BPB as a B200. However, H200s are typically 2-4x cheaper in cost, which results in substantially more efficient hardware usage.

Walnuts and pears you plant for your heirs (前人種樹，後人乘涼)

At rekursiv.ai, another motto we have is to “speedrun science.” We strive to build the most efficient harness for our agents to collaborate and hill-climb on each other’s results. Sharing a finding is only the beginning. The next scientist needs the fastest way to test what to change. The trigram timeline in Figure 3 shows one example:

From a failed trigram to six new campaigns

Remembering token pairs already helped. Would remembering triples help more? 7 September 2026 · UTC

1 · One projected design fails

The team adds memory for token triples through a narrow learned projection, hoping to extend the earlier bigram gain without a full-width table. The first run loses on seed 42, despite completing almost as many updates as its parent. The evidence worker keeps both the implementation and the negative result: this rejects the tested design, not all trigram memory.

BPB (lower is better): 0.9751 vs parent 0.9686

2 · The lead proposes a different test

The lead reads this result alongside an earlier projected-bigram failure. Perhaps the extra context is useful, but the narrow projection is getting in its way? The next proposal removes that projection and adds full-width trigrams on a new regularized parent, at a new scale. This is a revised recipe, not a projection-only ablation; the earlier failure remains part of the evidence.

New parent mean BPB: 0.96343 · candidate not yet run

3 · Tested code crosses the role boundary

The evidence worker prepares the implementation and checks its causal lookup, zero initialization, and optimizer ownership. The lead accepts the tested code; the integrator then adapts it into the shared package and submits the experiment. The handoff carries source and checks, not just an idea. Passing CPU checks is not a GPU quality result.

Candidate BPB: not yet measured

4 · All three matched seeds improve

After a promising first run, the team retains its result and tests two more seeds against the exact parent. All three improve. The evidence worker reconciles the source, configuration, job receipts, and per-seed results into a panel that another agent can inspect. The evidence is ready, but independent qualification is still pending.

Mean BPB: 0.96343 → 0.95987 · 3 of 3 improve

5 · The coordinator independently qualifies

The coordinator independently checks the original triplet and accepts the result, without backdating that decision to when the evidence became ready. The recipe can now enter the next handoff. The model improvement is supported by matched runs; prior peer-summary exposure and an operational intervention still limit claims about which team organization caused the discovery.

Qualified mean BPB: 0.95987

6 · Six campaigns share the starting recipe

Between 04:48:22 and 04:49:31, six campaigns launch from the shared recipe, with its settings, matched results, and failed history. Every run initializes weights fresh; no trained checkpoint is transferred. The next researchers can ask what improves on this starting point instead of repeating the same search. Choosing and preparing that starting point is the coordinator’s part of the story.

Inherited recipe mean BPB: 0.95987 · not six new results

Figure 3. A revised trigram design became the starting point for six campaigns in less than ninety minutes. The timeline follows the failed experiment, revised implementation, evaluation, independent review, and handoff to the next wave.

The handoff starts with finding the right experiment. In Trackinizer, a scientist can search for its node and follow the edges to related hypotheses, implementations, evidence, and later work. Figure 1 shows these connections across the campaigns. That gives the next researcher a path through the reasoning, not just a stored conclusion. Then the handoff needs to be runnable, not just readable.

Priml provides reusable training and evaluation components. Configgle lets an experiment inherit a named parent configuration and change the relevant components or settings. Together, they let researchers build on a common experimental base rather than reconstruct one another’s training pipelines. The aim is a shorter loop: find the relevant work, understand what happened, test a change, and leave the result for the next scientist.

In the trigram story, the revised component preserved the parent recipe around it; the team checked the implementation and compared matching seeds. A shared base did not remove the need for those checks. The coordinator then packaged the reviewed code, configurations, and findings for six new campaigns, separating inherited progress from each team’s new contribution.

The meta-campaign: improve the process that finds the next result

The coordinator has a second job: improve the campaigns themselves. Scientists change the training recipe; the meta-coordinator examines how they choose experiments, divide work, and pass on results. The handoffs above give it a record of where that process worked and where it got stuck.

Inside the handoff: roles, controls, receipts, and evidence

Several teams divided the work among a lead scientist, an integrator, and an evidence worker. The lead chose experiments and wrote down the intended change. The integrator owned the shared implementation and GPU submissions. The evidence worker checked implementations and results, then reported what they supported. Agents exchanged messages through a Trackinizer room for their campaign, naming the experiment and acknowledging handoffs. A result message could point to a stored audit that the next agent could inspect. Solo researchers performed the same functions themselves.

The records contained more than scores:

• Research intents named the parent recipe, proposed change, prediction, controls, and what the agent would do if the experiment failed.
• Implementation and evidence artifacts recorded source versions, checks, job receipts, raw-log references, per-seed scores, and timing. They kept preliminary results separate from independently reviewed results.
• Experiment records summarized configurations and outcomes. Beliefs stated the conclusions, with links to supporting or contradictory evidence.
• Campaign handoffs collected the current recipe, failed directions, open questions, and results still waiting for review. Study plans and prompt revisions were stored alongside the scientific work.

Those records also guided revisions to the instructions. When one researcher mistakenly ruled out custom components, later briefs made the permission explicit. Other changes clarified deadline decisions, runtime checks, and publication ownership. These were responses to observed problems, not proof that more instructions produce better research.

What changed in the instructions, and what happened afterward

The next wave needed more than the winning recipe. It also needed clearer instructions about what researchers were allowed to try. Earlier in the bigram search, one researcher mistakenly believed it could change configuration but not add components. After an explicit correction, it submitted a hashed-bigram experiment within minutes. The first panel scored 0.97990 BPB against a 0.98543 parent; later experiments reached 0.97110. Subsequent briefs made the permission clear.

• A deadline became a decision, not an automatic shutdown. Some agents interpreted minute 105 as a hard stop. The revised instruction asked whether staging, training, evaluation and recording could finish before the actual deadline. The researcher then admitted four jobs that finished on time. Their results did not improve its incumbent; the benefit was completing useful tests, not a demonstrated score gain.
• Refill instructions changed concrete actions. Instead of merely asking for busy GPUs, prompts required checking reviewed-ready work before a long wait. One campaign also composed compatible recording operations: 43 observed multi-label command lines, versus zero and three in its comparators. It accepted 106 jobs versus 99 and 92, but did not have the best endpoint.
• Runtime tracing caught ineffective experiments. Revised instructions required following a setting from its factory through overrides to the consuming operation. A researcher caught duplicate optimizer ownership and a requested iteration count that did not match the coefficients actually used. It also overturned a false veto. The four newly released experiments all lost to the incumbent: better checking need not immediately lower BPB.
• Publication gained an explicit owner. The designated publisher had to parse-check, atomically publish and read back the exact evidence revision. Moving that role from integrator to analyst coincided with a shorter median publication delay, 187 to 142 seconds. The analyst-publication campaign nevertheless submitted fewer jobs and ended with a worse score.
• Recovery instructions preserved the original operation. Prompts added receipt-first checks and a single owner for stalled submissions. That helped recover blocked work without duplicate submissions, but it did not guarantee timely coordinator service. A later review gap left 35 completed panels unqualified.

Behind 0.88982: the good, the bad, and the beautiful

The score came from a sequence of changes. Some gave the model more useful information; others made that information cheaper to learn from. And several attractive ideas failed once they reached the full training loop.

The good: what moved the score

The largest gains came from training data, model capacity, and learned n-gram memory ^{1 2}. Later kernel and optimizer changes added smaller gains. The recipe below groups the work by what it changed. Open a group for the implementation details.

Data and tokenization: change what the model learns from

• A Unigram16k tokenizer. Trained and integrated a 16,384-token Unigram model, replacing the inherited tokenizer ³.
• A smaller training packing buffer. Reduced the training document-packing buffer from 1,000 to 256 documents, while keeping evaluation packing separately configured.
• The donor-original training-data condition. Replaced the preceding training corpus with the tested donor-original corpus and integrated its frozen prepared rows into the winning recipe.

Capacity and training: spend the budget where it helps

• FFN learning rates adjusted for hidden width. Scaled feed-forward input-projection learning rates by sqrt(4 × model_width / hidden_width) to accommodate unequal layer sizes.
• Higher learning rates for both FFN projections. Applied a 1.25× multiplier to feed-forward input and output matrix learning rates.
• 256-token local attention. Reduced local attention windows to 256 tokens, retaining full-context attention in selected layers.
• A 96-sequence training batch. Selected 96 × 2,048 = 196,608 tokens per optimizer update, following local hardware and quality experiments.
• Batch-adjusted embedding learning rates. Applied square-root batch scaling to token embeddings, the output head, value embeddings, and n-gram tables when moving from 64 to 96 sequences.

Learned n-gram memory: make large tables practical

• Sparse n-gram parameter updates. Implemented RMSProp kernels that update only embedding rows touched by the current batch, avoiding full-table parameter and gradient processing.
• Fused n-gram value mixing. Wrote custom kernels that combine hashed-table lookups, gates, and attention-value additions, replacing separate embedding and tensor operations.
• FP32 n-gram gradient accumulators. Routed embedding-table gradients into persistent FP32 buffers, retaining higher-precision accumulation before optimizer updates.
• One FP32 optimizer moment per embedding row. Replaced elementwise RMSProp second moments with a row-wise mean of squared gradients, greatly reducing optimizer-state size ^{4 5}.
• Eager moment decay with sparse updates. Advanced inactive rows’ second moments every update while keeping parameter updates sparse, replacing the earlier lazy-decay implementation.
• Selective gradient-buffer clearing. Added touched-row bitmaps and a one-warp clearing kernel that zeros marked gradient rows instead of writing across entire accumulator tables.
• One-warp sparse optimizer launches. Tuned the sparse RMSProp implementation to use one warp per program in both its moment sweep and active-row update kernels.

Kernels and loss: fit more work into the training loop

• Saved-LSE cross-entropy backward. Saved per-token log-sum-exp values and recomputed probabilities during backward, reducing the loss function’s retained intermediate state ⁶.
• Known-bound loss stabilization. Used the softcap’s known upper bound to stabilize exponentials, eliminating the usual per-token maximum reduction.
• Custom QK/RoPE kernels for our model. Implemented normalization and rotation using paired rotary halves, targeting redundant loads.
• Single-rounding loss gradients. Combined probability subtraction and upstream-gradient scaling in FP32, rounding once when storing the final gradient.

Combining ideas from separate experiments

We also observed agents taking multiple types of ideas from the knowledge graph (Figure 1) and recombining them in new ways.

Memory and regularization: combine the representation with its training rule

Ideas combined	What happened	BPB improvement
Full-width trigram + regularized bigram memory	Keep learned bigram memory, add an unprojected trigram branch, and reuse selective memory regularization.	0.00356
Longer learning-rate decay + stronger n-gram contributions	Merge a schedule branch with a bigram/trigram scaling branch. Both routes lead to the same combined recipe.	0.00053
Large lexical memory + memory-specific regularization	Give the million-bucket context table its own regularized optimizer group, leaving token embeddings undecayed. Routing and decay change together.	0.00206

Architecture and schedules: make the pieces work together

Ideas combined	What happened	BPB improvement
Earlier attention inputs + batch-aware optimization	Combine attention-source reuse with batch 96 and selected learning-rate scaling.	0.00344
Final-heavy FFN + faster momentum ramp	Allocate more FFN capacity to the final global-attention layer and finish the momentum ramp earlier.	0.00050
Nonzero FFN outputs + stronger readout initialization	Merge two independently qualified initialization changes while preserving their exact random draws.	0.00051
Late-layer FFN allocation + nonzero learning-rate tail	Transfer a nonzero terminal learning rate to a model with redistributed layer capacity.	0.00040
RoPE frequency scale + nonzero learning-rate tail	Combine two separately qualified positional and schedule changes.	0.00043
Shallow-wide capacity + local attention	Combine a five-layer, 640-wide model with local attention while retaining global layers.	0.00260

Execution and arithmetic: test the complete combination

Ideas combined	What happened	BPB improvement
Softcap-bound loss + saved-LSE backward	Use the existing logit bound to remove a maximum reduction; a custom backward recomputes probabilities without a large FP32 tensor. Training and evaluation arithmetic change together.	0.00074
Sparse table updates + scheduled moment aging	Skip untouched table rows while accounting for their optimizer moments. The initial sparse version lowers BPB despite known numerical differences.	0.00160
CUDA graphs + fused QK normalization/RoPE	Layer operator fusion onto graph replay: two distinct execution optimizations.	0.00112

The bad: faster parts did not always make a better model

The training budget rewarded end-to-end progress. A faster isolated kernel, a busier GPU, or a cheaper-looking transition was not enough.

• Optimizer capture and grouped updates: fewer launches did not produce a faster complete step. The tested paths completed fewer updates and scored worse; gradient-buffer and scalar-staging changes were part of the treatment.

• Forward/backward capture: host overhead fell, but the full step remained slower. Removing private-pool memory inflation did not fix it; the remaining slowdown was not isolated.

• FP8 matrix multiplication: isolated kernels showed 1.12–2.05× BF16 throughput, but those timings excluded quantization. They did not establish a faster training loop or better BPB.

• Growing depth during training: the transition consumed scarce training time and changed the learning trajectory. Completed variants lost to the full-depth control. Subtracting the transition afterward cannot recover the validation score of a run that never happened.

The beautiful: the search changed our intuitions

Some of the largest improvements came from simple changes. Introducing and scaling learned bigram memory lowered one campaign’s mean from 0.98840 to 0.96913 BPB. On a matched final-hardware recipe, expanding the training supply from seven to fourteen shards lowered mean BPB from 0.96140 to 0.92088, with the backend, width, and batch held fixed. More useful information could matter more than another kernel tweak.

Where capacity went mattered too. Redistributing existing FFN channels toward the final global-attention layer helped where increasing every layer’s capacity had not paid for its cost. And two good ingredients did not have to be additive to be worth combining: RoPE ⁷ and learning-rate-tail changes beat both standalone recipes, but fell short of their additive prediction.

The surprises also changed what we needed to check. Light data filtering ⁸ helped on a search recipe but hurt after transfer; donor-original text beat the rewritten alternative in the tested final-hardware comparison.

Keep humans in the loop

We humans set study priorities, including team size and role organization, and could pause the search. In a review of 48 launched campaigns, 35 completed panels were found waiting for qualification after a service interruption. The same records that enabled the trigram handoff exposed a limit: evidence needs someone responsible for checking and passing it on. The coordinator’s own review and handoff process had to be part of what we improved.

Team size, context refresh, and inherited progress

The trigram team accepted 65 jobs, while a solo researcher accepted 112 in the same wave. The team nevertheless found the stronger recipe. In a later comparison, a solo researcher had the best incremental improvement and final score. Neither job count nor team size told us who would make the next useful discovery.

• A two-researcher team reached the original cohort’s 0.95610 qualified endpoint, while several larger teams cost more without consistently reaching better endpoints.
• Fifteen of sixteen researchers refreshed every thirty minutes still compacted. Persistent researchers also continued producing useful experiments after compaction.
• In one retrieval-policy cohort, inherited-recipe availability explained 97.79–100% of control-relative improvement area. Earlier review was not the same thing as discovering a better recipe.

Two early branches starting from the same packet reached qualified scores of 0.965700 and 0.985567. Learned bigram memory opened a larger gain than nearby optimizer refinements. With only two runs per model, different provider stacks, and human interventions, this did not establish a general ranking of models or team structures.

Where completed work waited before the next team could use it

Completed experiments could still wait a long time before other teams could use them. One candidate’s last required run finished 49.7 minutes before qualification. The timestamps did not separate active review from publication or service wait, so we recorded completion and qualification as different events.

GPU utilization told only part of this story. Median accepted-run turnaround was about 6.00–6.37 minutes across one set of workflow comparisons, while allocated GPU fractions ranged from 8.80% to 68.14%. We did not measure whether a useful, valid candidate was ready in every gap, so we could not count all unallocated time as avoidable idle time.

What we learned

NanoChat gave us a score. It also gave us a concrete example of the research system we want to build: one where a failed experiment can become another scientist’s starting point, and a useful discovery can travel across teams without everyone rebuilding the work behind it.

That is our vision for rekursiv.ai: AI scientists that accumulate knowledge, run the next experiment, and help us improve how research gets done. Speedrunning science means shortening the path from an idea to a result we can trust, then making that result useful to whoever comes next.

We are building the tools for that loop in the open. Try Trackinizer, Priml, and Configgle in your own research. Bring us a hard problem, a missing capability, or an experiment that failed. Join our Discord and let’s together build what the next scientist will inherit.

References