We built a custom harness that does auto-autoresearch, which enables a swarm of agents to build on each other's discoveries, preserve failed experiments, and improve how they do research. This let us improve on the previous NanoChat state of the art in three days (source code). Our work is built on top of trackinizer, sagent, priml, and configgle.
Figure 1. The 15,099-node knowledge graph built by our agent teams in three days. Nodes appear in creation order, connecting ideas, experiments, and findings.
If someone suddenly gave you large team of grad students, how would you organize them to do research effectively? This question has been on our minds at rekursiv.ai.
AI agents function in many similar ways to “intern savants”: they possess genuis-level knowledge, but they don’t know what’s important. Instead of approaching the problem head on, we pondered: “what if we could get our intern savants to figure out the best strategy for us?”.
After our ARC-AGI and Sudoku campaigns, we gave our team of AI Scientists a new challenge: improve a language model within a five-minute training budget on a single B200. The latest recipe reaches 0.887791 mean bits per byte (BPB) across 10 seeds on Karpathy’s NanoChat benchmark, as shown in the benchmark comparison.
Bits per byte (BPB) ↓ · lower is better
Figure 2. NanoChat benchmark: rekursiv.ai's experimental result averaged across 10 seeds, alongside reported reference scores. Each of our runs used 300 seconds (five minutes) of charged training on one B200. Lower BPB is better.
In three days, the agents launched 6,164 experiments. They explored a wide variety of ideas, spanning improvements to training data, architecture, tokenizers, and custom triton kernels. Those three days weren’t spent only improving NanoChat. The system was also learning how to do research: revising instructions, repairing handoffs, and changing how agents tested and built on one another’s ideas. We gave explicit directives to our system to devise new strategies to cover for its weaknesses and do research faster on each iteration.
This was enabled in large part because of Trackinizer, our open-source custom graph database that tracks the lineage of beliefs, and experiments, and analyses. Over time, the AI research team wove their rich tapestry of experimental ideas and results into the graph to produce their final results.
The final recipe is open source on GitHub. We encourage you to explore our tools on GitHub, or join our Discord for further discussion.
NanoChat gave us a short experiment cycle with room to change almost every part of training. A faster kernel could fit more updates into the budget, while a different architecture could learn more from each update. A new tokenizer could change how much text fit into the model’s context. The team had to test each idea in the full training loop to find out whether it helped.
We approached the problem using waves of campaigns on top of our meta-harness, each research wave lasting 2 hours long. When a wave completed, it would restart with a new harness proposed by the previous waves. Research teams also passed important findings and experimental results to the next teams to explore in parallel. Figure 3a shows the progress made across the waves, while Figure 3b shows the total wall-clock progress.
Figure 3a. A scatterplot of our research campaigns over time. A point represents each 5-minute experiment on Nanochat, and lines represent the pareto frontier of each campaign. The color gradient visually indicates wall-clock time, showing how restarted campaigns pick up where previous campaigns left off.
Figure 3b. The same runs ordered by completion time. Early on, the AI Scientists launched many experiments to explore a large space of ideas. As time went on, the knowledge graph accumulated a repository of failed experiments, making it more difficult to make quick progress. A phase shift happened, and the agents decided to spend more time to think of more novel ideas before launching new experiments, as shown by the tapering off of experiments per unit time in the plot.
We found that, while the challenge did require B200 hardware for the final evaluation, Hopper GPUs could work just as well. Therefore, we set out to use H200s as a proxy for the majority of the research campaigns before transferring over to the more expensive hardware. Two days were spent on a cluster of H200s over two and a half days, then transferred another half day training on a single B200.
At rekursiv.ai, another motto we have is to “speedrun science.” We strive to build the most efficient harness for our agents to collaborate and hill-climb on each other’s results. Sharing a finding is only the beginning. The next scientist needs to understand the test and choose what to change. The trigram timeline in Figure 4 shows one example:
Remembering token pairs already helped. Would remembering triples help more? 7 September 2026 · UTC
The team adds memory for token triples through a narrow learned projection, hoping to extend the earlier bigram gain without a full-width table. The first run loses on seed 42, despite completing almost as many updates as its parent. The evidence worker keeps both the implementation and the negative result: this rejects the tested design, not all trigram memory.
BPB (lower is better): 0.9751 vs parent 0.9686The lead reads this result alongside an earlier projected-bigram failure. Perhaps the extra context is useful, but the narrow projection is getting in its way? The next proposal removes that projection and adds full-width trigrams on a new regularized parent, at a new scale. This is a revised recipe, not a projection-only ablation; the earlier failure remains part of the evidence.
New parent mean BPB: 0.96343 · candidate not yet runThe evidence worker prepares the implementation and checks its causal lookup, zero initialization, and optimizer ownership. The lead accepts the tested code; the integrator then adapts it into the shared package and submits the experiment. The handoff carries source and checks, not just an idea. Passing CPU checks is not a GPU quality result.
Candidate BPB: not yet measuredAfter a promising first run, the team retains its result and tests two more seeds against the exact parent. All three improve. The evidence worker reconciles the source, configuration, job receipts, and per-seed results into a panel that another agent can inspect. The evidence is ready, but independent qualification is still pending.
Mean BPB: 0.96343 → 0.95987 · 3 of 3 improveThe coordinator independently checks the original triplet and accepts the result, without backdating that decision to when the evidence became ready. The recipe can now enter the next handoff. The model improvement is supported by matched runs; prior peer-summary exposure and an operational intervention still limit claims about which team organization caused the discovery.
Qualified mean BPB: 0.95987Between 04:48:22 and 04:49:31, six campaigns launch from the shared recipe, with its settings, matched results, and failed history. Every run initializes weights fresh; no trained checkpoint is transferred. The next researchers can ask what improves on this starting point instead of repeating the same search. Choosing and preparing that starting point is the coordinator’s part of the story.
Inherited recipe mean BPB: 0.95987 · not six new resultsFigure 4. A revised trigram design became the starting point for six campaigns in less than ninety minutes. The timeline follows the failed experiment, revised implementation, evaluation, independent review, and handoff to the next wave.
The handoff starts with finding the right experiment. In Trackinizer, a scientist can search for its node and follow the edges to related hypotheses, implementations, evidence, and later work. Figure 5 shows these connections across the campaigns. The next researcher can trace why an experiment was run and what its result supports.
Figure 5. A subset of the knowledge graph built by the agent teams. Explore the connections between their ideas, experiments, and findings.
Then the handoff needs to be runnable, not just readable.
To facilitate reliable ML science, we built Priml, a composable ML A/B test harness with reusable training and evaluation components. It’s all powered by Configgle, which lets an experiment inherit a named parent configuration and change the relevant components or settings. Together, they let human and AI researchers build on a common experimental base rather than reconstruct one another’s training pipelines. We want the next scientist to be able to find the evidence, reproduce the setup, and test a change without rebuilding the work behind it.
In the trigram story, the revised component preserved the parent recipe around it; the team checked the implementation and compared matching seeds. A shared base did not remove the need for those checks. The coordinator then packaged the reviewed code, configurations, and findings for six new campaigns, separating inherited progress from each team’s new contribution.
The AI meta-coordinator also changed how the campaigns worked. In one review, an AI reviewer found that engineers were told to return results to researchers whose sessions might already have ended. The coordinator revised the prompts to route late results through the supervisor and durable records. Research history became input to the next team’s instructions, not just its training recipe.
Several teams divided the work among a lead scientist, an integrator, and an evidence worker. The lead chose experiments and wrote down the intended change. The integrator owned the shared implementation and GPU submissions. The evidence worker checked implementations and results, then reported what they supported. Agents exchanged messages through a Trackinizer room for their campaign, naming the experiment and acknowledging handoffs. A result message could point to a stored audit that the next agent could inspect. Solo researchers performed the same functions themselves.
The records contained more than scores:
Those records also guided revisions to the instructions. When one researcher mistakenly ruled out custom components, later briefs made the permission explicit. Other changes clarified deadline decisions, runtime checks, and publication ownership. We could see which problems the new instructions addressed, but that alone did not tell us whether they produced better research.
The next wave needed more than the winning recipe. It also needed clearer instructions about what researchers were allowed to try. Earlier in the bigram search, one researcher mistakenly believed it could change configuration but not add components. After an explicit correction, it submitted a hashed-bigram experiment within minutes. The first panel scored 0.97990 BPB against a 0.98543 parent; later experiments reached 0.97110. Subsequent briefs made the permission clear.
Improving the instructions did not remove the need for human oversight. We set study priorities, including team size and role organization, and could pause the search. In a review of 48 launched campaigns, 35 completed panels were found waiting for qualification after a service interruption. The same records that enabled the trigram handoff exposed a limit: evidence needs someone responsible for checking and passing it on. The coordinator’s own review and handoff process had to be part of what we improved.
The trigram team accepted 65 jobs, while a solo researcher accepted 112 in the same wave. The team nevertheless found the stronger recipe. In a later comparison, a solo researcher had the best incremental improvement and final score. Neither job count nor team size told us who would make the next useful discovery.
Two early branches starting from the same packet reached qualified scores of 0.965700 and 0.985567. Learned bigram memory opened a larger gain than nearby optimizer refinements. With only two runs per model, different provider stacks, and human interventions, this did not establish a general ranking of models or team structures.
Completed experiments could still wait a long time before other teams could use them. One candidate’s last required run finished 49.7 minutes before qualification. The timestamps did not separate active review from publication or service wait, so we recorded completion and qualification as different events.
GPU utilization told only part of this story. Median accepted-run turnaround was about 6.00–6.37 minutes across one set of workflow comparisons, while allocated GPU fractions ranged from 8.80% to 68.14%. We did not measure whether a useful, valid candidate was ready in every gap, so we could not count all unallocated time as avoidable idle time.
All of this comes back to a question: What should we test next, and what would make us change our mind? Keeping that question useful gets harder as experiments accumulate and researchers hand work to one another. The meta-campaign worked on those handoffs and instructions. In NanoChat, the decisions looked like this.
There was no single discovery that made the final recipe. Across data, capacity, memory, and execution, the team had to decide which observation deserved another test and which result called for a different approach. The map opens the evidence behind those decisions, including what failed.
Four places the search changed the recipe. Choose a topic to explore what worked, what failed, and what surprised us.
Combined recipe · tested end to endUseful changes had to work together inside the training budget.
Figure 6. Topics in the search, not shares of the BPB gain. Equal-sized cards group related experiments; not every successful experiment became part of the final recipe. Stopped branches refer to the tested designs.
The B200 ran more updates but scored worse. Comparing learning curves, the scientist suspected repetition of the small training corpus, though it first explored wider models. We prioritized testing more data. The coordinator prepared extra shards; the scientist assigned a worker to exclude the held-out shard from training. With backend, width, and batch fixed, the paired comparison doubled training supply from seven to fourteen shards and took BPB from 0.96140 to 0.92088. Before asking how to fit even more updates into the budget, we needed to check what those updates were learning from.
The good: more useful training material. The seven-to-fourteen-shard comparison changed data supply and exposure while keeping the backend, width, and batch fixed. Simply increasing the shard count would have included validation shard 7, so the worker added explicit training-shard selection and checks that training and validation shards did not overlap. The gain supported trying more data; it did not isolate repetition from corpus composition and exposure. Donor-original text also beat the rewritten alternative in the tested final-hardware comparison.
The bad: a filter did not travel well. Light filtering 1 helped on an H200 search recipe but hurt after transfer to the B200 recipe. This was a reason to retest the treatment, not evidence that filtering always helps or always hurts.
The beautiful: the input pipeline was part of the model search. More training material delivered a large gain without a new architecture. A new tokenizer also changed how much text fit in a context, so its score could not be read as a clean tokenizer-algorithm comparison.
Data choices in the final recipe:
In the historical tokenizer panels, substitutions changed evaluation byte coverage as well as training inputs. Those panels should not be treated as comparisons over identical text exposure.
In five minutes, a bigger model has to earn back its cost. When uniform growth did not help, the next question was where capacity would pay. Moving feed-forward capacity toward the final global-attention layer helped where uniform growth had not.
Starting shallow and adding depth during training offered another way to spend the budget. But completed variants lost to the full-depth control: the transition itself took training time and changed the learning trajectory. The comparison had to include the transition, not just the model before and after growth.
The good: put capacity where it pays. Redistributing existing feed-forward channels toward the final global-attention layer improved results. Combining that allocation with an earlier momentum ramp beat both standalone changes on every seed in its three-seed panel.
The bad: growing a model was not free. Completed depth-growth variants lost to the full-depth control. The transition took training time and changed the learning trajectory. Subtracting its duration afterward could not recover the validation score of a run that never happened.
The beautiful: two wins need not add up to be worth combining. RoPE frequency 3 and learning-rate-tail changes beat both standalone recipes, but delivered less than their additive prediction. The useful question was whether the combination won, not whether each ingredient kept its original gain.
Capacity and training choices in the final recipe:
Three-seed means against the named comparator. These overlapping panels come from different stages of the search; their gains are not additive. Hardware, training budgets, and evaluation contexts vary across panels; these comparisons are not slices of the final score.
| Ideas combined | What happened | Comparator → combination BPB |
|---|---|---|
| Earlier attention inputs + batch-aware optimization | Combine attention-source reuse with batch 96 and selected learning-rate scaling. The combination beats both constituent recipes on all three seeds. | 0.90370 → 0.90026; Better batch/LR constituent |
| Final-heavy FFN + faster momentum ramp | Allocate more FFN capacity to the final global-attention layer and finish the momentum ramp earlier. It beats both standalone changes on every seed. | 0.95803 → 0.95753; Momentum-only constituent |
| Nonzero FFN outputs + stronger readout initialization | Merge two independently qualified initialization changes while preserving their exact random draws. The combination beats both component panels on all three seeds. | 0.93944 → 0.93893; Better FFN constituent |
| Late-layer FFN allocation + nonzero learning-rate tail | Transfer a nonzero terminal learning rate to a model with redistributed layer capacity. The mean gain is small but all three parent pairs improve. | 0.95710 → 0.95670; Late-FFN parent |
| RoPE frequency scale + nonzero learning-rate tail | Combine two separately qualified positional and schedule changes. It beats both components, although the observed improvement is less than additive. | 0.95830 → 0.95787; Better schedule constituent |
| Shallow-wide capacity + local attention | Combine a five-layer, 640-wide model with local attention while retaining global layers. Rows-per-pass is held fixed against the direct parent. | 0.99380 → 0.99120; Shallow-wide parent |
Learned n-gram memory adds lookup tables keyed by short token sequences, giving the model another way to use local context 4 5. Introducing and scaling bigram memory lowered one campaign’s mean from 0.98840 to 0.96913 BPB. Yet the first projected trigram made things worse. Was remembering triples a bad idea, or was this a bad way to use them?
The team tried a full-width design on a regularized parent, with a different scale. All three matched seeds improved. That kept the direction open without making the earlier failure disappear, or proving which design change mattered.
The good: simple memory made a large difference. Bigram memory’s 0.98840 → 0.96913 comparison covered both its introduction and scaling. Later, full-width trigram memory on a regularized bigram parent improved all three matched seeds, from 0.96343 to 0.95987 mean BPB.
The bad: the first projected trigram lost. It scored 0.9751 against a 0.9686 parent on the tested seed. That rejected implementation stayed in the record. It did not establish that trigram memory could never help.
The beautiful: revisiting the failure meant changing the experiment. The successful retry changed projection, regularization, and scale together; it did not isolate projection as the cause. The team could build on the failed test without pretending that the new one answered exactly the same question.
Making the memory practical in the final recipe:
Sparse updates saved full-table work, but optimizer moments still needed attention between visits to a row. The initial sparse version improved scores despite known arithmetic differences: a performance win was not a proof of numerical equivalence.
Three-seed means against the named comparator. These overlapping panels come from different stages of the search; their gains are not additive. Hardware, training budgets, and evaluation contexts vary across panels; these comparisons are not slices of the final score.
| Ideas combined | What happened | Comparator → combination BPB |
|---|---|---|
| Full-width trigram + regularized bigram memory | Keep learned bigram memory, add an unprojected trigram branch, and reuse selective memory regularization. All three exact-parent comparisons improve. | 0.96343 → 0.95987; Regularized-memory parent |
| Longer learning-rate decay + stronger n-gram contributions | Merge a schedule branch with a bigram/trigram scaling branch. Both routes lead to the same combined recipe, which beats both branches on every seed. | 0.95723 → 0.95670; Schedule branch |
| Large lexical memory + memory-specific regularization | Give the million-bucket context table its own regularized optimizer group, leaving token embeddings undecayed. Routing and decay change together. | 0.96913 → 0.96707; Plain-memory parent |
Before spending a training run on an optimization, the team could ask a cheaper question: does it change the work the machine actually does? One worker proposed rewriting a square as a multiplication to help the compiler fuse operations. The coordinator checked: the compiler already made that simplification. The worker withdrew the recommendation to screen it as a speed treatment and turned to inspecting the generated code.
Even a measured operator speedup needed another check. FP8’s isolated gains excluded quantization; the full training loop still had to benefit.
The good: optimize the work the recipe actually does. Saved-LSE loss backward reduced retained intermediate state; using the softcap’s known bound removed a maximum reduction. Their combination improved a three-seed mean from 0.89053 to 0.88979 on its eager-moment parent. This is a historical comparison, not the final ten-seed headline result.
The bad: faster parts were not enough.
The masked optimizer took 0.58–0.62 ms against its parent’s roughly 0.24 ms in the small probe. That justified declining this implementation, not rejecting sparse optimizers as a direction. Other apparent speedups needed checks beyond the isolated operation:
The beautiful: a speed change could also be a numerical change. Fused operations changed arithmetic ordering as well as execution cost. The combined recipes could improve without proving that their output was numerically equivalent or that extra updates explained the whole gain.
From the research room · check the generated code
9 September · excerpts from the session logs
Scientist: “Root’s note stands: inductor already lowers pow(2) to mul, so state what your fuse descendant actually changes in generated code.”
Worker: “I read the ‘pow’ token in the kernel name as proof of a pow instruction when it is only proof of a pow node in the graph.”
In the same reply: “It should NOT be screened as a speed treatment.”
The proposal already allowed for a small win or none. The correction changed what the team would check next; it did not prove that every proposed rewrite had zero effect.
Source: hosted Trackinizer AgentSession 19495, records 1570, 1861, 1939, and 1989; 9 September 2026, 05:53–06:23 UTC.
Kernel and loss choices in the final recipe:
Three-seed means against the named comparator. These overlapping panels come from different stages of the search; their gains are not additive. Hardware, training budgets, and evaluation contexts vary across panels; these comparisons are not slices of the final score.
| Ideas combined | What happened | Comparator → combination BPB |
|---|---|---|
| Softcap-bound loss + saved-LSE backward | Use the existing logit bound to remove a maximum reduction; a custom backward recomputes probabilities without a large FP32 tensor. Training and evaluation arithmetic change together. | 0.89053 → 0.88979; Eager-moment parent |
| Sparse table updates + scheduled moment aging | Skip untouched table rows while accounting for their optimizer moments. The initial sparse version lowers BPB despite known numerical differences; this is not an equivalence claim. | 0.89219 → 0.89059; Dense-clear parent |
| CUDA graphs + fused QK normalization/RoPE | Layer operator fusion onto graph replay: two distinct execution optimizations. All three seeds improve, but extra updates and numerical ordering are not isolated. | 0.89952 → 0.89840; Graph-enabled parent |
A useful result also changed what was worth testing next. Once final-layer feed-forward allocation and an earlier momentum ramp had each helped, the team tested them together. The combination beat both standalone changes on every seed in its three-seed panel. Other combinations gained less than the sum of their parts. The next experiment had to test the combination, not assume the isolated wins would add up.
The reasoning behind these experiments is familiar: check the data when more training stops helping, test where capacity pays, revisit a failed implementation, and measure the whole loop. Following through means implementing each idea, comparing the results, reviewing the evidence, and leaving work that someone else can use. NanoChat gave us a concrete test of that research system.
That is our vision for rekursiv.ai: AI scientists that accumulate knowledge, run the next experiment, and help us improve how research gets done. Speedrunning science means shortening the path from an idea to a result we can trust, then making that result useful to whoever comes next.
We are building the tools for that loop in the open. Try Trackinizer, Priml, and Configgle in your own research. Bring us a hard problem, a missing capability, or an experiment that failed. Join our Discord and let’s together build what the next scientist will inherit.
To cite this post:
@misc{rekursivai2026autoautoresearch,
author = {Kondratyuk, Dan and Lao, Junpeng and Chiu, Ming-Chang and Dillon, Joshua V.},
title = {(auto)²-research: {SoTA} on {Karpathy}'s {NanoChat} Benchmark},
year = {2026},
month = sep,
publisher = {rekursiv.ai},
url = {https://rekursiv.ai/blog/autoautoresearch/}
}
Scaling Language Models: Methods, Analysis & Insights from Training Gopher, arXiv:2112.11446. ↩
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment, arXiv:2606.27019. ↩
Round and Round We Go! What makes Rotary Positional Encodings useful?, arXiv:2410.06205. ↩
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models, arXiv:2601.07372. ↩
N-Grammer: Augmenting Transformers with latent n-grams, arXiv:2207.06366. ↩
Adam-mini: Use Fewer Learning Rates To Gain More, arXiv:2406.16793. ↩
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost, arXiv:1804.04235. ↩
Cut Your Losses in Large-Vocabulary Language Models, arXiv:2411.09009. ↩