Pushing the Limits of Autonomous ML Science on ARC-AGI

At rekursiv.ai, we’ve built an autonomous team of AI scientists who propose ideas, run experiments, learn from results, and repeat.

After a few days of work, here’s their result:

Figure 1. Our system prioritized efficiency on ARC-AGI-1, achieving higher accuracy than LLMs that cost orders of magnitude more. These efficiency gains also transferred to ARC-AGI-2. We also include estimated public evaluation results for TRM/HRM.

Our AI team achieved state-of-the-art Pareto efficiency on ARC-AGI-1 and ARC-AGI-2, reaching 71.4-75.5% on ARC-AGI-1 and 17.5% on ARC-AGI-2 at a 16× to 11,600× lower computational cost than its accuracy peers. Fig. 1 above and Figs. 2 and 3 below show that for each of the rekursiv.ai variants we have, at similar ARC-AGI-1 accuracy we dramatically reduce the cost:

• 59.8% @ $0.000035
  • Peers
    • o4-mini (High): 58.7% @ $0.406
    • o3 (High): 60.8% @ $0.500
  • Gain: $0.406 / $0.000035 = 11,600×
• 71.4% @ $0.018
  • Peers
    • GPT-5.2 (Medium): 72.7% @ $0.345
    • Opus 4.5 (Thinking,16K): 72.0% @ $0.620
    • GPT-5.1 (Thinking,High): 72.8% @ $0.674
  • Gain: $0.345 / $0.018 = 19×
• 75.5% @ $0.058
  • Peers
    • Opus 4.5 (Thinking,32K): 75.8% @ $0.950
    • Gemini 3 Pro: 75.0% @ $0.058
    • o3 (Preview,Low): 75.7% @ $200
  • Gain: $0.950 / $0.058 = 16×

We price the cost of our model assuming $1.50 per GPU-hour for inference on an H100.

The same AI team also achieved 100% Sudoku accuracy with a Neural Network – a first-of-its kind result.

Why ARC-AGI?

We focused on ARC-AGI because this problem represents a fundamental shortcoming in modern ML systems.

Modern ML systems are usually transductive learners who, unlike humans (inductive learners), don’t explicitly learn general rules and instead use examples to guess answers. Introduced in On the Measure of Intelligence, ARC-AGI is a benchmark of skill acquisition and generalization under constrained prior knowledge. Its defining property is that the puzzles are easy for humans yet remain hard for ML.

Accurate tiny models on ARC-AGI

We kickstarted our AI scientist team with a 44.9% control on ARC-AGI-1 based on the transformer variant of Tiny Recursive Models (TRM) (published as 44.6%). By hill-climbing on its own hypotheses, their single model version peaked at 71.4% test-set accuracy and the ensemble variant hit 75.5% for max delta of 30.6%. This treatment also transferred to ARC-AGI-2, improving from TRM’s 7.8% result to 17.5%.

But what’s more exciting is how they achieved these results. Fig. 2 shows the AI team’s extended campaign on the ARC-AGI-1 benchmark.

Figure 2. Each point is a completed full evaluation. The red roofline traces the running-best single-model accuracy. Red diamonds mark ensemble results, while gray circles are non-roofline measured runs. Crosses indicate failures.

Across 235 experiments, the team explored 15 major research directions. Seven directions produced a durable gain in accuracy, training speed, or both, while eight produced no significant improvement. Each experiment lasted up to 48 hours, with poor-performing experiments cut short by monitoring agents.

Unlike comparable methods, our result did not use test-time training (TTT). Avoiding this additional inference cost made our method highly efficient while it outperformed many prior methods that did rely on TTT.

What moved the score

The final gains came from a sequence of research campaigns. Below we rank some of their major improvements and unsuccessful approaches.

Figure 3. The chart traces the additive 30.6-point ladder from our 44.9% starting point to the 75.5% ensemble result, alongside several ideas that did not work.

When we first applied our research system to this problem, accuracy peaked at 23%. The AI team quickly discovered that we had a bug in our DDP logic causing gradients to be computed as though training used only a single GPU. After ruling out subtle discrepancies with the original source code, they reproduced the TRM baseline on 4 H100 GPUs at 44.9%.

Muon optimizer. The AI team’s first set of improvements came from transferring our best recipe from the companion Sudoku campaign, which used the Muon optimizer, high learning rates, and label smoothing. The feedforward gate in SwiGLU led to unstable gradients, so it was replaced with a gate-normalized SwiGLU sigmoid(gate) * RMSNorm(gate * value), using RMSNorm. This increased accuracy from 44.9% to 59.5%.

Simplified architecture and visual priors. The team searched the web for related ideas that might transfer to TRM. One key paper, VARC, treated ARC as a visual problem. The team found improvements when replacing the 1D transformer with a 2D conv-based ViT and 2D RoPE. However, after researching Universal Reasoning Model (URM), the team found, counterintuitively, that returning to a 1D depthwise conv integrated into the SwiGLU performed better. The team also fused TRM’s latent and answer states into one recurrent hidden state to improve accuracy from 59.5% to 67.6%.

A failed verifier campaign. An analyst examined the types of errors the model made. Across 1,000 augmented test inputs per puzzle, 60% had the correct grid ranked first, and 80% had the correct grid somewhere in the generated pool. This revealed a large selection gap: the model could generate correct answers but had trouble ranking the correct one first. Related work explores this same test-time scaling problem through stochastic recursive trajectories in GRAM and confidence-based trajectory selection in C-voting.

This relationship can be seen more clearly in the plot below.

Figure 4. Allowing more candidate answers exposes the gap between generating and selecting a solution. With two allowed guesses, the ensemble solved 74.0%, while 1,000 guesses solved 90% of puzzles.

At first, the team tried various voting and selection methods, including logit- and q-head-based signals, over pools of 25 candidates. None beat majority voting. This led the team to train a neural verifier that could outperform majority voting, inspired by learned best-of-N selection in Training Verifiers to Solve Math Word Problems. A small classification head on a frozen TRM model separated correct from incorrect candidates with a held-out AUC of 0.975. While a pairwise ranker improved accuracy by 25.6 percentage points on its training gate, it then lost 2.0% on the evaluation set because its learned ranking did not transfer to unseen puzzles. After trying a dozen more ideas, the team ultimately abandoned this campaign.

Enhance the training data. Based on prior work, the team attempted to integrate datasets like RE-ARC into training to improve generalization, but it had the opposite effect. TRM also adds training puzzles through dihedral and color-permuting augmentations. The team produced two additional augmentations: scaling and translation, which improved evaluation accuracy from 67.6% to 70.1%.

Reduce precision. The team attempted to cast the feed-forward layers to FP8 or FP4, but the small matrix multiplications gained no training speedup.

Puzzle corruption with feedback-and-repair. Adapting the technique used in our Sudoku experiments, the team randomly corrupted 7.5% of cells across ACT steps while propagating intermediate board predictions. The model therefore learned not only to continue its own partial solution, but also to repair its own mistakes. This increased accuracy from 70.1% to 71.4%.

HPS transfer. A literal branching form of Hypothesis-Pinning Search (HPS) from our Sudoku experiments did not improve on ARC. A tuned acceptance rule fired on only 2.2% of searches, and two-thirds of the accepted states were wrong. The team attempted to correct this with a novel halt-consistency loss intended to stop correct trajectories from drifting, but it produced no gains. HPS transfer was therefore limited to the feedback-and-repair mechanism (see above) rather than explicit hypothesis search. For more information on HPS, see our Sudoku deep-dive post.

Ensembles. The final improvement came from running models in parallel and averaging their predictions. A five-model ensemble increased accuracy from 71.4% to the final score of 75.5%.

Transferring the resulting ARC-AGI-1 recipe to ARC-AGI-2 raised accuracy from our 6% baseline to 17.5%. This suggests that improvements developed on one ARC generation can transfer to the next. Due to time constraints we never ran the AI team directly on ARC-2.

We expect our semi-private accuracy to be 4–6 percentage points lower than our public-evaluation accuracy. TRM drops 4.6 points between the two splits. We have not yet made a semi-private submission.

Take-Aways

The biggest improvements came from novel training and evaluation algorithms rather than hyperparameter optimization. This supports rekursiv.ai’s central thesis that AI progress is not bounded by compute, but rather by ideas.

Accelerating automated ML research requires systems to ask bold questions and invent new approaches. Regrettably – and unsurprisingly – most proposed novel ideas did not survive experimental testing. This exposes a current limitation: the system’s feedback loop is gated by the time it takes an experiment to run. The faster we can run experiments, the more discoveries we can make. We believe developing faster experiments which elicit novel discovery is the most important challenge left for us to solve.

Our goal was not simply to maximize benchmark scores, but rather to establish a stronger baseline and test whether our AI scientist team could turn new ideas into measurable improvements. These campaigns show that the team can explore many ideas efficiently in parallel and find novel solutions to hard challenges. Much work remains to develop approaches that generalize better to ARC-AGI-2 and ARC-AGI-3.

What’s next?

Broadly speaking our primary objectives are to:

1. Understand and overcome the limits of autonomous science.
2. Produce useful artifacts for the community.

In the coming weeks, our primary focus is to expand our system across harder and more varied challenges.

If you feel inspired to help, email [email protected].

If you’d just love to chat, drop us a line at [email protected].

As we release more of our system we’ll also be setting up a platform for realtime discussion and sharing of results. Our goal is to accelerate knowledge discovery and that starts with building a community of scientists, engineers, and hackers (both human and AI).

Citation

To cite this post:

@misc{rekursivai2026pushingarcagi,
  author = {Kondratyuk, Dan and Dillon, Joshua V.},
  title = {Pushing the Limits of Autonomous {ML} Science: ARC-AGI},
  year = {2026},
  month = jul,
  publisher = {rekursiv.ai},
  url = {https://rekursiv.ai/blog/pushing-limits-arc-agi/}
}

References

1. Addressing ARC via Procedural Example Generation, arXiv:2404.07353.
2. ARC Is a Vision Problem!, arXiv:2511.14761.
3. ARC Prize 2025: Technical Report, arXiv:2601.10904.
4. C-voting: Confidence-Based Test-Time Voting without Explicit Energy Functions, arXiv:2604.13521.
5. Generative Recursive Reasoning, arXiv:2605.19376.
6. GLU Variants Improve Transformer, arXiv:2002.05202.
7. Hierarchical Reasoning Model, arXiv:2506.21734.
8. Less is More: Recursive Reasoning with Tiny Networks, arXiv:2510.04871.
9. LoopViT: Scaling Visual ARC with Looped Transformers, arXiv:2602.02156.
10. Muon: An optimizer for hidden layers in neural networks, technical note.
11. Muon is Scalable for LLM Training, arXiv:2502.16982.
12. On the Measure of Intelligence, arXiv:1911.01547.
13. Probabilistic Tiny Recursive Model, arXiv:2605.19943.
14. Reasoning is a Modality, arXiv:2601.13562.
15. Root Mean Square Layer Normalization, arXiv:1910.07467.
16. Rotary Position Embedding for Vision Transformer, arXiv:2403.13298.
17. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, arXiv:1612.01474.
18. Slots, Transitions, Loops: Learning Composable World Models for ARC, arXiv:2606.12316.
19. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning, arXiv:2411.07279.
20. Training Verifiers to Solve Math Word Problems, arXiv:2110.14168.
21. Universal Reasoning Model, arXiv:2512.14693.

Appendix: selected benchmark results

This table compares the main small recursive-model results and the most relevant visual or test-time-training systems discussed above.

ARC-AGI-1 and ARC-AGI-2

Scores are percentages reported by each source under its own protocol, so they are not all leaderboard-equivalent. * URM reports pass@1. Its released evaluator aggregates inverse-mapped augmented views. † NVARC’s ARC-AGI-2 score is from the private evaluation. TTA is test-time augmentation and voting, and TTT is per-task test-time training.