What Is AutoResearch?
AutoResearch is a simple but very important idea: treat an LLM coding agent like a tireless research assistant that can modify training code, run experiments, measure the result, and keep only the improvements.
The project, published by Andrej Karpathy, is intentionally tiny. Instead of building a giant platform, it reduces the whole system to a few files, a fixed evaluator, and a repeatable loop. That constraint is the magic.
At a high level, the agent does this:
read instructions → run baseline → modify train.py → run experiment → measure val_bpb
→ keep if better / revert if worse → repeatThe result is not “AGI doing frontier science” — at least not yet, despite the excellent sci-fi README energy — but it is a very practical pattern for autonomous local search over real ML code.
A compact view of the AutoResearch contract: prepare.py stays fixed, train.py is the editable sandbox, and program.md defines the operating rules.
The core idea
AutoResearch gives the agent a narrow but meaningful playground:
- one repository,
- one editable training script,
- one fixed evaluation metric,
- one short experiment budget,
- one keep-or-revert rule.
That makes the loop much more disciplined than open-ended prompting.
Instead of asking a model to talk about research ideas, AutoResearch asks it to prove ideas inside code.
The three files that matter
The repository is deliberately built around three files:
| File | Role | Who changes it? |
|---|---|---|
prepare.py | data prep, tokenizer training, dataloader utilities, fixed evaluation | human during setup only |
train.py | model architecture, optimizer, hyperparameters, training loop | the agent |
program.md | research instructions and operating rules | the human |
This separation is what keeps the system understandable.
prepare.py is the guardrail
prepare.py defines fixed constants like the time budget, sequence length, and evaluation utilities. It also provides the ground-truth evaluator, including the val_bpb metric.
That file is intentionally treated as read-only during experiments.
This is critical, because if the agent could rewrite the evaluator, it could “win” by moving the goalposts. A clever cheater is not the same thing as a useful researcher.
train.py is the sandbox
train.py is where the experiments happen. It contains:
- model architecture,
- optimizer logic,
- hyperparameters,
- training schedules,
- the training loop.
The agent is allowed to change this file freely. That means it can try:
- smaller or larger models,
- different learning rates,
- altered attention patterns,
- different optimization behavior,
- changes to batch size and training dynamics.
program.md is the human steering wheel
program.md is a surprisingly elegant part of the design.
You do not need to hand-code every experimental rule into Python. Instead, the human writes operational instructions in Markdown: how to name runs, how to log results, when to keep a change, when to revert, and what simplicity trade-offs to prefer.
In other words: the human programs the research process, while the agent programs the training code.
How the experiment loop works
The loop in program.md is intentionally straightforward:
- Create a fresh experiment branch.
- Run the baseline first.
- Edit
train.pywith one experimental idea. - Commit the change.
- Run
uv run train.py > run.log 2>&1. - Extract the key metrics from the log.
- Record the result in
results.tsv. - Keep the commit if
val_bpbimproved. - Revert if it did not.
- Repeat.
This is basically a ratchet.
The branch only moves forward when the metric gets better. That means git history becomes a compact research memory, and results.tsv becomes a lightweight experiment ledger.
Why val_bpb matters
The main metric is validation bits per byte (val_bpb), and lower is better.
That choice matters for two reasons:
- It is a fixed target, so experiments are comparable.
- It is vocabulary-size-independent, which makes architectural comparisons fairer.
This is one of the strongest ideas in the repo: the agent is not rewarded for sounding convincing; it is rewarded for improving a stable evaluator.
Why the 5-minute time budget is clever
Every training run gets a fixed 5-minute wall-clock budget (excluding startup and compilation overhead).
That sounds small, but it is actually the feature.
A fixed budget means the agent is optimizing for what matters on that machine under that constraint. It cannot cheat by running a giant model for three times longer. Instead, it must search for the best result inside the same experimental box.
That also makes the system nicely overnight-friendly: if one run is roughly five minutes, the agent can complete about a dozen experiments per hour without constant supervision.
Why this feels different from classic AutoML
AutoResearch overlaps with AutoML, but it is not just another hyperparameter tuner.
Traditional search systems like Optuna or NAS usually explore a predefined search space. You decide the knobs in advance, and the optimizer tries combinations.
AutoResearch is more open-ended because the coding agent can rewrite arbitrary parts of train.py.
So the search space is not just:
- learning rate,
- depth,
- batch size.
It can also include:
- how the optimizer is structured,
- how the model layers interact,
- what scheduling logic exists,
- whether a simplification removes dead weight.
That is what makes it interesting. The agent is not just tuning a dashboard; it is revising the machine behind the dashboard.
What makes AutoResearch useful in practice
A few things stand out:
1. It turns agent autonomy into something measurable
A lot of AI-agent demos feel impressive until you ask, “Better by what metric?”
AutoResearch answers that cleanly: lower val_bpb or it does not count.
2. It keeps the problem scoped
Small repo. One editable file. One evaluator. One loop.
That limited scope is why the system is understandable and why failures are debuggable.
3. Git becomes part of the research system
Git is not just version control here. It becomes a memory mechanism:
- branches isolate runs,
- commits record ideas,
- resets discard failures,
- history preserves the winning path.
4. Markdown becomes an orchestration interface
program.md acts like a lightweight operational playbook for the agent. That is a neat pattern beyond AutoResearch itself.
You can imagine similar designs for:
- benchmark agents,
- data-cleaning agents,
- evaluation pipelines,
- codebase migration loops,
- prompt/program search systems.
Important limitations
AutoResearch is cool, but it is not magic. A few caveats matter.
It is still local search
The agent usually improves things by trying nearby ideas, not by inventing entirely new science from scratch. That means it can plateau, get stuck in habits, or miss better but less obvious directions.
It can overfit to the evaluator
Any fixed metric creates pressure to optimize for that exact metric. If your evaluator is weak, the agent can become “good at the test” instead of broadly good.
Results are hardware-specific
The repo explicitly optimizes under a fixed time budget on a specific machine. That is useful, but it also means your results may not transfer cleanly across hardware.
The default repo assumes a single NVIDIA GPU
The official project is designed around a single NVIDIA GPU. If you are on a smaller or different platform, the README points to community forks for macOS, Windows RTX, and AMD users.
So yes, the future is autonomous. But the future still asks about VRAM.
Why I find the design compelling
What I like most about AutoResearch is not just the benchmark chasing. It is the interface design.
The repo shows a strong pattern for agentic systems:
- keep the mutable surface area tiny,
- make the evaluator fixed,
- store state in ordinary developer tools,
- let humans steer through instructions instead of micromanaging every step.
That pattern is bigger than this repo. It is a useful blueprint for any system where an agent should improve something continuously without becoming unreviewable chaos.
Read next
If you want the hands-on version, I wrote a companion tutorial here:
Further reading
- AutoResearch GitHub repository
- Philipp Schmid’s overview of AutoResearch
- DataCamp guide to AutoResearch
- Data Science Dojo overview of AutoResearch
Final takeaway
AutoResearch is one of the clearest examples I have seen of a coding agent being useful because of constraints, not despite them.
Give the agent one file to change, one metric to satisfy, one short run budget, and a disciplined keep-or-revert loop — and suddenly “autonomous research” stops sounding like marketing and starts looking like an engineering pattern worth reusing.