Getting Started with AutoResearch

This tutorial shows how to set up AutoResearch , run the baseline experiment, and then try one simple, conservative follow-up change.

The goal is not to build a giant autonomous research lab on day one. The goal is to understand the loop clearly enough that you can trust what the agent is doing.

What you need

According to the official repo, the default setup expects:

a single NVIDIA GPU,
Python 3.10+,
uv.

If you are on macOS, Windows RTX, or AMD hardware, the README links to community forks for those platforms. For the cleanest first run, follow the official repo or a platform-specific fork that matches your machine.

Understand the three-file contract first

Before you run anything, understand the design:

File	Purpose	Rule
`prepare.py`	data prep, tokenizer training, dataloader, fixed evaluation	do not modify during the experiment loop
`train.py`	model, optimizer, hyperparameters, training loop	this is the file the agent edits
`program.md`	operational instructions for the agent	this is the human steering layer

That separation is important.

Rule of thumb: during the main experiment loop, the agent changes train.py, while prepare.py stays fixed so the evaluation remains trustworthy.

Step 1: Clone the repo and install dependencies


git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync

This installs the project dependencies into the local uv environment.

Step 2: Prepare the data and tokenizer

For the full default setup:


uv run prepare.py

If you only want a quick smoke test of the setup, prepare.py also supports a smaller shard count:


uv run prepare.py --num-shards 8

That will not represent a full research run, but it is useful for verifying that data download and tokenizer creation work correctly.

The repo stores its cache in ~/.cache/autoresearch/.

Step 3: Create a fresh research branch

The official workflow uses a dedicated branch per run:


git checkout -b autoresearch/mar30-demo

Use any tag you want, but make it a fresh branch so the experiment history stays clean.

Step 4: Initialize `results.tsv`

Create a TSV file with the header row expected by program.md:


commit	val_bpb	memory_gb	status	description

The repo’s convention is:

keep for improvements,
discard for worse or neutral changes,
crash for failed runs.

Also important: do not commit results.tsv. Keep it untracked as a local experiment ledger.

Step 5: Run the baseline exactly as-is

Your first run should always be the untouched baseline.


uv run train.py > run.log 2>&1

Then extract the important numbers.

On Linux / macOS


grep "^val_bpb:\|^peak_vram_mb:" run.log

On PowerShell


Select-String -Path .\run.log -Pattern '^val_bpb:','^peak_vram_mb:'

Record the baseline in results.tsv.

Example:


commit	val_bpb	memory_gb	status	description
abc1234	0.997900	44.0	keep	baseline

Your numbers will likely differ based on hardware, and that is expected.

Step 6: Read `program.md` like an operating manual

Before you let an agent start editing code, read program.md carefully.

Its core rules are:

run the baseline first,
only edit train.py,
keep experiments simple when possible,
log every run in results.tsv,
keep changes only if val_bpb improves,
revert if the result is worse,
keep running until the human stops the process.

That file is effectively the repo’s autonomous research policy.

Step 7: Start with one simple experiment

For a first manual experiment, keep it conservative.

One easy knob is DEPTH in train.py.

The default hyperparameter section includes values such as:


ASPECT_RATIO = 64
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
TOTAL_BATCH_SIZE = 2**19
DEPTH = 8
DEVICE_BATCH_SIZE = 128

A very simple first experiment is to reduce depth from 8 to 6 and leave everything else alone:


DEPTH = 6  # was 8

Why this is a good beginner experiment:

it changes one meaningful variable,
it keeps the diff tiny,
it lets you see how AutoResearch compares a smaller architecture against the baseline,
and it is easy to revert if the result gets worse.

After changing train.py, commit the change and run the experiment again:


git add train.py
git commit -m "test shallower model depth"
uv run train.py > run.log 2>&1

Then extract the metrics again and compare against the baseline.

Keep or revert

If val_bpb is lower, keep the commit and log it as keep.
If val_bpb is equal or higher, log it as discard and reset back.

Example discard flow:


git reset --hard HEAD~1

That keep-or-revert ratchet is the heart of AutoResearch.

Step 8: Hand the loop to a coding agent

Once you have verified the baseline manually, you can ask a coding agent to take over the experiment loop.

A good first prompt is boring on purpose:


Read README.md, program.md, prepare.py, and train.py.
We are on a fresh branch for a new AutoResearch run.

1. Verify setup is complete.
2. Run the baseline first and log it to results.tsv.
3. Only edit train.py.
4. Try one conservative experiment at a time.
5. Keep changes only if val_bpb improves.
6. Revert worse changes.
7. Redirect runs to run.log and extract val_bpb plus peak_vram_mb after each run.
8. Do not commit results.tsv.

That is enough to get started.

Notice what is not in the prompt:

no giant research agenda,
no vague “optimize everything,”
no permission to rewrite the evaluator.

Small scope is your friend here.

Smaller-compute notes

The official README also gives guidance for lighter hardware or community forks:

use a lower-entropy dataset like TinyStories,
reduce DEPTH,
reduce TOTAL_BATCH_SIZE,
consider WINDOW_PATTERN = "L",
lower MAX_SEQ_LEN and EVAL_TOKENS in prepare.py when adapting a fork for smaller hardware.

That last point is important: changing prepare.py is part of porting the setup to a different platform, not part of the normal experiment loop once the evaluator is fixed.

Common mistakes

Skipping the baseline — then you have nothing trustworthy to compare against.
Changing too many knobs at once — if the run gets better or worse, you will not know why.
Letting the agent touch prepare.py during experiments — that breaks the fairness of the evaluator.
Committing results.tsv — keep it as local experiment bookkeeping.
Using an open-ended prompt — the agent works better when the operating rules are explicit.

Final takeaway

The best way to start with AutoResearch is not to chase a hundred experiments immediately. It is to understand the contract:

fixed evaluator,
one editable training file,
baseline first,
keep or revert based on the metric.

Once that pattern clicks, AutoResearch stops looking like a flashy demo and starts looking like a very practical workflow for agent-guided iteration.