AOS/skills/experiment-suite/references/00-incremental-execution.md

Incremental Execution — How an Experiment Package Actually Gets Built

Why this exists

A high-quality experiment package has six artefacts (design / code / results / figures / report / quality-checked PDF) and each one is non-trivial. None of this fits into a single LLM tool call. Trying to "do it all in one shot" hits context limits and produces shallow content across the board. This reference defines the only execution mode that works: incremental, file-persisted, resumable.

The three invariants

Invariant 1 — Filesystem is the source of truth

Each artefact is written to its file the moment it's drafted. The data binding goes to data_contract.md, the design to experiment_design.md, code to experiment/<file>.py, results to results.json, figures to figures/, report to experiment_report.md. If the agent crashes or the user pivots, the files survive.

Invariant 2 — Progress is observable on disk

cd output/experiment-suite/<slug>

wc -w experiment_design.md                              # design length
test -s data_contract.md && echo "data contract present"
wc -l experiment/*.py                                    # code length per module
python -c "import json; d=json.load(open('results.json')); print(d.get('provenance'), '|', list(d.keys()))"
ls figures/*.pdf | wc -l                                # figure count
wc -w experiment_report.md                              # report length

If any artefact looks thin, return to the relevant reference and expand.

Invariant 3 — Each batch is small enough to succeed, atomic enough to retry

Batch granularity for this skill:

Operation	Batch size per turn	Total batches	Persistence
Data binding	1 contract document	1 turn	Write data_contract.md
Design rewrite	1 design document	1 turn	Write experiment_design.md
Code rewrite	1 module per turn	4–6 turns (model, data, train, evaluate, config, README)	Write each experiment/<file>
Results curation	1 results.json structure	1 turn	Write results.json
Figure generation	1 figure per turn	3–6 turns	Save script + render + manifest entry
Report rewrite	1–2 sections per turn	4–6 turns	Edit experiment_report.md incrementally

A turn that tries to write all the code in one shot truncates. A turn that tries 5 figures at once produces shallow figures. One per turn is the rhythm.

Recovery — when a session resumes mid-task

cd output/experiment-suite/<slug>

# Design done?
wc -w experiment_design.md       # < 200 words means stub; > 600 means real

# Data binding done?
sed -n '1,120p' data_contract.md

# Code done?
for f in experiment/*.py; do echo "$(basename "$f"): $(wc -l < "$f") lines"; done

# Results well-formed?
python -c "import json; d=json.load(open('results.json')); print('OK' if 'summary' in d else 'STUB')"

# Figures present?
ls figures/*.pdf 2>/dev/null

# Report done?
wc -w experiment_report.md

Resume the sub-step that's incomplete. Never redo work that's already on disk.

Checkpoint files

Maintain four small bookkeeping files in the slug dir:

• .data_progress.txt — one line per dataset discovery / binding decision
• .code_progress.txt — one line per code module written / rewritten
• .figure_progress.txt — one line per figure generated
• .report_progress.txt — one line per report section drafted

echo "dataset-bound: GEO/GSE123456" >> .data_progress.txt
echo "model.py" >> .code_progress.txt
echo "fig_02_method_comparison" >> .figure_progress.txt
echo "results-section-drafted" >> .report_progress.txt

Sanity check before delivery

test -s data_contract.md
test -s experiment_design.md && [ $(wc -w < experiment_design.md) -gt 700 ]
ls experiment/{model,data,train,evaluate}.py
python -c "import json; d=json.load(open('results.json')); assert 'summary' in d, 'no summary'"
[ $(ls figures/*.pdf 2>/dev/null | wc -l) -ge 3 ]
test -s experiment_report.md && [ $(wc -w < experiment_report.md) -gt 800 ]

Quick checklist

• Data contract written before code generation
• Design rewritten in 1 dedicated turn (not interleaved)
• Code rewritten one module per turn
• Results curated in 1 dedicated turn (schema + provenance)
• Figures generated one per turn, scripts saved alongside renders
• Report drafted section by section
• Four checkpoint files maintained
• Compile/run check after each major artefact (e.g., python -c "import experiment.model" after writing model.py)
• Never tried to "do it all in one turn"