AOS/skills/experiment-suite/references/06-quality-gate.md

Experiment-Suite Quality Gate — Self-Check Before Delivery

Why this exists

This checklist runs after drafting and assembling, before declaring the experiment package complete. If any gate fails, return to the relevant reference.

The gate is bright-line. Do not soften the targets to ship.

Hard gates (must pass)

G1 · Design depth

test -s experiment_design.md && [ $(wc -w < experiment_design.md) -ge 700 ]
test -s data_contract.md

Must include the seven sections from references/01-design-depth.md: hypothesis, datasets, baselines, metrics, protocol, ablations, limitations.

G2 · Code skeleton runs (imports cleanly)

cd experiment
for m in model data train evaluate; do
  python -c "import importlib; m = importlib.import_module('$m'); print('$m: OK')"
done

All four must import without error. Then check for placeholders:

grep -rE "^\s*pass\s*$|# TODO|raise NotImplementedError" experiment/*.py | head
# Should be empty (or only inside genuinely-optional hooks).

G3 · requirements.txt matches imports

grep -h "^import \|^from " experiment/*.py | awk '{print $2}' | cut -d. -f1 | sort -u
# Compare against requirements.txt — every third-party import should be listed.

G4 · README.md has install + launch instructions

grep -E "pip install|python train.py|python evaluate.py" experiment/README.md

Must include all three lines (install, launch, evaluate) or equivalents.

G5 · results.json schema valid

import json
r = json.load(open("results.json"))
required = {"task", "dataset", "metrics", "seeds", "provenance", "summary"}
assert not (required - set(r.keys())), f"missing: {required - set(r.keys())}"
assert r["provenance"].get("mode") in {"simulated", "measured", "mixed"}
print("OK")

If mode is measured, the provenance should also point to data_contract.md directly or indirectly.

G6 · Figures present and publication-grade

[ $(ls figures/*.pdf 2>/dev/null | wc -l) -ge 3 ]
test -f figures/manifest.json
test -f figures/figure_contract.md

For each figure:

• vector PDF (not raster)
• has a sibling make_fig_*.py script
• listed in manifest.json with id, path_pdf, caption, section
• should also have sibling .svg / .tiff when the environment supports them

G7 · Report depth

test -s experiment_report.md && [ $(wc -w < experiment_report.md) -ge 800 ]

Must contain the seven sections from references/05-report-structure.md. Search for section headings:

grep -E "^## (Problem|Design|Method|Results|Analysis|Limitations|Reproduction)" experiment_report.md | wc -l
# Should be ≥ 6

G8 · Provenance disclosed wherever numbers appear

If results.json provenance mode is simulated:

grep -i "simulated" experiment_report.md     # must appear in header + figure captions
grep -i "simulated" figures/manifest.json    # captions should say "Numbers are simulated."

If mode is measured:

grep -i "simulated" experiment_report.md     # should NOT appear (otherwise inconsistent)
grep -i "measured\|three-seed\|hardware" experiment_report.md   # should appear in §1 header

If mode is mixed:

grep -i "mixed\|partially\|by_method" experiment_report.md      # should explicitly explain

If mode is measured and the data were agent-discovered or reused public data:

grep -i "data contract" experiment_report.md
grep -i "repository\|accession\|doi\|source" data_contract.md

Soft gates (should pass)

S1 · No marketing prose in the report

grep -E "state-of-the-art|cutting-edge|comprehensive|extensive|novel" experiment_report.md | head

These phrases aren't categorically wrong but should be sparse and earned.

S2 · No hidden numbers

# Numbers in the report must be traceable to results.json
# (Manual check; eyeball that every number in the headline table appears in results.json["summary"].)

S3 · Code is callable end-to-end (smoke)

For experiments where compute permits:

cd experiment
# Run with --quick or a tiny config to verify the pipeline doesn't crash.
# This is optional but a powerful confidence check.

S4 · Visuals make sense

Open each figure PDF. Check:

• Axes labelled, legend readable at print scale
• Watermark "simulated" present iff mode is simulated
• Colors from the publication palette (no default cycle)
• Hero panel / supporting panel hierarchy matches figures/figure_contract.md

S5 · Multi-seed reporting

import json
r = json.load(open("results.json"))
n_seeds = r.get("seeds", [])
if len(n_seeds) < 3:
    print(f"WARNING: only {len(n_seeds)} seeds — disclose in report.")

Final report format

When all gates pass:

Experiment package ready: output/experiment-suite/<slug>/

Stats:
  Design:        912 words, 7 sections
  Code:          model.py 184 lines, data.py 96 lines, train.py 128 lines, evaluate.py 64 lines
  Results:       schema valid, mode=simulated, 3 seeds, 4 datasets × 4 horizons
  Figures:       5 (2 method comparison, 1 ablation panel, 1 heatmap, 1 training curves)
  Report:        1240 words, 8 sections
  Disclosure:    "simulated" present in report header, every figure caption, results.json provenance

Quality gate: PASSED (G1–G8 hard, S1–S5 soft).

If any hard gate cannot honestly be cleared:

"Code skeleton imports cleanly but train.py has a NotImplementedError in the loss-weighting branch — leaving in place because the design called for it as an optional extension; the main loss path is functional."

That's a legitimate deviation. Padding the design to 700 words with filler is not.

Quick checklist

• G1 design ≥ 700 words, all 7 sections present
• data_contract.md exists and matches the design / results provenance
• G2 all 4 modules import cleanly; no pass/TODO/NotImplementedError in core paths
• G3 requirements.txt matches actual imports
• G4 README has install + launch + evaluate commands
• G5 results.json schema valid with provenance
• G6 ≥ 3 figures, vector PDF, all in manifest
• Figure contract written before plotting; export bundle present when possible
• G7 report ≥ 800 words, 7 sections
• G8 provenance consistent across report / figures / results.json
• S1–S5 soft gates reviewed
• Honest report — don't pad to clear gates