AOS/skills/experiment-suite/references/01-design-depth.md

Experiment Design Depth

Why this exists

The pipeline emits a 5-line design stub like:

**Task:** Transformer-based time series forecasting
**Dataset:** Synthetic dataset
**Methods:** Baseline A, Baseline B
**Metrics:** Accuracy, F1-Score, Runtime

That's not a design — it's a placeholder. A real experiment design tells someone else why your decisions are the right ones. This reference defines what a real design contains.

Hard target: experiment_design.md ≥ 700 words, structured into the seven sections below. Every choice (dataset, baseline, metric, ablation) carries a justification, not just a name.

Required structure

1. Research question and hypothesis

State the question precisely (one paragraph) and convert it into a falsifiable hypothesis. The hypothesis should name the expected effect direction and a rough magnitude.

We test whether channel-independent patched attention reduces long-horizon MSE by ≥ 5% on ETT benchmarks compared to point-wise attention, while not increasing compute by more than 1.3×.

If the question doesn't compress into a falsifiable form, weaken it until it does. Vague questions ("how does X work?") produce vague experiments.

2. Datasets

For each dataset:

• Name and citation
• Source route — user-supplied, agent-discovered, reused public, controlled, or synthetic fallback
• Repository / accession / DOI / stable URL when available
• Why this dataset — what makes it appropriate for the question (size, domain, structure)
• Splits — train / val / test sizes, how they were obtained
• Pre-processing — normalisation, windowing, encoding
• Access / licence condition — public, controlled, licensed, or local-only
• Known limitations — biases, age, label-noise issues

If the experiment uses 1 dataset, justify why one is enough. If 5+, justify why so many.

The dataset section must agree with $RUN/data_contract.md. Do not let the design and the runtime binding drift apart.

3. Baselines

A real comparison needs the right baselines. Group them by family:

• Lower-bound baselines: dumb baselines (mean, last-value, linear regression). If your method doesn't beat these, something is wrong.
• Same-paradigm baselines: other methods solving the same problem with the same paradigm.
• Adjacent-paradigm baselines: rival paradigms (e.g., for a Transformer paper: linear, MLP, state-space).
• Optional zero-shot reference: foundation models / pretrained baselines.

For each baseline cite a primary source, name the implementation used (released code or our re-implementation), and note the hyperparameter budget allocated.

4. Metrics

Three failure modes here:

• Reporting too few metrics (one number per method) hides regressions.
• Reporting too many metrics (10+) spreads the signal thin.

Aim for 3–6 metrics of distinct types: at least one accuracy-like, one cost-like (compute / latency / memory), and one robustness-like (worst-case / variance / OOD).

For each metric: name, units, direction (lower / higher better), why it matters for the research question, what value range is meaningful.

5. Protocol

• Seeds: minimum 3, ideally 5. Report mean ± std, not single-run numbers.
• Hyperparameter search: same budget per method (e.g., 30 trials). Document the search space, including ranges and distributions.
• Hardware: name the accelerator(s) and approximate wall-clock per configuration.
• Implementation: framework, key library versions, deterministic flags.
• Cross-validation: forecasting is autoregressive — use rolling-origin / time-series CV per~\cite{bergmeir2018cv} (or its equivalent for your task), not random k-fold.

6. Ablations

A method usually has 3–6 components. The ablation plan removes them one at a time on a representative dataset / horizon. List the planned ablations explicitly:

Ablation	What it removes	Expected effect direction	Why it tests our hypothesis

Without an ablation plan, the headline number cannot be attributed to any component.

7. Risk and limitations

Be explicit about what the experiment can't test:

• Datasets / domains the experiment does not cover
• Failure modes the metrics don't measure
• Confounds (e.g., compute differences, data-leakage risks)
• Negative results we'd accept as the hypothesis being wrong

A design without limitations is a design that doesn't know its weaknesses.

Compute / time budget

End with a small budget table:

Item	Per-config cost	Total
Method × seeds × hyperparameter trials	30 min × 3 × 30 = 45 hrs	one-time
Ablation re-runs	30 min × 3 × 6 = 9 hrs	re-used datasets
Total	≈ 54 hrs on 1 A100	~ 2.5 days

This calibrates the user on the cost of running the experiment for real and supports the simulated-vs-measured choice in references/03-results-protocol.md.

Anti-patterns

• No hypothesis, just a question — "how does X compare to Y?" is not a hypothesis. State the expected direction.
• Cherry-picked baselines — picking only the baselines you'll beat. Include baselines that might beat you.
• Single seed — never trust a single-seed number.
• No ablation plan — the headline number with no component attribution is not a finding.
• Real-data brag with synthetic data — if the dataset is synthetic, say so up front in §2.

Quick checklist

• Hypothesis is falsifiable, with direction + rough magnitude
• Each dataset has its rationale, splits, pre-processing, limitations
• Baselines grouped by family; each has a primary citation and budget
• 3–6 metrics covering accuracy + cost + robustness, each with rationale
• Protocol explicit: seeds, hyperparameter budget, hardware, framework
• Ablation plan with expected effect direction per ablation
• Limitations and risks named
• Compute budget concrete (hours, not "TBD")
• Total length ≥ 700 words