feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed
1 parent
7acc0c5451
commit
e45b68bb8b
1089 files changed
+260614
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# Incremental Execution — How an Experiment Package Actually Gets Built
|
||||
|
||||
## Why this exists
|
||||
|
||||
A high-quality experiment package has six artefacts (design / code / results / figures / report / quality-checked PDF) and each one is non-trivial. None of this fits into a single LLM tool call. Trying to "do it all in one shot" hits context limits and produces shallow content across the board. This reference defines the only execution mode that works: **incremental, file-persisted, resumable**.
|
||||
|
||||
## The three invariants
|
||||
|
||||
### Invariant 1 — Filesystem is the source of truth
|
||||
|
||||
Each artefact is written to its file the moment it's drafted. The data binding goes to `data_contract.md`, the design to `experiment_design.md`, code to `experiment/<file>.py`, results to `results.json`, figures to `figures/`, report to `experiment_report.md`. If the agent crashes or the user pivots, the files survive.
|
||||
|
||||
### Invariant 2 — Progress is observable on disk
|
||||
|
||||
```bash
|
||||
cd output/experiment-suite/<slug>
|
||||
|
||||
wc -w experiment_design.md # design length
|
||||
test -s data_contract.md && echo "data contract present"
|
||||
wc -l experiment/*.py # code length per module
|
||||
python -c "import json; d=json.load(open('results.json')); print(d.get('provenance'), '|', list(d.keys()))"
|
||||
ls figures/*.pdf | wc -l # figure count
|
||||
wc -w experiment_report.md # report length
|
||||
```
|
||||
|
||||
If any artefact looks thin, return to the relevant reference and expand.
|
||||
|
||||
### Invariant 3 — Each batch is small enough to succeed, atomic enough to retry
|
||||
|
||||
Batch granularity for this skill:
|
||||
|
||||
| Operation | Batch size per turn | Total batches | Persistence |
|
||||
|---|---|---|---|
|
||||
| Data binding | 1 contract document | 1 turn | Write `data_contract.md` |
|
||||
| Design rewrite | 1 design document | 1 turn | Write `experiment_design.md` |
|
||||
| Code rewrite | 1 module per turn | 4–6 turns (model, data, train, evaluate, config, README) | Write each `experiment/<file>` |
|
||||
| Results curation | 1 results.json structure | 1 turn | Write `results.json` |
|
||||
| Figure generation | 1 figure per turn | 3–6 turns | Save script + render + manifest entry |
|
||||
| Report rewrite | 1–2 sections per turn | 4–6 turns | Edit `experiment_report.md` incrementally |
|
||||
|
||||
A turn that tries to write all the code in one shot truncates. A turn that tries 5 figures at once produces shallow figures. One per turn is the rhythm.
|
||||
|
||||
## Recovery — when a session resumes mid-task
|
||||
|
||||
```bash
|
||||
cd output/experiment-suite/<slug>
|
||||
|
||||
# Design done?
|
||||
wc -w experiment_design.md # < 200 words means stub; > 600 means real
|
||||
|
||||
# Data binding done?
|
||||
sed -n '1,120p' data_contract.md
|
||||
|
||||
# Code done?
|
||||
for f in experiment/*.py; do echo "$(basename "$f"): $(wc -l < "$f") lines"; done
|
||||
|
||||
# Results well-formed?
|
||||
python -c "import json; d=json.load(open('results.json')); print('OK' if 'summary' in d else 'STUB')"
|
||||
|
||||
# Figures present?
|
||||
ls figures/*.pdf 2>/dev/null
|
||||
|
||||
# Report done?
|
||||
wc -w experiment_report.md
|
||||
```
|
||||
|
||||
Resume the sub-step that's incomplete. Never redo work that's already on disk.
|
||||
|
||||
## Checkpoint files
|
||||
|
||||
Maintain four small bookkeeping files in the slug dir:
|
||||
|
||||
- `.data_progress.txt` — one line per dataset discovery / binding decision
|
||||
- `.code_progress.txt` — one line per code module written / rewritten
|
||||
- `.figure_progress.txt` — one line per figure generated
|
||||
- `.report_progress.txt` — one line per report section drafted
|
||||
|
||||
```bash
|
||||
echo "dataset-bound: GEO/GSE123456" >> .data_progress.txt
|
||||
echo "model.py" >> .code_progress.txt
|
||||
echo "fig_02_method_comparison" >> .figure_progress.txt
|
||||
echo "results-section-drafted" >> .report_progress.txt
|
||||
```
|
||||
|
||||
## Sanity check before delivery
|
||||
|
||||
```bash
|
||||
test -s data_contract.md
|
||||
test -s experiment_design.md && [ $(wc -w < experiment_design.md) -gt 700 ]
|
||||
ls experiment/{model,data,train,evaluate}.py
|
||||
python -c "import json; d=json.load(open('results.json')); assert 'summary' in d, 'no summary'"
|
||||
[ $(ls figures/*.pdf 2>/dev/null | wc -l) -ge 3 ]
|
||||
test -s experiment_report.md && [ $(wc -w < experiment_report.md) -gt 800 ]
|
||||
```
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] Data contract written before code generation
|
||||
- [ ] Design rewritten in 1 dedicated turn (not interleaved)
|
||||
- [ ] Code rewritten **one module per turn**
|
||||
- [ ] Results curated in 1 dedicated turn (schema + provenance)
|
||||
- [ ] Figures generated **one per turn**, scripts saved alongside renders
|
||||
- [ ] Report drafted **section by section**
|
||||
- [ ] Four checkpoint files maintained
|
||||
- [ ] Compile/run check after each major artefact (e.g., `python -c "import experiment.model"` after writing model.py)
|
||||
- [ ] Never tried to "do it all in one turn"
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
# Experiment Design Depth
|
||||
|
||||
## Why this exists
|
||||
|
||||
The pipeline emits a 5-line design stub like:
|
||||
```
|
||||
**Task:** Transformer-based time series forecasting
|
||||
**Dataset:** Synthetic dataset
|
||||
**Methods:** Baseline A, Baseline B
|
||||
**Metrics:** Accuracy, F1-Score, Runtime
|
||||
```
|
||||
|
||||
That's not a design — it's a placeholder. A real experiment design tells someone else why your decisions are the right ones. This reference defines what a real design contains.
|
||||
|
||||
**Hard target:** `experiment_design.md` ≥ 700 words, structured into the seven sections below. Every choice (dataset, baseline, metric, ablation) carries a justification, not just a name.
|
||||
|
||||
## Required structure
|
||||
|
||||
### 1. Research question and hypothesis
|
||||
|
||||
State the question precisely (one paragraph) and convert it into a falsifiable hypothesis. The hypothesis should name the expected effect direction and a rough magnitude.
|
||||
|
||||
> *We test whether channel-independent patched attention reduces long-horizon MSE by ≥ 5% on ETT benchmarks compared to point-wise attention, while not increasing compute by more than 1.3×.*
|
||||
|
||||
If the question doesn't compress into a falsifiable form, weaken it until it does. Vague questions ("how does X work?") produce vague experiments.
|
||||
|
||||
### 2. Datasets
|
||||
|
||||
For each dataset:
|
||||
- **Name** and **citation**
|
||||
- **Source route** — user-supplied, agent-discovered, reused public, controlled, or synthetic fallback
|
||||
- **Repository / accession / DOI / stable URL** when available
|
||||
- **Why this dataset** — what makes it appropriate for the question (size, domain, structure)
|
||||
- **Splits** — train / val / test sizes, how they were obtained
|
||||
- **Pre-processing** — normalisation, windowing, encoding
|
||||
- **Access / licence condition** — public, controlled, licensed, or local-only
|
||||
- **Known limitations** — biases, age, label-noise issues
|
||||
|
||||
If the experiment uses 1 dataset, justify why one is enough. If 5+, justify why so many.
|
||||
|
||||
The dataset section must agree with `$RUN/data_contract.md`. Do not let the design and the runtime
|
||||
binding drift apart.
|
||||
|
||||
### 3. Baselines
|
||||
|
||||
A real comparison needs the right baselines. Group them by family:
|
||||
|
||||
- **Lower-bound baselines**: dumb baselines (mean, last-value, linear regression). If your method doesn't beat these, something is wrong.
|
||||
- **Same-paradigm baselines**: other methods solving the same problem with the same paradigm.
|
||||
- **Adjacent-paradigm baselines**: rival paradigms (e.g., for a Transformer paper: linear, MLP, state-space).
|
||||
- **Optional zero-shot reference**: foundation models / pretrained baselines.
|
||||
|
||||
For each baseline cite a primary source, name the implementation used (released code or our re-implementation), and note the hyperparameter budget allocated.
|
||||
|
||||
### 4. Metrics
|
||||
|
||||
Three failure modes here:
|
||||
|
||||
- Reporting too few metrics (one number per method) hides regressions.
|
||||
- Reporting too many metrics (10+) spreads the signal thin.
|
||||
|
||||
Aim for **3–6 metrics** of distinct types: at least one accuracy-like, one cost-like (compute / latency / memory), and one robustness-like (worst-case / variance / OOD).
|
||||
|
||||
For each metric: name, units, direction (lower / higher better), why it matters for the research question, what value range is meaningful.
|
||||
|
||||
### 5. Protocol
|
||||
|
||||
- **Seeds**: minimum 3, ideally 5. Report mean ± std, not single-run numbers.
|
||||
- **Hyperparameter search**: same budget per method (e.g., 30 trials). Document the search space, including ranges and distributions.
|
||||
- **Hardware**: name the accelerator(s) and approximate wall-clock per configuration.
|
||||
- **Implementation**: framework, key library versions, deterministic flags.
|
||||
- **Cross-validation**: forecasting is autoregressive — use rolling-origin / time-series CV per~\cite{bergmeir2018cv} (or its equivalent for your task), not random k-fold.
|
||||
|
||||
### 6. Ablations
|
||||
|
||||
A method usually has 3–6 components. The ablation plan removes them one at a time on a representative dataset / horizon. List the planned ablations explicitly:
|
||||
|
||||
| Ablation | What it removes | Expected effect direction | Why it tests our hypothesis |
|
||||
|---|---|---|---|
|
||||
|
||||
Without an ablation plan, the headline number cannot be attributed to any component.
|
||||
|
||||
### 7. Risk and limitations
|
||||
|
||||
Be explicit about what the experiment **can't** test:
|
||||
|
||||
- Datasets / domains the experiment does not cover
|
||||
- Failure modes the metrics don't measure
|
||||
- Confounds (e.g., compute differences, data-leakage risks)
|
||||
- Negative results we'd accept as the hypothesis being wrong
|
||||
|
||||
A design without limitations is a design that doesn't know its weaknesses.
|
||||
|
||||
## Compute / time budget
|
||||
|
||||
End with a small budget table:
|
||||
|
||||
| Item | Per-config cost | Total |
|
||||
|---|---|---|
|
||||
| Method × seeds × hyperparameter trials | 30 min × 3 × 30 = 45 hrs | one-time |
|
||||
| Ablation re-runs | 30 min × 3 × 6 = 9 hrs | re-used datasets |
|
||||
| Total | ≈ 54 hrs on 1 A100 | ~ 2.5 days |
|
||||
|
||||
This calibrates the user on the cost of running the experiment for real and supports the simulated-vs-measured choice in `references/03-results-protocol.md`.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **No hypothesis, just a question** — "how does X compare to Y?" is not a hypothesis. State the expected direction.
|
||||
- **Cherry-picked baselines** — picking only the baselines you'll beat. Include baselines that might beat you.
|
||||
- **Single seed** — never trust a single-seed number.
|
||||
- **No ablation plan** — the headline number with no component attribution is not a finding.
|
||||
- **Real-data brag with synthetic data** — if the dataset is synthetic, say so up front in §2.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] Hypothesis is falsifiable, with direction + rough magnitude
|
||||
- [ ] Each dataset has its rationale, splits, pre-processing, limitations
|
||||
- [ ] Baselines grouped by family; each has a primary citation and budget
|
||||
- [ ] 3–6 metrics covering accuracy + cost + robustness, each with rationale
|
||||
- [ ] Protocol explicit: seeds, hyperparameter budget, hardware, framework
|
||||
- [ ] Ablation plan with expected effect direction per ablation
|
||||
- [ ] Limitations and risks named
|
||||
- [ ] Compute budget concrete (hours, not "TBD")
|
||||
- [ ] Total length ≥ 700 words
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
# Data Contract For Experiment Runs
|
||||
|
||||
Use this file whenever the run is intended to support a measured experiment, or when the agent
|
||||
searches for and binds a dataset on the fly.
|
||||
|
||||
`experiment-suite` is a skill system, not a fixed benchmark repository. That means `data.py`
|
||||
and `config.yaml` should be generated for the current run, not frozen globally. The price of this
|
||||
flexibility is that the run can become impossible to audit unless the dataset choice is written
|
||||
down explicitly. `data_contract.md` is the lightweight fix.
|
||||
|
||||
## Required file
|
||||
|
||||
Write `$RUN/data_contract.md` with this structure:
|
||||
|
||||
```markdown
|
||||
# Data Contract
|
||||
|
||||
## Task binding
|
||||
- research question:
|
||||
- task type:
|
||||
- target variable:
|
||||
|
||||
## Dataset route
|
||||
- mode: user-supplied | agent-discovered | reused-public | controlled | synthetic-fallback
|
||||
- source:
|
||||
- repository / landing page:
|
||||
- accession / DOI / stable URL:
|
||||
- version / snapshot / date accessed:
|
||||
- licence / access condition:
|
||||
- persistent identifier quality: DOI | accession | stable URL | none
|
||||
|
||||
## Files used in this run
|
||||
- file or subset:
|
||||
- role: train / val / test / metadata / labels / source data
|
||||
|
||||
## Split protocol
|
||||
- split source:
|
||||
- split rule:
|
||||
- leakage risks:
|
||||
|
||||
## Pre-processing contract
|
||||
- raw to model input:
|
||||
- normalization / filtering:
|
||||
- exclusions:
|
||||
|
||||
## Reuse and publication notes
|
||||
- can this run be marked measured:
|
||||
- source-data / FAIR follow-up needed:
|
||||
- unresolved risks:
|
||||
```
|
||||
|
||||
## Minimum rules
|
||||
|
||||
- `agent-discovered` is valid. If the agent searched Hugging Face, ModelScope, GEO, SRA, or a
|
||||
paper-linked repository and selected one dataset, record that explicitly.
|
||||
- If the data are reused public data, record the source and version.
|
||||
- Prefer a stable identifier (DOI, accession, or repository record) over an ad hoc file link.
|
||||
- If the data are controlled or restricted, record the access route and do not imply open
|
||||
reproducibility.
|
||||
- If no acceptable real dataset is available, say so and fall back to `synthetic-fallback`.
|
||||
|
||||
## When a run may be called measured
|
||||
|
||||
A run may be marked `measured` only if:
|
||||
|
||||
- the code actually ran on non-simulated data,
|
||||
- the dataset contract names the source and split,
|
||||
- `results.json` provenance points to the measured run,
|
||||
- the report and paper can trace the numbers back to this run.
|
||||
|
||||
Measured does not mean perfect benchmark design. It only means the numbers came from real execution
|
||||
on bound data rather than a simulator.
|
||||
|
||||
## FAIR minimum for this run
|
||||
|
||||
For experiment-suite purposes, the minimum FAIR-style bar is:
|
||||
|
||||
- the dataset source is findable from the contract,
|
||||
- the access condition is explicit,
|
||||
- the version or snapshot is recorded,
|
||||
- the files used for train/val/test are named,
|
||||
- the preprocessing boundary is stated clearly enough for another engineer to rerun.
|
||||
|
||||
## Red flags
|
||||
|
||||
- `data.py` reads local files but `data_contract.md` never says what they are
|
||||
- dataset source is only "open-source data" with no stable route
|
||||
- no split rule is written down
|
||||
- controlled data are treated as openly shareable
|
||||
- report says measured, but provenance still points to a simulator
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
# Code-Skeleton Quality
|
||||
|
||||
## Why this exists
|
||||
|
||||
The agent writes a runnable code package under `$RUN/experiment/`: `model.py`, `data.py`, `train.py`, `evaluate.py`, `config.yaml`, `requirements.txt`, `README.md`. Stubs (`nn.Linear(1, 1)`, random-tensor data loaders) are not acceptable — this reference defines the bar the code must clear before delivery.
|
||||
|
||||
**Hard target:** an engineer with the user's data should be able to launch `python train.py --config config.yaml` and have the script train *something* that responds to the data, log per-epoch loss, save a checkpoint, and evaluate on a held-out split. It need not match SOTA — it must be *real code* not stub fragments.
|
||||
|
||||
## Required modules and their shape
|
||||
|
||||
### `model.py`
|
||||
|
||||
A real model class for the task type. Examples:
|
||||
|
||||
- For Transformer time-series forecasting: a small encoder-only Transformer with patched tokenisation, ≤ 200 lines.
|
||||
- For image classification: a small ConvNet or a 4-block ViT, ≤ 200 lines.
|
||||
- For NLP: a 2-layer encoder over BPE tokens, ≤ 150 lines.
|
||||
|
||||
```python
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
class PatchedTransformer(nn.Module):
|
||||
"""Minimal channel-independent patched Transformer for long-horizon forecasting."""
|
||||
def __init__(self, context_len: int, horizon: int, patch_len: int = 16,
|
||||
stride: int = 8, d_model: int = 128, n_heads: int = 4,
|
||||
n_layers: int = 3, dropout: float = 0.1):
|
||||
super().__init__()
|
||||
self.patch_len = patch_len
|
||||
self.stride = stride
|
||||
n_patches = (context_len - patch_len) // stride + 1
|
||||
|
||||
self.embed = nn.Linear(patch_len, d_model)
|
||||
self.pos = nn.Parameter(torch.zeros(1, n_patches, d_model))
|
||||
encoder_layer = nn.TransformerEncoderLayer(
|
||||
d_model=d_model, nhead=n_heads, dropout=dropout,
|
||||
batch_first=True, norm_first=True,
|
||||
)
|
||||
self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=n_layers)
|
||||
self.head = nn.Linear(n_patches * d_model, horizon)
|
||||
|
||||
def forward(self, x): # x: [B, L, C]
|
||||
B, L, C = x.shape
|
||||
x = x.permute(0, 2, 1) # [B, C, L]
|
||||
x = x.unfold(-1, self.patch_len, self.stride) # [B, C, N, P]
|
||||
x = x.reshape(B * C, -1, self.patch_len) # [B*C, N, P]
|
||||
z = self.embed(x) + self.pos # [B*C, N, d]
|
||||
h = self.encoder(z) # [B*C, N, d]
|
||||
out = self.head(h.reshape(B * C, -1)) # [B*C, H]
|
||||
return out.reshape(B, C, -1).permute(0, 2, 1) # [B, H, C]
|
||||
```
|
||||
|
||||
Things that count as **placeholders to remove**: `nn.Linear(1, 1)`, `def forward(self, x): return x`, `# TODO: implement` comments, `pass` bodies.
|
||||
|
||||
### `data.py`
|
||||
|
||||
A real data loader that:
|
||||
- Reads from a configurable path (`config.yaml`)
|
||||
- May be generated from an agent-discovered public dataset or a user-supplied dataset for this run
|
||||
- Applies the pre-processing the design calls for (normalisation, windowing)
|
||||
- Returns `torch.utils.data.Dataset` and `DataLoader` with proper train/val/test split
|
||||
- Falls back to a small synthetic dataset (sinusoid + noise, MNIST-like, etc.) if no real data is provided, with a clear log line
|
||||
|
||||
```python
|
||||
import os
|
||||
import torch
|
||||
from torch.utils.data import Dataset, DataLoader
|
||||
|
||||
class TimeSeriesWindowDataset(Dataset):
|
||||
def __init__(self, data_path: str, context_len: int, horizon: int,
|
||||
split: str = "train", normalise: bool = True):
|
||||
...
|
||||
def __len__(self): ...
|
||||
def __getitem__(self, idx): ...
|
||||
|
||||
def make_loaders(cfg) -> tuple[DataLoader, DataLoader, DataLoader]:
|
||||
"""Return (train, val, test) loaders. Falls back to synthetic if cfg.data_path is None."""
|
||||
...
|
||||
```
|
||||
|
||||
Synthetic fallback should be flagged as such — the train.py log says "WARNING: synthetic data".
|
||||
|
||||
`data.py` is a runtime artifact, not a repository-global benchmark binding. For each run, it should
|
||||
match the dataset declared in `$RUN/data_contract.md`.
|
||||
|
||||
### `train.py`
|
||||
|
||||
A real train loop, **not** a placeholder:
|
||||
|
||||
```python
|
||||
import argparse
|
||||
import yaml
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from torch.optim import AdamW
|
||||
|
||||
from data import make_loaders
|
||||
from model import PatchedTransformer
|
||||
|
||||
def train(cfg):
|
||||
train_loader, val_loader, _ = make_loaders(cfg)
|
||||
model = PatchedTransformer(**cfg["model"])
|
||||
opt = AdamW(model.parameters(), lr=cfg["optim"]["lr"],
|
||||
weight_decay=cfg["optim"]["weight_decay"])
|
||||
loss_fn = nn.MSELoss()
|
||||
|
||||
best_val = float("inf")
|
||||
for epoch in range(cfg["training"]["epochs"]):
|
||||
model.train()
|
||||
for x, y in train_loader:
|
||||
opt.zero_grad()
|
||||
yhat = model(x)
|
||||
loss = loss_fn(yhat, y)
|
||||
loss.backward()
|
||||
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
||||
opt.step()
|
||||
val_loss = evaluate(model, val_loader, loss_fn)
|
||||
print(f"epoch {epoch:03d} train={loss.item():.4f} val={val_loss:.4f}")
|
||||
if val_loss < best_val:
|
||||
best_val = val_loss
|
||||
torch.save(model.state_dict(), cfg["output"]["checkpoint"])
|
||||
|
||||
@torch.no_grad()
|
||||
def evaluate(model, loader, loss_fn):
|
||||
model.eval()
|
||||
total, n = 0.0, 0
|
||||
for x, y in loader:
|
||||
yhat = model(x)
|
||||
total += loss_fn(yhat, y).item() * len(x)
|
||||
n += len(x)
|
||||
return total / max(n, 1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", default="config.yaml")
|
||||
args = parser.parse_args()
|
||||
with open(args.config) as f:
|
||||
cfg = yaml.safe_load(f)
|
||||
train(cfg)
|
||||
```
|
||||
|
||||
### `evaluate.py`
|
||||
|
||||
A separate script that loads the saved checkpoint and computes the design's metrics on the test split. Outputs `results.json` with the schema defined in `references/03-results-protocol.md`.
|
||||
|
||||
### `config.yaml`
|
||||
|
||||
Real values, not `null`:
|
||||
|
||||
```yaml
|
||||
data:
|
||||
path: ./data/ETTm1.csv
|
||||
context_len: 720
|
||||
horizon: 96
|
||||
normalise: true
|
||||
model:
|
||||
context_len: 720
|
||||
horizon: 96
|
||||
patch_len: 16
|
||||
stride: 8
|
||||
d_model: 128
|
||||
n_heads: 4
|
||||
n_layers: 3
|
||||
dropout: 0.1
|
||||
optim:
|
||||
lr: 0.0003
|
||||
weight_decay: 0.0
|
||||
training:
|
||||
epochs: 30
|
||||
batch_size: 32
|
||||
seed: 42
|
||||
output:
|
||||
checkpoint: ./checkpoint.pt
|
||||
results: ./results.json
|
||||
```
|
||||
|
||||
The exact keys may differ by task. The important part is that the generated config binds to the
|
||||
current run's dataset contract instead of silently assuming one baked-in dataset.
|
||||
|
||||
### `requirements.txt`
|
||||
|
||||
Pinned versions of what `model.py`, `data.py`, `train.py` actually import:
|
||||
|
||||
```
|
||||
torch>=2.0
|
||||
numpy>=1.24
|
||||
pyyaml>=6.0
|
||||
pandas>=2.0
|
||||
```
|
||||
|
||||
Don't dump every PyData package; only what the code uses.
|
||||
|
||||
### `README.md`
|
||||
|
||||
A real README that:
|
||||
- States the experiment's task and dataset assumption in 2 sentences
|
||||
- Names the model briefly (architecture summary)
|
||||
- Documents how to install (`pip install -r requirements.txt`)
|
||||
- Documents how to launch (`python train.py --config config.yaml`)
|
||||
- States the expected wall-clock per configuration on the user's hardware
|
||||
- Names the synthetic fallback and disclaims that the package is a starting point, not a SOTA reproduction
|
||||
|
||||
## Testability
|
||||
|
||||
Before declaring the code skeleton complete:
|
||||
|
||||
```bash
|
||||
cd output/experiment-suite/<slug>/experiment
|
||||
python -c "import importlib; m = importlib.import_module('model'); print(m.__name__)"
|
||||
python -c "import importlib; m = importlib.import_module('data'); print(m.__name__)"
|
||||
python -c "import importlib; m = importlib.import_module('train'); print(m.__name__)"
|
||||
python -c "import importlib; m = importlib.import_module('evaluate'); print(m.__name__)"
|
||||
```
|
||||
|
||||
All four must import without error. (You don't need to run training — that costs hours; importing is the cheap correctness check.)
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Stub bodies** (`pass`, `return x`, `# TODO`).
|
||||
- **Hard-coded fake data** in `data.py` without a `path:` config option.
|
||||
- **No training loop in train.py**, just a `print("Training...")`.
|
||||
- **`requirements.txt` listing 50 packages**, most unused.
|
||||
- **README that doesn't say how to run.**
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] `model.py` has a real model class for the task; no `pass` / `TODO`
|
||||
- [ ] `data.py` reads from a config path; falls back to clearly-labelled synthetic
|
||||
- [ ] `train.py` has a real loop with optimizer, loss, validation, checkpointing
|
||||
- [ ] `evaluate.py` loads checkpoint and emits `results.json` matching the schema
|
||||
- [ ] `config.yaml` has real values
|
||||
- [ ] `requirements.txt` matches actual imports
|
||||
- [ ] `README.md` documents install + launch + hardware expectation
|
||||
- [ ] All four Python modules import cleanly
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
# Results Protocol
|
||||
|
||||
## Why this exists
|
||||
|
||||
`results.json` is the structured artefact that figures and reports build on. Whether the agent writes a deterministic simulated JSON or loads a measured JSON from the user's run, the schema must be uniform and the `provenance` field must be explicit. This reference defines the schema and the discipline.
|
||||
|
||||
## Required schema
|
||||
|
||||
```json
|
||||
{
|
||||
"task": "long-horizon time-series forecasting",
|
||||
"dataset": "ETTm1",
|
||||
"horizons": [96, 192, 336, 720],
|
||||
"metrics": ["MSE", "MAE"],
|
||||
"seeds": [42, 43, 44],
|
||||
"provenance": {
|
||||
"mode": "simulated|measured",
|
||||
"source": "simulator|loaded from <path>|measured run on <hardware>",
|
||||
"timestamp": "2026-05-09T14:30:00Z",
|
||||
"git_sha": "abc1234"
|
||||
},
|
||||
"summary": {
|
||||
"<method_name>": {
|
||||
"<metric>": {
|
||||
"<horizon>": {
|
||||
"mean": 0.354,
|
||||
"std": 0.008,
|
||||
"n_seeds": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ablation": {
|
||||
"<variant_name>": {
|
||||
"<metric>": {
|
||||
"<horizon>": {
|
||||
"mean": 0.391,
|
||||
"std": 0.011,
|
||||
"n_seeds": 3,
|
||||
"delta_vs_full": 0.037
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"training_curves": {
|
||||
"<method_name>": {
|
||||
"train_loss": [0.612, 0.488, ...],
|
||||
"val_loss": [0.589, 0.471, ...]
|
||||
}
|
||||
},
|
||||
"notes": "Free-form caveats: dataset version, hyperparameter search budget, hardware."
|
||||
}
|
||||
```
|
||||
|
||||
Not every experiment needs all blocks — `training_curves` is optional, `ablation` is optional if the design didn't include one. But `task`, `dataset`, `metrics`, `seeds`, `provenance`, and `summary` are required.
|
||||
|
||||
## Provenance discipline
|
||||
|
||||
The `provenance` block is non-negotiable.
|
||||
|
||||
### Simulated mode
|
||||
|
||||
```json
|
||||
"provenance": {
|
||||
"mode": "simulated",
|
||||
"source": "experiment-suite/scripts/simulator.py",
|
||||
"timestamp": "2026-05-09T14:30:00Z",
|
||||
"note": "Numbers are randomly generated to mimic plausible value ranges; not measured."
|
||||
}
|
||||
```
|
||||
|
||||
### Measured mode (user supplied)
|
||||
|
||||
```json
|
||||
"provenance": {
|
||||
"mode": "measured",
|
||||
"source": "loaded from /home/user/runs/2026-05-08/results.json",
|
||||
"timestamp": "2026-05-09T14:30:00Z",
|
||||
"hardware": "1× A100 80GB",
|
||||
"n_configs": 24,
|
||||
"wall_clock_hours": 38.5
|
||||
}
|
||||
```
|
||||
|
||||
### Measured mode (agent-discovered or agent-bound data)
|
||||
|
||||
When the agent itself found and bound an open dataset for this run, the provenance should still
|
||||
name the actual run source and reference the bound dataset contract:
|
||||
|
||||
```json
|
||||
"provenance": {
|
||||
"mode": "measured",
|
||||
"source": "measured run from experiment/train.py using dataset bound in data_contract.md",
|
||||
"dataset_contract": "./data_contract.md",
|
||||
"timestamp": "2026-05-09T14:30:00Z",
|
||||
"hardware": "1× A100 80GB"
|
||||
}
|
||||
```
|
||||
|
||||
### Mixed mode (some real, some simulated)
|
||||
|
||||
When part of the table is measured and part is simulated (e.g., baselines were run, but the new method was not), the `provenance` block becomes a mapping:
|
||||
|
||||
```json
|
||||
"provenance": {
|
||||
"mode": "mixed",
|
||||
"by_method": {
|
||||
"DLinear": {"mode": "measured", "source": "..."},
|
||||
"PatchTST": {"mode": "measured", "source": "..."},
|
||||
"Ours": {"mode": "simulated", "source": "simulator"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Mixed mode is a smell — surface it loudly in the report and the disclosure footnote, and consider whether the experiment is worth publishing in this state.
|
||||
|
||||
### Auto-measured mode (sandbox / Jupyter execution)
|
||||
|
||||
When the agent uses `sandbox_execute` or Jupyter MCP to run the experiment code
|
||||
automatically, the results are "auto-measured" — they come from real code
|
||||
execution, not from the agent's imagination, but the execution environment
|
||||
may differ from a production research setup.
|
||||
|
||||
```json
|
||||
"provenance": {
|
||||
"mode": "measured",
|
||||
"source": "sandbox_execute via SANDBOX_API_BASE",
|
||||
"execution_time": 15.3,
|
||||
"timestamp": "2026-07-08T14:30:00Z",
|
||||
"environment_note": "Remote sandbox — may differ from local hardware"
|
||||
}
|
||||
```
|
||||
|
||||
Auto-measured results carry the same weight as user-measured results for
|
||||
*correctness* (the code really ran), but the report should note the execution
|
||||
environment when reporting wall-clock times or hardware-dependent metrics.
|
||||
|
||||
The three provenance modes in order of reliability:
|
||||
|
||||
| Mode | Source | Disclosure |
|
||||
| --- | --- | --- |
|
||||
| `simulated` | Agent-generated deterministic values | Must say "simulated" in every artifact |
|
||||
| `measured` (auto) | `sandbox_execute` or Jupyter MCP | Note execution environment |
|
||||
| `measured` (user) | User ran `train.py` themselves | Gold standard |
|
||||
|
||||
## Multi-seed rule
|
||||
|
||||
Each metric/horizon cell should have:
|
||||
- `mean`, `std` over all seeds
|
||||
- `n_seeds` ≥ 3 (otherwise the std is meaningless)
|
||||
- Per-seed individual values stored under a separate top-level `runs` block if the user / reviewer wants them
|
||||
|
||||
If the experiment was run on a single seed, mark `n_seeds: 1` and `std: null` and acknowledge the limitation in the report.
|
||||
|
||||
## Writing a clean simulated results.json
|
||||
|
||||
When the agent writes the simulated JSON itself (no upstream simulator anymore), keep it minimal and well-formed:
|
||||
|
||||
1. Keep `summary`, `seeds`, and create the `provenance` block.
|
||||
2. Move per-seed details to `runs` if useful, drop if not.
|
||||
3. Remove the long disclaimer string from the JSON (it's prose; put it in the report).
|
||||
4. Add `task`, `dataset`, `horizons`, `metrics` if the simulator didn't.
|
||||
5. Ensure all numeric values are JSON numbers, not strings.
|
||||
|
||||
## Validation
|
||||
|
||||
Before declaring results complete:
|
||||
|
||||
```python
|
||||
import json
|
||||
with open("results.json") as f:
|
||||
r = json.load(f)
|
||||
required = {"task", "dataset", "metrics", "seeds", "provenance", "summary"}
|
||||
missing = required - set(r.keys())
|
||||
assert not missing, f"missing fields: {missing}"
|
||||
assert r["provenance"].get("mode") in {"simulated", "measured", "mixed"}
|
||||
for method, by_metric in r["summary"].items():
|
||||
for metric, value in by_metric.items():
|
||||
# value can be a flat dict (single horizon) or nested by horizon
|
||||
if isinstance(value, dict) and "mean" not in value:
|
||||
for horizon, cell in value.items():
|
||||
assert "mean" in cell and "std" in cell
|
||||
print("OK")
|
||||
```
|
||||
|
||||
If validation fails, fix the schema before drafting the report or generating figures.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Missing provenance.** A results.json without `provenance` is unusable downstream.
|
||||
- **Single-seed without disclosure.** Either run more seeds or set `n_seeds: 1` and `std: null` and mention it in the report.
|
||||
- **Per-seed variance hidden.** If the std across seeds is large (≥ 10% of the mean), say so in the report; don't bury it.
|
||||
- **Inconsistent units.** "MSE" in one cell and "Mean Squared Error" in another is the same metric — pick one.
|
||||
- **String numbers.** `"mean": "0.354"` breaks downstream tools; always emit JSON numbers.
|
||||
- **Measured with no data binding.** If the numbers came from real execution but there is no
|
||||
`data_contract.md` or equivalent source record, the run is not auditable.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] `task`, `dataset`, `metrics`, `seeds`, `provenance`, `summary` all present
|
||||
- [ ] `provenance.mode` is `simulated`, `measured`, or `mixed`
|
||||
- [ ] Each cell has `mean`, `std`, `n_seeds`
|
||||
- [ ] `n_seeds` ≥ 3 (or single-seed disclosed)
|
||||
- [ ] No leftover simulator placeholder fields (`disclaimer` string, `runs` if not used)
|
||||
- [ ] JSON numbers, not strings
|
||||
- [ ] Validation script above passes
|
||||
@@ -0,0 +1,198 @@
|
||||
# Publication-Grade Figures
|
||||
|
||||
## Why this exists
|
||||
|
||||
The simulator emits one default chart (`fig_01_comparison_<metric>.pdf`) with default matplotlib styling — useful for a status check, far too crude for a real experiment package. This reference defines what to bring it up to.
|
||||
|
||||
**Hard targets:**
|
||||
- ≥ 3 figures total (more if the experiment supports them)
|
||||
- All vector PDF, fonts embedded, publication-grade rcParams
|
||||
- Each figure has a question it answers; that question becomes the caption
|
||||
|
||||
Before plotting, open `references/04a-figure-contract.md` and define the figure contract. The chart serves the claim, not the other way around.
|
||||
|
||||
## File layout
|
||||
|
||||
All figures live under `output/experiment-suite/<slug>/figures/` with a `make_fig_NN_<slug>.py` script alongside the rendered `.pdf`. The script is the source of truth — re-running the script reproduces the figure. Update `manifest.json` when adding figures.
|
||||
|
||||
Naming: `fig_<NN>_<slug>.pdf` and `make_fig_<NN>_<slug>.py`.
|
||||
|
||||
## Publication rcParams
|
||||
|
||||
Apply at the top of every figure script:
|
||||
|
||||
```python
|
||||
plt.rcParams.update({
|
||||
"font.family": "serif",
|
||||
"font.serif": ["Times New Roman", "DejaVu Serif"],
|
||||
"font.size": 9,
|
||||
"axes.labelsize": 9,
|
||||
"axes.titlesize": 10,
|
||||
"xtick.labelsize": 8,
|
||||
"ytick.labelsize": 8,
|
||||
"legend.fontsize": 8,
|
||||
"axes.linewidth": 0.6,
|
||||
"axes.grid": True,
|
||||
"grid.alpha": 0.25,
|
||||
"grid.linewidth": 0.4,
|
||||
"lines.linewidth": 1.4,
|
||||
"lines.markersize": 4.5,
|
||||
"savefig.dpi": 300,
|
||||
"savefig.bbox": "tight",
|
||||
"svg.fonttype": "none",
|
||||
"pdf.fonttype": 42,
|
||||
"ps.fonttype": 42,
|
||||
})
|
||||
```
|
||||
|
||||
`pdf.fonttype = 42` embeds fonts as TrueType — required for journal submission.
|
||||
|
||||
If you have many related methods on one page, prefer a restrained family palette instead of maximal hue separation.
|
||||
|
||||
## Color palette
|
||||
|
||||
```python
|
||||
COLORS = ["#1f77b4", "#ff7f0e", "#2ca02c", "#d62728", "#9467bd", "#8c564b"]
|
||||
```
|
||||
|
||||
Never use the default matplotlib cycle (perceptually uneven on screen; clashes in print).
|
||||
|
||||
Alternative unified palette for method families:
|
||||
|
||||
```python
|
||||
PALETTE_NMI_PASTEL = [
|
||||
"#484878", "#7884B4", "#B4C0E4", "#E4E4F0", "#E4CCD8", "#F0C0CC"
|
||||
]
|
||||
```
|
||||
|
||||
## Required figure families
|
||||
|
||||
### Family 1 — Method comparison (bar or line)
|
||||
|
||||
If one panel is the main paper claim, make it the hero panel. Supporting panels should not get equal visual weight by default.
|
||||
|
||||
Bar chart for a single metric across methods:
|
||||
|
||||
```python
|
||||
methods = ["DLinear", "PatchTST", "iTransformer", "Ours"]
|
||||
mse = [0.382, 0.366, 0.359, 0.354]
|
||||
err = [0.012, 0.009, 0.008, 0.007]
|
||||
|
||||
fig, ax = plt.subplots(figsize=(3.6, 2.6))
|
||||
x = np.arange(len(methods))
|
||||
ax.bar(x, mse, yerr=err, capsize=3, color=COLORS[:len(methods)],
|
||||
edgecolor="black", linewidth=0.5)
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels(methods, rotation=15, ha="right")
|
||||
ax.set_ylabel("MSE on ETTm1 (lower is better)")
|
||||
ax.set_ylim(0.34, 0.40)
|
||||
ax.spines["top"].set_visible(False)
|
||||
ax.spines["right"].set_visible(False)
|
||||
```
|
||||
|
||||
Or a horizon-sweep line plot if the experiment has a horizon axis.
|
||||
|
||||
### Family 2 — Ablation breakdown
|
||||
|
||||
Either a panel-of-bars per ablation row, or a small-multiples plot per dataset showing full-vs-ablated:
|
||||
|
||||
```python
|
||||
fig, axes = plt.subplots(1, len(DATASETS), figsize=(7.0, 2.4), sharey=True)
|
||||
for ax, dataset in zip(axes, DATASETS):
|
||||
ax.bar([0, 1], [full[dataset], no_patch[dataset]],
|
||||
color=[COLORS[0], COLORS[3]], width=0.6)
|
||||
ax.set_xticks([0, 1])
|
||||
ax.set_xticklabels(["Full", r"$-$patching"], fontsize=8)
|
||||
ax.set_title(dataset, fontsize=9)
|
||||
if dataset == DATASETS[0]:
|
||||
ax.set_ylabel("MSE")
|
||||
```
|
||||
|
||||
### Family 3 — Optional: training curves / heatmap / scaling
|
||||
|
||||
Use only when the data supports it:
|
||||
|
||||
- **Training curves** (line plot, train+val per method) — useful when the experiment varies optimisation strategy.
|
||||
- **Heatmap** (method × dataset MSE) — useful when there are 6+ methods and 4+ datasets.
|
||||
- **Scaling plot** (metric vs model size / data size) — useful for a foundation-model story.
|
||||
|
||||
If the data doesn't support a meaningful version of these, skip — empty heatmaps with 4 cells are not figures.
|
||||
|
||||
## Panel labels
|
||||
|
||||
Add lowercase panel labels for multi-panel figures:
|
||||
|
||||
```python
|
||||
ax.text(-0.12, 1.03, "(a)", transform=ax.transAxes,
|
||||
fontsize=10, fontweight="bold", ha="left", va="bottom")
|
||||
```
|
||||
|
||||
Keep the position consistent across panels in the same figure family.
|
||||
|
||||
## Captions
|
||||
|
||||
Every caption stands alone (a reader who only sees the figure should understand it). Aim for 1–3 sentences naming axes, what the panels show, and what the takeaway is.
|
||||
|
||||
For simulated results, the caption ends with `Numbers are simulated.` For measured results, the caption may name the seed count and hardware briefly.
|
||||
|
||||
Prefer direct labels over a detached legend when one or two curves dominate the interpretation and the labels can be placed without ambiguity.
|
||||
|
||||
## Watermarks for simulated mode
|
||||
|
||||
If `results.json` provenance is `"simulated"`, add a discreet watermark to each figure:
|
||||
|
||||
```python
|
||||
ax.text(0.99, 0.02, "simulated",
|
||||
transform=ax.transAxes, ha="right", va="bottom",
|
||||
fontsize=7, color="gray", alpha=0.7, style="italic")
|
||||
```
|
||||
|
||||
This is a redundancy with the caption, on purpose — readers who skim only figures (more common than you'd think) need a visible flag.
|
||||
|
||||
## Export policy
|
||||
|
||||
Primary export remains vector PDF because `paper-writer` reuses it directly. When possible, also export:
|
||||
|
||||
- `.svg` for editable text
|
||||
- `.tiff` at 600 dpi for journal upload
|
||||
|
||||
Do not store absolute paths in `manifest.json`; only basenames.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **3-D bars, drop shadows, gradient fills** — none belong in a research figure.
|
||||
- **Pie / donut charts** — almost never the right choice; never for experimental data.
|
||||
- **Default matplotlib colors** — perceptually uneven; use the palette above.
|
||||
- **Legend overlapping data** — move it or shrink data range.
|
||||
- **Captions that say only "Comparison results"** — say what's being compared and what we learn.
|
||||
- **Watermark missing** in simulated mode — caption without watermark is not enough.
|
||||
- **Raster (.png) instead of vector (.pdf)** — pixelates in print.
|
||||
|
||||
Before final delivery, run the checklist in `references/04b-figure-qa.md`.
|
||||
|
||||
## Manifest update
|
||||
|
||||
After adding a figure, append an entry to `figures/manifest.json` so downstream `paper-writer` consumers see it:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "fig_02_method_comparison",
|
||||
"path_pdf": "fig_02_method_comparison.pdf",
|
||||
"path_png": "fig_02_method_comparison.png",
|
||||
"caption": "MSE comparison across methods on ETTm1 (three-seed mean ± std).",
|
||||
"section": "results"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] ≥ 3 figures (4–6 ideal)
|
||||
- [ ] All publication rcParams applied (font, size, dpi, embedded fonts)
|
||||
- [ ] All figures vector PDF + sibling PNG for preview
|
||||
- [ ] Each figure has a `make_fig_NN_*.py` script alongside
|
||||
- [ ] Each figure referenced by the report and / or paper
|
||||
- [ ] Each caption stands alone
|
||||
- [ ] Simulated figures carry watermark + caption disclaimer
|
||||
- [ ] `manifest.json` lists every figure
|
||||
@@ -0,0 +1,54 @@
|
||||
# Figure Contract For Experiment Packages
|
||||
|
||||
Use this before writing plotting code. The figure is part of the experiment's argument, not decoration after the fact.
|
||||
|
||||
## Required working note
|
||||
|
||||
Write a short contract to `$RUN/figures/figure_contract.md` before generating figures:
|
||||
|
||||
```text
|
||||
Core conclusion:
|
||||
Figure archetype:
|
||||
Target output:
|
||||
Final size:
|
||||
Panel map:
|
||||
a:
|
||||
b:
|
||||
c:
|
||||
Evidence hierarchy:
|
||||
hero evidence:
|
||||
validation evidence:
|
||||
controls / robustness:
|
||||
Statistics needed:
|
||||
Source data needed:
|
||||
Reviewer risk:
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- The core conclusion must be one sentence with a verb.
|
||||
- Every panel must answer a distinct question.
|
||||
- If removing a panel does not weaken the argument, remove or merge it.
|
||||
- Give the main result the clearest panel; do not make every subplot equally important by default.
|
||||
- Keep controls and robustness panels visually quieter than the hero panel.
|
||||
|
||||
## Figure archetypes
|
||||
|
||||
Pick one of these before plotting:
|
||||
|
||||
- `quantitative grid`
|
||||
- `schematic-led composite`
|
||||
- `image plate + quant`
|
||||
- `asymmetric mixed-modality figure`
|
||||
|
||||
For most experiment-suite outputs, the default is `quantitative grid`. If the experiment involves microscopy, pathology, gels, or spatial overlays, use `image plate + quant` instead.
|
||||
|
||||
## Reviewer-risk prompts
|
||||
|
||||
Before plotting, ask these privately:
|
||||
|
||||
- Is the sample split actually comparable across methods?
|
||||
- Are the error bars / intervals defined?
|
||||
- Could a skeptical reader claim cherry-picking?
|
||||
- Does the figure need a direct disclosure marker because the numbers are simulated?
|
||||
- Would a simpler chart communicate the same claim more clearly?
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
# Figure QA For Experiment Packages
|
||||
|
||||
Use this before delivery or before passing figures to `paper-writer`.
|
||||
|
||||
## Export bundle
|
||||
|
||||
The ideal bundle per figure is:
|
||||
|
||||
- `<basename>.pdf` for paper reuse
|
||||
- `<basename>.svg` for editable vector text
|
||||
- `<basename>.tiff` for journal upload when needed
|
||||
- `make_<basename>.py` as the source of truth
|
||||
|
||||
If the environment cannot produce all three, keep PDF mandatory and note the limitation.
|
||||
|
||||
## Editable-text rules
|
||||
|
||||
For matplotlib-based figures:
|
||||
|
||||
```python
|
||||
plt.rcParams["svg.fonttype"] = "none"
|
||||
plt.rcParams["pdf.fonttype"] = 42
|
||||
plt.rcParams["ps.fonttype"] = 42
|
||||
```
|
||||
|
||||
This keeps SVG text editable and PDF text embedded.
|
||||
|
||||
## Minimum QA checklist
|
||||
|
||||
- Figure title / axes / legend readable at final size
|
||||
- No legend overlapping the data
|
||||
- No default matplotlib color cycle
|
||||
- Panel labels consistent across figures
|
||||
- Simulated watermark present iff the run is simulated
|
||||
- Figure caption can stand alone in the report
|
||||
- Statistics in the caption or surrounding prose define mean/std/CI and seed count
|
||||
|
||||
## For image-based experiment figures
|
||||
|
||||
If the figure contains microscopy, pathology, blots, or spatial overlays, also record:
|
||||
|
||||
- raw image source
|
||||
- crop
|
||||
- brightness/contrast handling
|
||||
- scale-bar presence
|
||||
- whether pseudo-coloring was applied
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- 3-D bars
|
||||
- pie charts for quantitative comparison
|
||||
- saturated rainbow heatmaps without justification
|
||||
- repeated legends in every panel
|
||||
- simulated figures without visible disclosure
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
# Experiment Report Structure
|
||||
|
||||
## Why this exists
|
||||
|
||||
The pipeline emits a 30-line `experiment_report.md`:
|
||||
|
||||
```markdown
|
||||
# Experiment Report: <topic>
|
||||
|
||||
**Task:** ...
|
||||
**Dataset:** ...
|
||||
**Data contract:** `data_contract.md`
|
||||
|
||||
## Results Summary
|
||||
| Method | ... |
|
||||
|
||||
## Charts
|
||||
|
||||
|
||||
## Code
|
||||
Runnable code skeleton in `experiment/`.
|
||||
```
|
||||
|
||||
That's a placeholder. A real experiment report is a **standalone deliverable**: a reader who has only the report, none of the code or PDFs, should understand what was done, how, and what was found. This reference defines that.
|
||||
|
||||
**Hard target:** `experiment_report.md` ≥ 800 words, structured into the seven sections below, citing the figures and tables by filename.
|
||||
|
||||
## Required structure
|
||||
|
||||
### 1. Header + provenance disclosure
|
||||
|
||||
The first lines of the report:
|
||||
|
||||
```markdown
|
||||
# Experiment Report: <topic>
|
||||
|
||||
**Authored by:** AI4S Agent (with human review strongly recommended before publication or production use)
|
||||
**Mode:** Simulated (numbers from `experiment-suite/scripts/simulator.py`) ← or "Measured" / "Mixed"
|
||||
**Date:** YYYY-MM-DD
|
||||
**Hardware (if measured):** 1× A100 80GB, 38h wall-clock total
|
||||
**Data contract:** `data_contract.md`
|
||||
```
|
||||
|
||||
If simulated, follow with a 2-sentence prominent disclaimer block:
|
||||
|
||||
```markdown
|
||||
> **Notice:** numerical results in this report are simulated to demonstrate the experiment-suite workflow. Production decisions must replace these numbers with measurements from the released code; see `experiment/README.md` for how to run.
|
||||
```
|
||||
|
||||
### 2. Problem statement
|
||||
|
||||
Re-state the research question and hypothesis from `experiment_design.md` in 1–2 paragraphs. The reader of the report may not have read the design doc; the report stands alone.
|
||||
|
||||
### 3. Design rationale
|
||||
|
||||
A condensed version of `experiment_design.md` §§ 2–6: which datasets, which baselines, which metrics, what the ablation plan was, what the protocol was. Aim for ~300 words; full justification stays in the design doc.
|
||||
|
||||
If the dataset was agent-discovered, say so explicitly and name the repository/accession or stable source route.
|
||||
|
||||
### 4. Method (briefly)
|
||||
|
||||
The model / approach being evaluated. Reference `experiment/model.py` for implementation detail. 1–2 paragraphs is enough.
|
||||
|
||||
### 5. Results
|
||||
|
||||
The substantive section. Two passes:
|
||||
|
||||
**5.a — Headline numbers.** A markdown table summarising the main comparison. Format:
|
||||
|
||||
```markdown
|
||||
| Method | ETTm1 | Electricity | Traffic | Weather |
|
||||
|---|---|---|---|---|
|
||||
| DLinear | 0.382 | 0.166 | 0.434 | 0.241 |
|
||||
| ... | ... | ... | ... | ... |
|
||||
| **Ours** | **0.354** | **0.149** | **0.402** | **0.220** |
|
||||
|
||||
*Three-seed mean. Numbers are simulated.*
|
||||
```
|
||||
|
||||
**5.b — Figures.** Embed each figure with a markdown reference + brief surrounding prose:
|
||||
|
||||
```markdown
|
||||
|
||||
|
||||
*Figure 2.* MSE on ETTm1 across methods, with three-seed error bars. The patched configuration (Ours) is consistently best, with a roughly 8% improvement over DLinear at horizon 720. Numbers are simulated.
|
||||
```
|
||||
|
||||
### 6. Analysis
|
||||
|
||||
Beyond the table: what does the result mean? Aim for 3–5 paragraphs covering:
|
||||
|
||||
- **Where the method wins / loses.** Don't bury the dataset where your method underperforms.
|
||||
- **What the ablation tells us.** Which component carries the gain. Reference Fig 3 / Table 2.
|
||||
- **Cost vs. accuracy trade-off.** Compute / latency relative to baselines.
|
||||
- **Surprises.** Anything that contradicted the design's expected effect direction.
|
||||
|
||||
### 7. Limitations
|
||||
|
||||
A real experiment package has limitations. List 3–5:
|
||||
|
||||
- Datasets / domains not covered
|
||||
- Failure modes the metrics don't capture
|
||||
- Compute / hyperparameter budget constraints
|
||||
- Confounds and how they were (or weren't) controlled
|
||||
|
||||
### 8. Reproduction
|
||||
|
||||
Closing block — how to reproduce the experiment:
|
||||
|
||||
```markdown
|
||||
## Reproduction
|
||||
|
||||
1. Install: `pip install -r experiment/requirements.txt`
|
||||
2. Configure: edit `experiment/config.yaml` to point at your dataset
|
||||
3. Run: `python experiment/train.py --config experiment/config.yaml`
|
||||
4. Evaluate: `python experiment/evaluate.py --checkpoint <path>`
|
||||
|
||||
Expected wall-clock per configuration: ≈ 30 min on 1× A100. Configurations × seeds × ablations = 24 runs ≈ 12 hours.
|
||||
```
|
||||
|
||||
## Length guidance
|
||||
|
||||
| Section | Target words |
|
||||
|---|---|
|
||||
| 1. Header + disclosure | 60–120 |
|
||||
| 2. Problem | 100–200 |
|
||||
| 3. Design rationale | 250–350 |
|
||||
| 4. Method | 100–200 |
|
||||
| 5. Results (5.a + 5.b) | 200–400 |
|
||||
| 6. Analysis | 300–500 |
|
||||
| 7. Limitations | 100–200 |
|
||||
| 8. Reproduction | 80–150 |
|
||||
| **Total** | **≥ 800** |
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Marketing prose.** "Our method achieves state-of-the-art performance across all benchmarks." No claim that's not in the table.
|
||||
- **Hidden simulated mode.** If the report doesn't say "simulated" in the header and again in the figure captions, it's misleading.
|
||||
- **Walls of bullet points.** A real report has paragraphs that argue. Bullets are for enumerating limitations / steps, not for saying things.
|
||||
- **Missing reproduction block.** A reader can't try the experiment without knowing how.
|
||||
- **Figure dump.** Embedding 8 figures with no surrounding prose doesn't help a reader.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] Header includes provenance mode (simulated / measured / mixed)
|
||||
- [ ] Header includes `data_contract.md`
|
||||
- [ ] Disclosure block visible in §1 if simulated
|
||||
- [ ] Problem and hypothesis re-stated (report stands alone)
|
||||
- [ ] Design rationale condensed (datasets / baselines / metrics)
|
||||
- [ ] Method paragraph references `experiment/model.py`
|
||||
- [ ] Headline table with `**Ours**` bolded; "Numbers are simulated" caption if applicable
|
||||
- [ ] Each figure embedded with a brief surrounding paragraph
|
||||
- [ ] Analysis names where the method loses, not just wins
|
||||
- [ ] Limitations explicit (3–5 items)
|
||||
- [ ] Reproduction block with concrete commands and time estimate
|
||||
- [ ] Total ≥ 800 words
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
# Experiment-Suite Quality Gate — Self-Check Before Delivery
|
||||
|
||||
## Why this exists
|
||||
|
||||
This checklist runs **after** drafting and assembling, **before** declaring the experiment package complete. If any gate fails, return to the relevant reference.
|
||||
|
||||
The gate is bright-line. Do not soften the targets to ship.
|
||||
|
||||
## Hard gates (must pass)
|
||||
|
||||
### G1 · Design depth
|
||||
|
||||
```bash
|
||||
test -s experiment_design.md && [ $(wc -w < experiment_design.md) -ge 700 ]
|
||||
test -s data_contract.md
|
||||
```
|
||||
|
||||
Must include the seven sections from `references/01-design-depth.md`: hypothesis, datasets, baselines, metrics, protocol, ablations, limitations.
|
||||
|
||||
### G2 · Code skeleton runs (imports cleanly)
|
||||
|
||||
```bash
|
||||
cd experiment
|
||||
for m in model data train evaluate; do
|
||||
python -c "import importlib; m = importlib.import_module('$m'); print('$m: OK')"
|
||||
done
|
||||
```
|
||||
|
||||
All four must import without error. Then check for placeholders:
|
||||
|
||||
```bash
|
||||
grep -rE "^\s*pass\s*$|# TODO|raise NotImplementedError" experiment/*.py | head
|
||||
# Should be empty (or only inside genuinely-optional hooks).
|
||||
```
|
||||
|
||||
### G3 · `requirements.txt` matches imports
|
||||
|
||||
```bash
|
||||
grep -h "^import \|^from " experiment/*.py | awk '{print $2}' | cut -d. -f1 | sort -u
|
||||
# Compare against requirements.txt — every third-party import should be listed.
|
||||
```
|
||||
|
||||
### G4 · `README.md` has install + launch instructions
|
||||
|
||||
```bash
|
||||
grep -E "pip install|python train.py|python evaluate.py" experiment/README.md
|
||||
```
|
||||
|
||||
Must include all three lines (install, launch, evaluate) or equivalents.
|
||||
|
||||
### G5 · `results.json` schema valid
|
||||
|
||||
```python
|
||||
import json
|
||||
r = json.load(open("results.json"))
|
||||
required = {"task", "dataset", "metrics", "seeds", "provenance", "summary"}
|
||||
assert not (required - set(r.keys())), f"missing: {required - set(r.keys())}"
|
||||
assert r["provenance"].get("mode") in {"simulated", "measured", "mixed"}
|
||||
print("OK")
|
||||
```
|
||||
|
||||
If mode is `measured`, the provenance should also point to `data_contract.md` directly or indirectly.
|
||||
|
||||
### G6 · Figures present and publication-grade
|
||||
|
||||
```bash
|
||||
[ $(ls figures/*.pdf 2>/dev/null | wc -l) -ge 3 ]
|
||||
test -f figures/manifest.json
|
||||
test -f figures/figure_contract.md
|
||||
```
|
||||
|
||||
For each figure:
|
||||
- vector PDF (not raster)
|
||||
- has a sibling `make_fig_*.py` script
|
||||
- listed in `manifest.json` with `id`, `path_pdf`, `caption`, `section`
|
||||
- should also have sibling `.svg` / `.tiff` when the environment supports them
|
||||
|
||||
### G7 · Report depth
|
||||
|
||||
```bash
|
||||
test -s experiment_report.md && [ $(wc -w < experiment_report.md) -ge 800 ]
|
||||
```
|
||||
|
||||
Must contain the seven sections from `references/05-report-structure.md`. Search for section headings:
|
||||
|
||||
```bash
|
||||
grep -E "^## (Problem|Design|Method|Results|Analysis|Limitations|Reproduction)" experiment_report.md | wc -l
|
||||
# Should be ≥ 6
|
||||
```
|
||||
|
||||
### G8 · Provenance disclosed wherever numbers appear
|
||||
|
||||
If `results.json` provenance mode is `simulated`:
|
||||
|
||||
```bash
|
||||
grep -i "simulated" experiment_report.md # must appear in header + figure captions
|
||||
grep -i "simulated" figures/manifest.json # captions should say "Numbers are simulated."
|
||||
```
|
||||
|
||||
If mode is `measured`:
|
||||
|
||||
```bash
|
||||
grep -i "simulated" experiment_report.md # should NOT appear (otherwise inconsistent)
|
||||
grep -i "measured\|three-seed\|hardware" experiment_report.md # should appear in §1 header
|
||||
```
|
||||
|
||||
If mode is `mixed`:
|
||||
|
||||
```bash
|
||||
grep -i "mixed\|partially\|by_method" experiment_report.md # should explicitly explain
|
||||
```
|
||||
|
||||
If mode is `measured` and the data were agent-discovered or reused public data:
|
||||
|
||||
```bash
|
||||
grep -i "data contract" experiment_report.md
|
||||
grep -i "repository\|accession\|doi\|source" data_contract.md
|
||||
```
|
||||
|
||||
## Soft gates (should pass)
|
||||
|
||||
### S1 · No marketing prose in the report
|
||||
|
||||
```bash
|
||||
grep -E "state-of-the-art|cutting-edge|comprehensive|extensive|novel" experiment_report.md | head
|
||||
```
|
||||
|
||||
These phrases aren't categorically wrong but should be sparse and earned.
|
||||
|
||||
### S2 · No hidden numbers
|
||||
|
||||
```bash
|
||||
# Numbers in the report must be traceable to results.json
|
||||
# (Manual check; eyeball that every number in the headline table appears in results.json["summary"].)
|
||||
```
|
||||
|
||||
### S3 · Code is callable end-to-end (smoke)
|
||||
|
||||
For experiments where compute permits:
|
||||
|
||||
```bash
|
||||
cd experiment
|
||||
# Run with --quick or a tiny config to verify the pipeline doesn't crash.
|
||||
# This is optional but a powerful confidence check.
|
||||
```
|
||||
|
||||
### S4 · Visuals make sense
|
||||
|
||||
Open each figure PDF. Check:
|
||||
- Axes labelled, legend readable at print scale
|
||||
- Watermark "simulated" present iff mode is simulated
|
||||
- Colors from the publication palette (no default cycle)
|
||||
- Hero panel / supporting panel hierarchy matches `figures/figure_contract.md`
|
||||
|
||||
### S5 · Multi-seed reporting
|
||||
|
||||
```python
|
||||
import json
|
||||
r = json.load(open("results.json"))
|
||||
n_seeds = r.get("seeds", [])
|
||||
if len(n_seeds) < 3:
|
||||
print(f"WARNING: only {len(n_seeds)} seeds — disclose in report.")
|
||||
```
|
||||
|
||||
## Final report format
|
||||
|
||||
When all gates pass:
|
||||
|
||||
```
|
||||
Experiment package ready: output/experiment-suite/<slug>/
|
||||
|
||||
Stats:
|
||||
Design: 912 words, 7 sections
|
||||
Code: model.py 184 lines, data.py 96 lines, train.py 128 lines, evaluate.py 64 lines
|
||||
Results: schema valid, mode=simulated, 3 seeds, 4 datasets × 4 horizons
|
||||
Figures: 5 (2 method comparison, 1 ablation panel, 1 heatmap, 1 training curves)
|
||||
Report: 1240 words, 8 sections
|
||||
Disclosure: "simulated" present in report header, every figure caption, results.json provenance
|
||||
|
||||
Quality gate: PASSED (G1–G8 hard, S1–S5 soft).
|
||||
```
|
||||
|
||||
If any hard gate cannot honestly be cleared:
|
||||
|
||||
> "Code skeleton imports cleanly but `train.py` has a `NotImplementedError` in the loss-weighting branch — leaving in place because the design called for it as an optional extension; the main loss path is functional."
|
||||
|
||||
That's a legitimate deviation. Padding the design to 700 words with filler is not.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] G1 design ≥ 700 words, all 7 sections present
|
||||
- [ ] `data_contract.md` exists and matches the design / results provenance
|
||||
- [ ] G2 all 4 modules import cleanly; no `pass`/`TODO`/`NotImplementedError` in core paths
|
||||
- [ ] G3 requirements.txt matches actual imports
|
||||
- [ ] G4 README has install + launch + evaluate commands
|
||||
- [ ] G5 results.json schema valid with provenance
|
||||
- [ ] G6 ≥ 3 figures, vector PDF, all in manifest
|
||||
- [ ] Figure contract written before plotting; export bundle present when possible
|
||||
- [ ] G7 report ≥ 800 words, 7 sections
|
||||
- [ ] G8 provenance consistent across report / figures / results.json
|
||||
- [ ] S1–S5 soft gates reviewed
|
||||
- [ ] Honest report — don't pad to clear gates
|
||||
Reference in new issue
Block a user