Z User
e45b68bb8b
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
125 lines
5.9 KiB
Markdown
Executable File
125 lines
5.9 KiB
Markdown
Executable File
# Experiment Design Depth
|
||
|
||
## Why this exists
|
||
|
||
The pipeline emits a 5-line design stub like:
|
||
```
|
||
**Task:** Transformer-based time series forecasting
|
||
**Dataset:** Synthetic dataset
|
||
**Methods:** Baseline A, Baseline B
|
||
**Metrics:** Accuracy, F1-Score, Runtime
|
||
```
|
||
|
||
That's not a design — it's a placeholder. A real experiment design tells someone else why your decisions are the right ones. This reference defines what a real design contains.
|
||
|
||
**Hard target:** `experiment_design.md` ≥ 700 words, structured into the seven sections below. Every choice (dataset, baseline, metric, ablation) carries a justification, not just a name.
|
||
|
||
## Required structure
|
||
|
||
### 1. Research question and hypothesis
|
||
|
||
State the question precisely (one paragraph) and convert it into a falsifiable hypothesis. The hypothesis should name the expected effect direction and a rough magnitude.
|
||
|
||
> *We test whether channel-independent patched attention reduces long-horizon MSE by ≥ 5% on ETT benchmarks compared to point-wise attention, while not increasing compute by more than 1.3×.*
|
||
|
||
If the question doesn't compress into a falsifiable form, weaken it until it does. Vague questions ("how does X work?") produce vague experiments.
|
||
|
||
### 2. Datasets
|
||
|
||
For each dataset:
|
||
- **Name** and **citation**
|
||
- **Source route** — user-supplied, agent-discovered, reused public, controlled, or synthetic fallback
|
||
- **Repository / accession / DOI / stable URL** when available
|
||
- **Why this dataset** — what makes it appropriate for the question (size, domain, structure)
|
||
- **Splits** — train / val / test sizes, how they were obtained
|
||
- **Pre-processing** — normalisation, windowing, encoding
|
||
- **Access / licence condition** — public, controlled, licensed, or local-only
|
||
- **Known limitations** — biases, age, label-noise issues
|
||
|
||
If the experiment uses 1 dataset, justify why one is enough. If 5+, justify why so many.
|
||
|
||
The dataset section must agree with `$RUN/data_contract.md`. Do not let the design and the runtime
|
||
binding drift apart.
|
||
|
||
### 3. Baselines
|
||
|
||
A real comparison needs the right baselines. Group them by family:
|
||
|
||
- **Lower-bound baselines**: dumb baselines (mean, last-value, linear regression). If your method doesn't beat these, something is wrong.
|
||
- **Same-paradigm baselines**: other methods solving the same problem with the same paradigm.
|
||
- **Adjacent-paradigm baselines**: rival paradigms (e.g., for a Transformer paper: linear, MLP, state-space).
|
||
- **Optional zero-shot reference**: foundation models / pretrained baselines.
|
||
|
||
For each baseline cite a primary source, name the implementation used (released code or our re-implementation), and note the hyperparameter budget allocated.
|
||
|
||
### 4. Metrics
|
||
|
||
Three failure modes here:
|
||
|
||
- Reporting too few metrics (one number per method) hides regressions.
|
||
- Reporting too many metrics (10+) spreads the signal thin.
|
||
|
||
Aim for **3–6 metrics** of distinct types: at least one accuracy-like, one cost-like (compute / latency / memory), and one robustness-like (worst-case / variance / OOD).
|
||
|
||
For each metric: name, units, direction (lower / higher better), why it matters for the research question, what value range is meaningful.
|
||
|
||
### 5. Protocol
|
||
|
||
- **Seeds**: minimum 3, ideally 5. Report mean ± std, not single-run numbers.
|
||
- **Hyperparameter search**: same budget per method (e.g., 30 trials). Document the search space, including ranges and distributions.
|
||
- **Hardware**: name the accelerator(s) and approximate wall-clock per configuration.
|
||
- **Implementation**: framework, key library versions, deterministic flags.
|
||
- **Cross-validation**: forecasting is autoregressive — use rolling-origin / time-series CV per~\cite{bergmeir2018cv} (or its equivalent for your task), not random k-fold.
|
||
|
||
### 6. Ablations
|
||
|
||
A method usually has 3–6 components. The ablation plan removes them one at a time on a representative dataset / horizon. List the planned ablations explicitly:
|
||
|
||
| Ablation | What it removes | Expected effect direction | Why it tests our hypothesis |
|
||
|---|---|---|---|
|
||
|
||
Without an ablation plan, the headline number cannot be attributed to any component.
|
||
|
||
### 7. Risk and limitations
|
||
|
||
Be explicit about what the experiment **can't** test:
|
||
|
||
- Datasets / domains the experiment does not cover
|
||
- Failure modes the metrics don't measure
|
||
- Confounds (e.g., compute differences, data-leakage risks)
|
||
- Negative results we'd accept as the hypothesis being wrong
|
||
|
||
A design without limitations is a design that doesn't know its weaknesses.
|
||
|
||
## Compute / time budget
|
||
|
||
End with a small budget table:
|
||
|
||
| Item | Per-config cost | Total |
|
||
|---|---|---|
|
||
| Method × seeds × hyperparameter trials | 30 min × 3 × 30 = 45 hrs | one-time |
|
||
| Ablation re-runs | 30 min × 3 × 6 = 9 hrs | re-used datasets |
|
||
| Total | ≈ 54 hrs on 1 A100 | ~ 2.5 days |
|
||
|
||
This calibrates the user on the cost of running the experiment for real and supports the simulated-vs-measured choice in `references/03-results-protocol.md`.
|
||
|
||
## Anti-patterns
|
||
|
||
- **No hypothesis, just a question** — "how does X compare to Y?" is not a hypothesis. State the expected direction.
|
||
- **Cherry-picked baselines** — picking only the baselines you'll beat. Include baselines that might beat you.
|
||
- **Single seed** — never trust a single-seed number.
|
||
- **No ablation plan** — the headline number with no component attribution is not a finding.
|
||
- **Real-data brag with synthetic data** — if the dataset is synthetic, say so up front in §2.
|
||
|
||
## Quick checklist
|
||
|
||
- [ ] Hypothesis is falsifiable, with direction + rough magnitude
|
||
- [ ] Each dataset has its rationale, splits, pre-processing, limitations
|
||
- [ ] Baselines grouped by family; each has a primary citation and budget
|
||
- [ ] 3–6 metrics covering accuracy + cost + robustness, each with rationale
|
||
- [ ] Protocol explicit: seeds, hyperparameter budget, hardware, framework
|
||
- [ ] Ablation plan with expected effect direction per ablation
|
||
- [ ] Limitations and risks named
|
||
- [ ] Compute budget concrete (hours, not "TBD")
|
||
- [ ] Total length ≥ 700 words
|