Z User
e45b68bb8b
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
203 lines
6.3 KiB
Markdown
Executable File
203 lines
6.3 KiB
Markdown
Executable File
# Experiment-Suite Quality Gate — Self-Check Before Delivery
|
||
|
||
## Why this exists
|
||
|
||
This checklist runs **after** drafting and assembling, **before** declaring the experiment package complete. If any gate fails, return to the relevant reference.
|
||
|
||
The gate is bright-line. Do not soften the targets to ship.
|
||
|
||
## Hard gates (must pass)
|
||
|
||
### G1 · Design depth
|
||
|
||
```bash
|
||
test -s experiment_design.md && [ $(wc -w < experiment_design.md) -ge 700 ]
|
||
test -s data_contract.md
|
||
```
|
||
|
||
Must include the seven sections from `references/01-design-depth.md`: hypothesis, datasets, baselines, metrics, protocol, ablations, limitations.
|
||
|
||
### G2 · Code skeleton runs (imports cleanly)
|
||
|
||
```bash
|
||
cd experiment
|
||
for m in model data train evaluate; do
|
||
python -c "import importlib; m = importlib.import_module('$m'); print('$m: OK')"
|
||
done
|
||
```
|
||
|
||
All four must import without error. Then check for placeholders:
|
||
|
||
```bash
|
||
grep -rE "^\s*pass\s*$|# TODO|raise NotImplementedError" experiment/*.py | head
|
||
# Should be empty (or only inside genuinely-optional hooks).
|
||
```
|
||
|
||
### G3 · `requirements.txt` matches imports
|
||
|
||
```bash
|
||
grep -h "^import \|^from " experiment/*.py | awk '{print $2}' | cut -d. -f1 | sort -u
|
||
# Compare against requirements.txt — every third-party import should be listed.
|
||
```
|
||
|
||
### G4 · `README.md` has install + launch instructions
|
||
|
||
```bash
|
||
grep -E "pip install|python train.py|python evaluate.py" experiment/README.md
|
||
```
|
||
|
||
Must include all three lines (install, launch, evaluate) or equivalents.
|
||
|
||
### G5 · `results.json` schema valid
|
||
|
||
```python
|
||
import json
|
||
r = json.load(open("results.json"))
|
||
required = {"task", "dataset", "metrics", "seeds", "provenance", "summary"}
|
||
assert not (required - set(r.keys())), f"missing: {required - set(r.keys())}"
|
||
assert r["provenance"].get("mode") in {"simulated", "measured", "mixed"}
|
||
print("OK")
|
||
```
|
||
|
||
If mode is `measured`, the provenance should also point to `data_contract.md` directly or indirectly.
|
||
|
||
### G6 · Figures present and publication-grade
|
||
|
||
```bash
|
||
[ $(ls figures/*.pdf 2>/dev/null | wc -l) -ge 3 ]
|
||
test -f figures/manifest.json
|
||
test -f figures/figure_contract.md
|
||
```
|
||
|
||
For each figure:
|
||
- vector PDF (not raster)
|
||
- has a sibling `make_fig_*.py` script
|
||
- listed in `manifest.json` with `id`, `path_pdf`, `caption`, `section`
|
||
- should also have sibling `.svg` / `.tiff` when the environment supports them
|
||
|
||
### G7 · Report depth
|
||
|
||
```bash
|
||
test -s experiment_report.md && [ $(wc -w < experiment_report.md) -ge 800 ]
|
||
```
|
||
|
||
Must contain the seven sections from `references/05-report-structure.md`. Search for section headings:
|
||
|
||
```bash
|
||
grep -E "^## (Problem|Design|Method|Results|Analysis|Limitations|Reproduction)" experiment_report.md | wc -l
|
||
# Should be ≥ 6
|
||
```
|
||
|
||
### G8 · Provenance disclosed wherever numbers appear
|
||
|
||
If `results.json` provenance mode is `simulated`:
|
||
|
||
```bash
|
||
grep -i "simulated" experiment_report.md # must appear in header + figure captions
|
||
grep -i "simulated" figures/manifest.json # captions should say "Numbers are simulated."
|
||
```
|
||
|
||
If mode is `measured`:
|
||
|
||
```bash
|
||
grep -i "simulated" experiment_report.md # should NOT appear (otherwise inconsistent)
|
||
grep -i "measured\|three-seed\|hardware" experiment_report.md # should appear in §1 header
|
||
```
|
||
|
||
If mode is `mixed`:
|
||
|
||
```bash
|
||
grep -i "mixed\|partially\|by_method" experiment_report.md # should explicitly explain
|
||
```
|
||
|
||
If mode is `measured` and the data were agent-discovered or reused public data:
|
||
|
||
```bash
|
||
grep -i "data contract" experiment_report.md
|
||
grep -i "repository\|accession\|doi\|source" data_contract.md
|
||
```
|
||
|
||
## Soft gates (should pass)
|
||
|
||
### S1 · No marketing prose in the report
|
||
|
||
```bash
|
||
grep -E "state-of-the-art|cutting-edge|comprehensive|extensive|novel" experiment_report.md | head
|
||
```
|
||
|
||
These phrases aren't categorically wrong but should be sparse and earned.
|
||
|
||
### S2 · No hidden numbers
|
||
|
||
```bash
|
||
# Numbers in the report must be traceable to results.json
|
||
# (Manual check; eyeball that every number in the headline table appears in results.json["summary"].)
|
||
```
|
||
|
||
### S3 · Code is callable end-to-end (smoke)
|
||
|
||
For experiments where compute permits:
|
||
|
||
```bash
|
||
cd experiment
|
||
# Run with --quick or a tiny config to verify the pipeline doesn't crash.
|
||
# This is optional but a powerful confidence check.
|
||
```
|
||
|
||
### S4 · Visuals make sense
|
||
|
||
Open each figure PDF. Check:
|
||
- Axes labelled, legend readable at print scale
|
||
- Watermark "simulated" present iff mode is simulated
|
||
- Colors from the publication palette (no default cycle)
|
||
- Hero panel / supporting panel hierarchy matches `figures/figure_contract.md`
|
||
|
||
### S5 · Multi-seed reporting
|
||
|
||
```python
|
||
import json
|
||
r = json.load(open("results.json"))
|
||
n_seeds = r.get("seeds", [])
|
||
if len(n_seeds) < 3:
|
||
print(f"WARNING: only {len(n_seeds)} seeds — disclose in report.")
|
||
```
|
||
|
||
## Final report format
|
||
|
||
When all gates pass:
|
||
|
||
```
|
||
Experiment package ready: output/experiment-suite/<slug>/
|
||
|
||
Stats:
|
||
Design: 912 words, 7 sections
|
||
Code: model.py 184 lines, data.py 96 lines, train.py 128 lines, evaluate.py 64 lines
|
||
Results: schema valid, mode=simulated, 3 seeds, 4 datasets × 4 horizons
|
||
Figures: 5 (2 method comparison, 1 ablation panel, 1 heatmap, 1 training curves)
|
||
Report: 1240 words, 8 sections
|
||
Disclosure: "simulated" present in report header, every figure caption, results.json provenance
|
||
|
||
Quality gate: PASSED (G1–G8 hard, S1–S5 soft).
|
||
```
|
||
|
||
If any hard gate cannot honestly be cleared:
|
||
|
||
> "Code skeleton imports cleanly but `train.py` has a `NotImplementedError` in the loss-weighting branch — leaving in place because the design called for it as an optional extension; the main loss path is functional."
|
||
|
||
That's a legitimate deviation. Padding the design to 700 words with filler is not.
|
||
|
||
## Quick checklist
|
||
|
||
- [ ] G1 design ≥ 700 words, all 7 sections present
|
||
- [ ] `data_contract.md` exists and matches the design / results provenance
|
||
- [ ] G2 all 4 modules import cleanly; no `pass`/`TODO`/`NotImplementedError` in core paths
|
||
- [ ] G3 requirements.txt matches actual imports
|
||
- [ ] G4 README has install + launch + evaluate commands
|
||
- [ ] G5 results.json schema valid with provenance
|
||
- [ ] G6 ≥ 3 figures, vector PDF, all in manifest
|
||
- [ ] Figure contract written before plotting; export bundle present when possible
|
||
- [ ] G7 report ≥ 800 words, 7 sections
|
||
- [ ] G8 provenance consistent across report / figures / results.json
|
||
- [ ] S1–S5 soft gates reviewed
|
||
- [ ] Honest report — don't pad to clear gates
|