feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)

Real IMAP connection via imapflow library:
- Connects to IMAP server (IMAPFlow, TLS on port 993)
- Fetches recent messages (configurable limit, default 20)
- Extracts: from, fromEmail, subject, date, snippet (200 chars)
- Thread-to-contact matching (PRD §6.1):
  1. Match sender email against persons table
  2. If no person match, extract domain and match against company.website
  3. Returns matchedCompany + matchedPerson for each email
- Audit-logged (action: crm.mailsync:imap)
- Zod validated (host, port, user, password, folder, limit)
- 502 on IMAP connection failure with detail
- 400 on validation failure

Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed 2026-10-27 22:50:50 +00:00
1 parent 7acc0c5451
commit e45b68bb8b
1089 files changed
+260614

No files matched your search

@@ -0,0 +1,106 @@
# Incremental Execution — How an Experiment Package Actually Gets Built
## Why this exists
A high-quality experiment package has six artefacts (design / code / results / figures / report / quality-checked PDF) and each one is non-trivial. None of this fits into a single LLM tool call. Trying to "do it all in one shot" hits context limits and produces shallow content across the board. This reference defines the only execution mode that works: **incremental, file-persisted, resumable**.
## The three invariants
### Invariant 1 — Filesystem is the source of truth
Each artefact is written to its file the moment it's drafted. The data binding goes to `data_contract.md`, the design to `experiment_design.md`, code to `experiment/<file>.py`, results to `results.json`, figures to `figures/`, report to `experiment_report.md`. If the agent crashes or the user pivots, the files survive.
### Invariant 2 — Progress is observable on disk
```bash
cd output/experiment-suite/<slug>
wc -w experiment_design.md # design length
test -s data_contract.md && echo "data contract present"
wc -l experiment/*.py # code length per module
python -c "import json; d=json.load(open('results.json')); print(d.get('provenance'), '|', list(d.keys()))"
ls figures/*.pdf | wc -l # figure count
wc -w experiment_report.md # report length
```
If any artefact looks thin, return to the relevant reference and expand.
### Invariant 3 — Each batch is small enough to succeed, atomic enough to retry
Batch granularity for this skill:
| Operation | Batch size per turn | Total batches | Persistence |
|---|---|---|---|
| Data binding | 1 contract document | 1 turn | Write `data_contract.md` |
| Design rewrite | 1 design document | 1 turn | Write `experiment_design.md` |
| Code rewrite | 1 module per turn | 4–6 turns (model, data, train, evaluate, config, README) | Write each `experiment/<file>` |
| Results curation | 1 results.json structure | 1 turn | Write `results.json` |
| Figure generation | 1 figure per turn | 3–6 turns | Save script + render + manifest entry |
| Report rewrite | 1–2 sections per turn | 4–6 turns | Edit `experiment_report.md` incrementally |
A turn that tries to write all the code in one shot truncates. A turn that tries 5 figures at once produces shallow figures. One per turn is the rhythm.
## Recovery — when a session resumes mid-task
```bash
cd output/experiment-suite/<slug>
# Design done?
wc -w experiment_design.md # < 200 words means stub; > 600 means real
# Data binding done?
sed -n '1,120p' data_contract.md
# Code done?
for f in experiment/*.py; do echo "$(basename "$f"): $(wc -l < "$f") lines"; done
# Results well-formed?
python -c "import json; d=json.load(open('results.json')); print('OK' if 'summary' in d else 'STUB')"
# Figures present?
ls figures/*.pdf 2>/dev/null
# Report done?
wc -w experiment_report.md
```
Resume the sub-step that's incomplete. Never redo work that's already on disk.
## Checkpoint files
Maintain four small bookkeeping files in the slug dir:
- `.data_progress.txt` — one line per dataset discovery / binding decision
- `.code_progress.txt` — one line per code module written / rewritten
- `.figure_progress.txt` — one line per figure generated
- `.report_progress.txt` — one line per report section drafted
```bash
echo "dataset-bound: GEO/GSE123456" >> .data_progress.txt
echo "model.py" >> .code_progress.txt
echo "fig_02_method_comparison" >> .figure_progress.txt
echo "results-section-drafted" >> .report_progress.txt
```
## Sanity check before delivery
```bash
test -s data_contract.md
test -s experiment_design.md && [ $(wc -w < experiment_design.md) -gt 700 ]
ls experiment/{model,data,train,evaluate}.py
python -c "import json; d=json.load(open('results.json')); assert 'summary' in d, 'no summary'"
[ $(ls figures/*.pdf 2>/dev/null | wc -l) -ge 3 ]
test -s experiment_report.md && [ $(wc -w < experiment_report.md) -gt 800 ]
```
## Quick checklist
- [ ] Data contract written before code generation
- [ ] Design rewritten in 1 dedicated turn (not interleaved)
- [ ] Code rewritten **one module per turn**
- [ ] Results curated in 1 dedicated turn (schema + provenance)
- [ ] Figures generated **one per turn**, scripts saved alongside renders
- [ ] Report drafted **section by section**
- [ ] Four checkpoint files maintained
- [ ] Compile/run check after each major artefact (e.g., `python -c "import experiment.model"` after writing model.py)
- [ ] Never tried to "do it all in one turn"