feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed
1 parent
7acc0c5451
commit
e45b68bb8b
1089 files changed
+260614
No files matched your search
@@ -0,0 +1,153 @@
|
||||
# Incremental Execution — How a Survey Actually Gets Built
|
||||
|
||||
## Why this exists
|
||||
|
||||
A survey with 100+ references and 7 multi-paragraph sections does not fit into a single LLM tool call. WebSearch returns ~10 useful candidates per query, not 100. A single message can write one solid section, not seven. If you treat the build as "do it all in one shot", you will hit context limits, fail mid-task, and have nothing to recover.
|
||||
|
||||
This reference defines the only execution mode that works: **incremental, file-persisted, resumable**. Every other reference (`01`–`05`) describes *what* to produce; this one describes *how* to produce it without losing work.
|
||||
|
||||
## The three invariants
|
||||
|
||||
### Invariant 1 — Filesystem is the source of truth
|
||||
|
||||
Every intermediate result is written to disk **immediately**. A new BibTeX entry is appended to `bibliography.bib` the moment it is extracted from a WebFetch result, not after the batch finishes. A rewritten section is Written to `sections/<name>.tex` the moment it is drafted, not after all seven are drafted. A generated figure is committed to `figures/fig_NN_<name>.pdf` and the matching script is saved alongside.
|
||||
|
||||
If the conversation context fills, the agent crashes, or the user pivots, **the files survive**. Resumption reads the files and continues.
|
||||
|
||||
### Invariant 2 — Progress is observable via filesystem, not memory
|
||||
|
||||
```bash
|
||||
cd output/literature-survey/<slug>/survey_paper
|
||||
grep -c "^@" bibliography.bib # bib entries so far
|
||||
ls figures/*.pdf | wc -l # figures so far
|
||||
for f in sections/*.tex; do
|
||||
echo "$(basename "$f"): $(wc -w < "$f") words"
|
||||
done # per-section drafted size
|
||||
```
|
||||
|
||||
If the count does not match the target, do another batch. Don't rely on remembering what you did.
|
||||
|
||||
### Invariant 3 — Each batch is small enough to succeed, atomic enough to retry
|
||||
|
||||
A "batch" is one WebSearch+WebFetch cycle, or one section's prose, or one figure's script+render. Failure → redo just that batch. Success → write to disk and move on.
|
||||
|
||||
## Batch sizes that work
|
||||
|
||||
| Operation | Batch size per turn | Total batches | Persistence |
|
||||
|---|---|---|---|
|
||||
| Bibliography expansion | 1 WebSearch query → triage → ~5–15 WebFetch'd entries | 10–15 batches to reach 100+ | Append to `bibliography.bib` after each |
|
||||
| Section drafting | 1 section per turn | 7 batches total | Write each `sections/<name>.tex` immediately |
|
||||
| Figure generation | 1 figure per turn (TikZ inline or matplotlib script + render) | 6–10 batches | Save script + render alongside |
|
||||
| Compile + check | 1 cycle after major artefact | Several | `main.log` + grep |
|
||||
|
||||
A turn that tries to do 5 sections produces shallow prose and probably truncates. A turn that tries to "WebSearch all 15 queries at once" hits tool budgets. One per turn is the rhythm.
|
||||
|
||||
## Bibliography expansion — incremental loop
|
||||
|
||||
```
|
||||
read existing bibliography.bib
|
||||
n_entries = grep -c "^@" bibliography.bib
|
||||
target = 110 # aim above the 100 recommended floor (hard gate 60)
|
||||
|
||||
query_plan = [...] # 12–20 angles from 01-bibliography-expansion.md
|
||||
already_run = read_state(.bib_progress) # set of executed query strings
|
||||
seen_urls = extract_urls_from_bib() # already-cited URLs
|
||||
|
||||
for q in query_plan:
|
||||
if q in already_run: continue
|
||||
candidates = WebSearch(q) # ~5–15 hits
|
||||
for c in triage(candidates): # filter by venue, dedup
|
||||
if c.url in seen_urls: continue
|
||||
meta = WebFetch(c.url) # canonical title/authors/year/venue
|
||||
if meta is bad: continue
|
||||
append_bibtex(meta, bibliography.bib)
|
||||
seen_urls.add(c.url)
|
||||
append(.bib_progress, q) # persist that query was run
|
||||
if grep -c "^@" bibliography.bib >= target: break
|
||||
```
|
||||
|
||||
In Claude Code execution this is **one WebSearch + one or more WebFetch + Edit/Write per turn** for ~15 turns. If turn 7 fails, turn 8 reads the progress file, sees queries 1–6 are done, restarts from 7. **No work lost.**
|
||||
|
||||
## Section drafting — one section per turn
|
||||
|
||||
Order: introduction → background → methods (themed survey) → discussion → conclusion → related work → abstract.
|
||||
|
||||
For each section:
|
||||
|
||||
1. Read `references/03-survey-section-playbook.md` rules for that section (open it fresh — don't trust cached recollection).
|
||||
2. Read `bibliography.bib` and pick relevant keys for this section.
|
||||
3. Draft the section in one go (one section is 600–1500 words for a survey; a single LLM turn can comfortably produce that).
|
||||
4. **Immediately** Write the result to `sections/<name>.tex`.
|
||||
5. Compile (`pdflatex` once + `bibtex` + `pdflatex` ×2). Check `main.log` for new undefined citations.
|
||||
6. Fix errors before moving to the next section.
|
||||
|
||||
### "Abstract last" rule
|
||||
|
||||
The abstract names the field's structure as you have organised it. Until you have actually organised it (in Background through Discussion), you can't write a good abstract. Drafting it last takes 5 minutes; drafting it first wastes 30.
|
||||
|
||||
## Figure generation — one figure per turn
|
||||
|
||||
For each figure (6–10 total):
|
||||
|
||||
1. Open `references/02-survey-figures.md` for the family that fits this figure (taxonomy / timeline / matrix / architecture / quantitative).
|
||||
2. Decide what the figure *answers* in the survey. That answer becomes the caption.
|
||||
3. Write the figure (TikZ inline in the relevant `sections/*.tex`, or a `figures/make_fig_NN_*.py` script).
|
||||
4. Render (TikZ compiles via main.tex; matplotlib via `cd figures && python make_fig_NN_*.py`).
|
||||
5. Verify the PDF exists and is non-trivial.
|
||||
6. Reference it from the appropriate section with a caption.
|
||||
7. Recompile and check.
|
||||
|
||||
## Recovery — what to do when a session resumes mid-task
|
||||
|
||||
```bash
|
||||
cd output/literature-survey/<slug>/survey_paper
|
||||
|
||||
grep -c "^@" bibliography.bib # bib progress
|
||||
cat .bib_progress.txt 2>/dev/null | wc -l # queries executed
|
||||
for f in sections/*.tex; do
|
||||
echo "$(basename "$f"): $(wc -w < "$f") words"
|
||||
done # which sections drafted
|
||||
ls figures/*.pdf 2>/dev/null
|
||||
ls -la main.pdf
|
||||
grep -c "Citation .* undefined" main.log
|
||||
```
|
||||
|
||||
Resume the sub-step that's incomplete. **Never** redo work that's already on disk; only fill in what's missing.
|
||||
|
||||
## Checkpoint files
|
||||
|
||||
Three small bookkeeping files in the survey paper directory:
|
||||
|
||||
- `.bib_progress.txt` — one line per executed query string
|
||||
- `.section_progress.txt` — one line per drafted section
|
||||
- `.figure_progress.txt` — one line per generated figure id
|
||||
|
||||
```bash
|
||||
echo "transformer time series Informer Autoformer" >> .bib_progress.txt
|
||||
```
|
||||
|
||||
## Atomic write rule
|
||||
|
||||
Use Write/Edit tools (atomic). Never stream partial content via `python -c "open(...).write(...)"`. Don't parallelize bib appends — they must serialise.
|
||||
|
||||
## Sanity check before delivery
|
||||
|
||||
```bash
|
||||
ls -la sections/ # all 7 .tex files non-trivial?
|
||||
ls -la figures/*.pdf # 6+ figure PDFs present?
|
||||
test -s main.pdf
|
||||
grep -c "^@" bibliography.bib # ≥ 60 (100+ recommended)
|
||||
total=0; for f in sections/*.tex; do
|
||||
total=$((total + $(grep -o "\\\\cite{" "$f" | wc -l)))
|
||||
done
|
||||
echo "TOTAL cites: $total" # ≥ 60 across all sections
|
||||
```
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] Bibliography assembled in **12+ small WebSearch+WebFetch batches**, one query per turn
|
||||
- [ ] Sections written **one per turn**, abstract last
|
||||
- [ ] Figures generated **one per turn**
|
||||
- [ ] Three checkpoint files maintained
|
||||
- [ ] Compile+verify after each major artefact
|
||||
- [ ] Never tried to "do it all in one turn"
|
||||
Reference in new issue
Block a user