Z User
e45b68bb8b
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
154 lines
7.5 KiB
Markdown
Executable File
154 lines
7.5 KiB
Markdown
Executable File
# Incremental Execution — How a Survey Actually Gets Built
|
||
|
||
## Why this exists
|
||
|
||
A survey with 100+ references and 7 multi-paragraph sections does not fit into a single LLM tool call. WebSearch returns ~10 useful candidates per query, not 100. A single message can write one solid section, not seven. If you treat the build as "do it all in one shot", you will hit context limits, fail mid-task, and have nothing to recover.
|
||
|
||
This reference defines the only execution mode that works: **incremental, file-persisted, resumable**. Every other reference (`01`–`05`) describes *what* to produce; this one describes *how* to produce it without losing work.
|
||
|
||
## The three invariants
|
||
|
||
### Invariant 1 — Filesystem is the source of truth
|
||
|
||
Every intermediate result is written to disk **immediately**. A new BibTeX entry is appended to `bibliography.bib` the moment it is extracted from a WebFetch result, not after the batch finishes. A rewritten section is Written to `sections/<name>.tex` the moment it is drafted, not after all seven are drafted. A generated figure is committed to `figures/fig_NN_<name>.pdf` and the matching script is saved alongside.
|
||
|
||
If the conversation context fills, the agent crashes, or the user pivots, **the files survive**. Resumption reads the files and continues.
|
||
|
||
### Invariant 2 — Progress is observable via filesystem, not memory
|
||
|
||
```bash
|
||
cd output/literature-survey/<slug>/survey_paper
|
||
grep -c "^@" bibliography.bib # bib entries so far
|
||
ls figures/*.pdf | wc -l # figures so far
|
||
for f in sections/*.tex; do
|
||
echo "$(basename "$f"): $(wc -w < "$f") words"
|
||
done # per-section drafted size
|
||
```
|
||
|
||
If the count does not match the target, do another batch. Don't rely on remembering what you did.
|
||
|
||
### Invariant 3 — Each batch is small enough to succeed, atomic enough to retry
|
||
|
||
A "batch" is one WebSearch+WebFetch cycle, or one section's prose, or one figure's script+render. Failure → redo just that batch. Success → write to disk and move on.
|
||
|
||
## Batch sizes that work
|
||
|
||
| Operation | Batch size per turn | Total batches | Persistence |
|
||
|---|---|---|---|
|
||
| Bibliography expansion | 1 WebSearch query → triage → ~5–15 WebFetch'd entries | 10–15 batches to reach 100+ | Append to `bibliography.bib` after each |
|
||
| Section drafting | 1 section per turn | 7 batches total | Write each `sections/<name>.tex` immediately |
|
||
| Figure generation | 1 figure per turn (TikZ inline or matplotlib script + render) | 6–10 batches | Save script + render alongside |
|
||
| Compile + check | 1 cycle after major artefact | Several | `main.log` + grep |
|
||
|
||
A turn that tries to do 5 sections produces shallow prose and probably truncates. A turn that tries to "WebSearch all 15 queries at once" hits tool budgets. One per turn is the rhythm.
|
||
|
||
## Bibliography expansion — incremental loop
|
||
|
||
```
|
||
read existing bibliography.bib
|
||
n_entries = grep -c "^@" bibliography.bib
|
||
target = 110 # aim above the 100 recommended floor (hard gate 60)
|
||
|
||
query_plan = [...] # 12–20 angles from 01-bibliography-expansion.md
|
||
already_run = read_state(.bib_progress) # set of executed query strings
|
||
seen_urls = extract_urls_from_bib() # already-cited URLs
|
||
|
||
for q in query_plan:
|
||
if q in already_run: continue
|
||
candidates = WebSearch(q) # ~5–15 hits
|
||
for c in triage(candidates): # filter by venue, dedup
|
||
if c.url in seen_urls: continue
|
||
meta = WebFetch(c.url) # canonical title/authors/year/venue
|
||
if meta is bad: continue
|
||
append_bibtex(meta, bibliography.bib)
|
||
seen_urls.add(c.url)
|
||
append(.bib_progress, q) # persist that query was run
|
||
if grep -c "^@" bibliography.bib >= target: break
|
||
```
|
||
|
||
In Claude Code execution this is **one WebSearch + one or more WebFetch + Edit/Write per turn** for ~15 turns. If turn 7 fails, turn 8 reads the progress file, sees queries 1–6 are done, restarts from 7. **No work lost.**
|
||
|
||
## Section drafting — one section per turn
|
||
|
||
Order: introduction → background → methods (themed survey) → discussion → conclusion → related work → abstract.
|
||
|
||
For each section:
|
||
|
||
1. Read `references/03-survey-section-playbook.md` rules for that section (open it fresh — don't trust cached recollection).
|
||
2. Read `bibliography.bib` and pick relevant keys for this section.
|
||
3. Draft the section in one go (one section is 600–1500 words for a survey; a single LLM turn can comfortably produce that).
|
||
4. **Immediately** Write the result to `sections/<name>.tex`.
|
||
5. Compile (`pdflatex` once + `bibtex` + `pdflatex` ×2). Check `main.log` for new undefined citations.
|
||
6. Fix errors before moving to the next section.
|
||
|
||
### "Abstract last" rule
|
||
|
||
The abstract names the field's structure as you have organised it. Until you have actually organised it (in Background through Discussion), you can't write a good abstract. Drafting it last takes 5 minutes; drafting it first wastes 30.
|
||
|
||
## Figure generation — one figure per turn
|
||
|
||
For each figure (6–10 total):
|
||
|
||
1. Open `references/02-survey-figures.md` for the family that fits this figure (taxonomy / timeline / matrix / architecture / quantitative).
|
||
2. Decide what the figure *answers* in the survey. That answer becomes the caption.
|
||
3. Write the figure (TikZ inline in the relevant `sections/*.tex`, or a `figures/make_fig_NN_*.py` script).
|
||
4. Render (TikZ compiles via main.tex; matplotlib via `cd figures && python make_fig_NN_*.py`).
|
||
5. Verify the PDF exists and is non-trivial.
|
||
6. Reference it from the appropriate section with a caption.
|
||
7. Recompile and check.
|
||
|
||
## Recovery — what to do when a session resumes mid-task
|
||
|
||
```bash
|
||
cd output/literature-survey/<slug>/survey_paper
|
||
|
||
grep -c "^@" bibliography.bib # bib progress
|
||
cat .bib_progress.txt 2>/dev/null | wc -l # queries executed
|
||
for f in sections/*.tex; do
|
||
echo "$(basename "$f"): $(wc -w < "$f") words"
|
||
done # which sections drafted
|
||
ls figures/*.pdf 2>/dev/null
|
||
ls -la main.pdf
|
||
grep -c "Citation .* undefined" main.log
|
||
```
|
||
|
||
Resume the sub-step that's incomplete. **Never** redo work that's already on disk; only fill in what's missing.
|
||
|
||
## Checkpoint files
|
||
|
||
Three small bookkeeping files in the survey paper directory:
|
||
|
||
- `.bib_progress.txt` — one line per executed query string
|
||
- `.section_progress.txt` — one line per drafted section
|
||
- `.figure_progress.txt` — one line per generated figure id
|
||
|
||
```bash
|
||
echo "transformer time series Informer Autoformer" >> .bib_progress.txt
|
||
```
|
||
|
||
## Atomic write rule
|
||
|
||
Use Write/Edit tools (atomic). Never stream partial content via `python -c "open(...).write(...)"`. Don't parallelize bib appends — they must serialise.
|
||
|
||
## Sanity check before delivery
|
||
|
||
```bash
|
||
ls -la sections/ # all 7 .tex files non-trivial?
|
||
ls -la figures/*.pdf # 6+ figure PDFs present?
|
||
test -s main.pdf
|
||
grep -c "^@" bibliography.bib # ≥ 60 (100+ recommended)
|
||
total=0; for f in sections/*.tex; do
|
||
total=$((total + $(grep -o "\\\\cite{" "$f" | wc -l)))
|
||
done
|
||
echo "TOTAL cites: $total" # ≥ 60 across all sections
|
||
```
|
||
|
||
## Quick checklist
|
||
|
||
- [ ] Bibliography assembled in **12+ small WebSearch+WebFetch batches**, one query per turn
|
||
- [ ] Sections written **one per turn**, abstract last
|
||
- [ ] Figures generated **one per turn**
|
||
- [ ] Three checkpoint files maintained
|
||
- [ ] Compile+verify after each major artefact
|
||
- [ ] Never tried to "do it all in one turn"
|