feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed
1 parent
7acc0c5451
commit
e45b68bb8b
1089 files changed
+260614
No files matched your search
+303
@@ -0,0 +1,303 @@
|
||||
# Survey Quality Gate — Self-Check Before Delivery
|
||||
|
||||
## Why this exists
|
||||
|
||||
Even a thorough first pass produces a survey with one or two issues — citations that don't resolve, a figure that's too small, a section caption that's a fragment. This checklist runs **after** drafting and recompiling, **before** telling the user "it's done". If any item fails, return to the relevant reference.
|
||||
|
||||
The gate is bright-line. Do not soften the targets to ship.
|
||||
|
||||
## Hard gates (must pass)
|
||||
|
||||
### G1 · PDF compiles cleanly
|
||||
|
||||
```bash
|
||||
cd output/literature-survey/<slug>/survey_paper
|
||||
pdflatex -interaction=nonstopmode main.tex >/dev/null 2>&1
|
||||
bibtex main >/dev/null 2>&1
|
||||
pdflatex -interaction=nonstopmode main.tex >/dev/null 2>&1
|
||||
pdflatex -interaction=nonstopmode main.tex >/dev/null 2>&1
|
||||
|
||||
test -s main.pdf || echo "FAIL: main.pdf missing or empty"
|
||||
echo "undefined cites: $(grep -c 'Citation .* undefined' main.log) (must be 0)"
|
||||
echo "undefined refs: $(grep -c 'Reference .* undefined' main.log) (must be 0)"
|
||||
|
||||
# Overfull boxes = content spilling past the margin (the "elements run off the
|
||||
# page / overlap" complaint). List the worst offenders with how far they overrun:
|
||||
grep 'Overfull \\hbox' main.log | grep -oE '\([0-9.]+pt too wide\)' | sort -t'(' -k2 -rn | head
|
||||
echo "overfull boxes: $(grep -c 'Overfull \\hbox' main.log) (target 0)"
|
||||
```
|
||||
|
||||
**Overfull is a remediation trigger, not a tolerated warning** (unattended runs have no human to eyeball the PDF). Any overfull box > ~2pt means something visibly crosses the margin — fix it, don't ship it:
|
||||
|
||||
- **Figure/table overrun** → it isn't wrapped per `02`/`04`. Wrap TikZ in `\resizebox{\linewidth}{!}{…}`; add `width=\linewidth` to the include; shrink/`tabularx` the table. This alone clears almost all overfull boxes.
|
||||
- **A long unbreakable token** (URL, code, long math) → `\sloppy`, `\url{}`, or manual break.
|
||||
- Recompile and re-grep until the count is 0 (a stray ≤ 1–2pt box from microtype is acceptable; anything larger is not).
|
||||
|
||||
### G2 · Bibliography size
|
||||
|
||||
```bash
|
||||
grep -c "^@" bibliography.bib # must be ≥ 60 (100+ recommended)
|
||||
```
|
||||
|
||||
No fabricated entries:
|
||||
|
||||
```bash
|
||||
# Every entry should have a url= field that the agent fetched in this session
|
||||
grep -c "url = {" bibliography.bib # should be close to total entry count
|
||||
```
|
||||
|
||||
If many entries lack `url=`, you may have leaked memory-sourced entries; audit them.
|
||||
|
||||
### G2.5 · No orphan bib entries
|
||||
|
||||
A bib that contains entries never `\cite{}`-d in the prose is **padding**.
|
||||
The common failure mode is reaching G2's count by appending famous papers
|
||||
from adjacent fields (LLM, vision, foundational deep learning, …) that
|
||||
the survey never uses. This gate detects orphans; **detection is not
|
||||
termination — it triggers remediation**.
|
||||
|
||||
```bash
|
||||
# every key listed in bibliography.bib must appear in at least one \cite,
|
||||
# \citep, \citet, etc. in main.tex or sections/*.tex
|
||||
bib_keys=$(grep -oE '^@[a-z]+\{[^,]+,' bibliography.bib \
|
||||
| sed -E 's/^@[a-z]+\{//; s/,$//' | sort -u)
|
||||
cite_keys=$(grep -hoE '\\cite[a-z]*\{[^}]+\}' main.tex sections/*.tex 2>/dev/null \
|
||||
| grep -oE '\{[^}]+\}' | tr ',' '\n' | tr -d '{} \n' \
|
||||
| sort -u | sed '/^$/d')
|
||||
orphans=$(comm -23 <(echo "$bib_keys") <(echo "$cite_keys"))
|
||||
orphan_count=$(echo "$orphans" | grep -c .)
|
||||
echo "orphan bib entries: $orphan_count (must be 0)"
|
||||
test "$orphan_count" -eq 0 || {
|
||||
echo "ACTION REQUIRED · $orphan_count orphans (first 10):"
|
||||
echo "$orphans" | head -10
|
||||
}
|
||||
```
|
||||
|
||||
**This is a remediation trigger, not a task-failure signal.** If
|
||||
`orphan_count > 0`, DO NOT exit the step or report failure. Loop on
|
||||
remediation until orphans reach 0, in this priority order:
|
||||
|
||||
1. **Cite them in prose** (preferred). Most orphans are real on-topic
|
||||
papers you researched but skipped writing about. Go back to
|
||||
`03-survey-section-playbook.md` and add each to the appropriate
|
||||
section with a one-to-three-sentence treatment. A survey *is* the
|
||||
argument that every cited paper deserves citation. Re-run the gate.
|
||||
2. **Prune + lower scope.** If after honest effort you cannot extend prose
|
||||
to cover the orphans, lower the target toward the 60 floor (100+ recommended)
|
||||
and prune the bib to entries you actually cite. Update the abstract
|
||||
to reflect the narrower scope. Re-run the gate.
|
||||
3. **Prune + honest shortfall report.** If even the focused target
|
||||
cannot be honestly reached, prune to the cited entries and add a
|
||||
note to the introduction / abstract: "Scope limited to N entries
|
||||
due to limited on-topic literature; expanded coverage left to
|
||||
future work." Then re-run the gate. Never quietly pad — but a
|
||||
smaller, honest bib is acceptable output.
|
||||
|
||||
Only after `orphan_count == 0` may you proceed past this gate.
|
||||
|
||||
### G2.6 · Temporal coverage matches the stated intent
|
||||
|
||||
If the topic named a year or recency window (the §0 *recency-led* posture in `01-bibliography-expansion.md`), the bib must actually contain real entries from it. This is the gate that catches "asked for 2026, delivered all 2024".
|
||||
|
||||
```bash
|
||||
# Histogram the bib by year — eyeball it against the recorded posture.
|
||||
grep -oE 'year[[:space:]]*=[[:space:]]*[{"]?[[:space:]]*(19|20)[0-9]{2}' bibliography.bib \
|
||||
| grep -oE '(19|20)[0-9]{2}' | sort | uniq -c
|
||||
```
|
||||
|
||||
There is **no universal threshold** — judge against the posture in `.bib_progress.txt`:
|
||||
|
||||
- **Recency-led** (topic named year Y / "latest"): the bulk of entries fall in the window, and there is a meaningful count *from Y itself*. Zero entries from the named year is an automatic fail; a handful is a warning — search harder before shipping.
|
||||
- **Timeline-spanning / balanced**: the bib reaches the last ~6–12 months, not stopping 2 years short.
|
||||
|
||||
**Detection is a remediation trigger, not a stop.** If coverage doesn't match intent, loop:
|
||||
|
||||
1. **Search harder with date filters.** Re-run the §3.5 date-sorted arXiv queries with the explicit year; the newest papers are almost never in the model's prior knowledge, so relevance-ranked search alone keeps missing them. Add the real entries, recompile, re-check.
|
||||
2. **Honest shortfall note** — only if, after genuine date-filtered searching, the literature in that window is truly thin: add one sentence to the introduction/abstract stating the actual coverage. Never silently substitute old papers for the requested recent ones.
|
||||
|
||||
### G3 · Page count
|
||||
|
||||
```bash
|
||||
pdfinfo main.pdf | grep Pages # must be ≥ 6 (8+ recommended)
|
||||
```
|
||||
|
||||
A survey under 5 pages is a digest, not a survey. Re-scope or expand.
|
||||
|
||||
### G4 · Cite count across sections
|
||||
|
||||
```bash
|
||||
total=0
|
||||
for f in sections/*.tex; do
|
||||
c=$(grep -o "\\\\cite{" "$f" | wc -l)
|
||||
total=$((total + c))
|
||||
done
|
||||
echo "TOTAL \\cite{} markers: $total" # must be ≥ 60
|
||||
```
|
||||
|
||||
Per-section minimums:
|
||||
|
||||
| Section | Min `\cite{` count |
|
||||
|---|---|
|
||||
| abstract | 3 |
|
||||
| introduction | 10 |
|
||||
| background | 6 |
|
||||
| methods | 25 |
|
||||
| discussion | 6 |
|
||||
| conclusion | 3 |
|
||||
| related_work | 5 |
|
||||
|
||||
Methods carries most of the load — that is correct for a survey.
|
||||
|
||||
### G5 · Required figures present
|
||||
|
||||
```bash
|
||||
grep -rh "begin{figure" sections/ | wc -l # must be ≥ 6
|
||||
```
|
||||
|
||||
Specifically, the survey must have:
|
||||
|
||||
- ≥ 1 taxonomy / classification diagram (TikZ-style figure tag in Background or Methods)
|
||||
- ≥ 1 chronological timeline (matplotlib-style figure tag, typically in Introduction)
|
||||
- ≥ 1 coverage / capability matrix (heatmap-style figure tag, typically in Discussion)
|
||||
|
||||
```bash
|
||||
grep -l "fig:taxonomy" sections/*.tex # must list the section that owns the taxonomy fig
|
||||
grep -l "fig:timeline" sections/*.tex
|
||||
grep -l "fig:capability\|fig:coverage" sections/*.tex
|
||||
```
|
||||
|
||||
(These three are the *common* survey figures, not a mandatory trio — per `02`'s design process, let the topic decide. The bar is "every figure makes a claim visible", not "fill these slots".)
|
||||
|
||||
### G5.5 · No template leakage; Nature/Science look
|
||||
|
||||
The figures must be designed for this survey, not copied from the playbook's worked examples (a real run once shipped time-series leaves — `Informer`, `PatchTST` — inside an LLM survey).
|
||||
|
||||
```bash
|
||||
# Sample/example tokens that must NOT appear unless the topic genuinely is time-series:
|
||||
grep -rIE "Informer|Autoformer|FEDformer|PatchTST|iTransformer|Moirai|TimeGPT|Time-Series Forecasting" \
|
||||
sections/ figures/ *.tex 2>/dev/null | grep -v "%" # expect empty for non-TS topics
|
||||
# Figures should be sans-serif (Nature), not the default serif:
|
||||
grep -rn 'font.family.*serif' figures/*.py 2>/dev/null # expect empty (use the sans Nature preamble)
|
||||
# Taxonomies must be forest-based (auto-spaced), not hand-tuned overlapping trees:
|
||||
grep -rn 'sibling distance' sections/*.tex 2>/dev/null # expect empty
|
||||
```
|
||||
|
||||
If any fires: redraw the offending figure for *this* topic with the Nature language in `02` (sans-serif, Wong palette, despined, panel labels, finding-first caption). This is a remediation trigger — fix and re-run, don't ship a leaked or generic-looking figure.
|
||||
|
||||
### G6 · Comparison table present
|
||||
|
||||
```bash
|
||||
grep -rh "begin{table" sections/ | wc -l # must be ≥ 1
|
||||
```
|
||||
|
||||
A survey should have at least one method-comparison table (often spanning the column with `\begin{table*}`).
|
||||
|
||||
### G7 · Disclosure footnote correct
|
||||
|
||||
The `\author{}` block in `main.tex` carries a `\thanks` footnote with the always-on human-review clause:
|
||||
|
||||
```bash
|
||||
grep -E "Human review|human review" main.tex # must match
|
||||
```
|
||||
|
||||
Surveys do **not** include the simulated clause:
|
||||
|
||||
```bash
|
||||
grep -i "simulated" main.tex sections/abstract.tex # should be empty
|
||||
```
|
||||
|
||||
If your survey somehow ended up with the simulated clause, remove it — surveys don't carry numerical experiments to simulate.
|
||||
|
||||
### G8 · Cross-reference integrity
|
||||
|
||||
```bash
|
||||
grep -ohE '\\label\{[^}]+\}' sections/*.tex | sed 's/.*\\label{//;s/}.*//' | sort -u > /tmp/labels
|
||||
grep -ohE '\\ref\{[^}]+\}' sections/*.tex | sed 's/.*\\ref{//;s/}.*//' | sort -u > /tmp/refs
|
||||
diff <(comm -23 /tmp/labels /tmp/refs) <(echo) # labels with no ref → bad
|
||||
diff <(comm -13 /tmp/labels /tmp/refs) <(echo) # refs with no label → bad
|
||||
```
|
||||
|
||||
## Soft gates (should pass)
|
||||
|
||||
### S1 · Notation consistency
|
||||
|
||||
Skim all sections for the same concept under different symbols. Pick one and globally replace.
|
||||
|
||||
### S2 · Caption quality
|
||||
|
||||
Read each caption alone (cover the figure/table). If it doesn't convey what's shown, lengthen.
|
||||
|
||||
### S3 · Anti-pattern sweep
|
||||
|
||||
```bash
|
||||
# Marketing prose
|
||||
grep -E "comprehensive overview|extensive review|cutting-edge|state-of-the-art" sections/*.tex | head
|
||||
# These are not always wrong, but if they're frequent the prose is fluffy.
|
||||
|
||||
grep -E "the authors|This paper presents|This survey provides" sections/*.tex # should be empty / minimal
|
||||
|
||||
# Citation dumps (≥ 6 keys in one \cite{})
|
||||
grep -oE '\\cite\{[^}]+\}' sections/*.tex | awk -F, '{print NF}' | sort -nr | head -3
|
||||
# top number should be ≤ 5
|
||||
```
|
||||
|
||||
### S4 · Prior surveys actually cited
|
||||
|
||||
```bash
|
||||
# Open Related Work; check it cites at least 5 prior surveys by name
|
||||
wc -l sections/related_work.tex
|
||||
```
|
||||
|
||||
A survey that does not engage with prior surveys is rude (and incomplete).
|
||||
|
||||
### S5 · Figures look right
|
||||
|
||||
Unattended runs can't rely on a human opening the PDF, so this is now two layers:
|
||||
|
||||
**Machine check (always):** G1's overfull list above must be empty (or ≤ ~2pt). A figure that overruns the margin shows up there; a figure correctly wrapped in `\resizebox`/`width=\linewidth` cannot. If you have `pdftoppm`/ImageMagick, rasterize the figure pages (`pdftoppm -png -f <p> -l <p> main.pdf /tmp/chk`) and confirm no glyphs touch the page edge.
|
||||
|
||||
**Taste check (when a reviewer is available):** open `main.pdf` and verify:
|
||||
- Taxonomy figure: leaves are readable (not shrunk to illegibility by `\resizebox` — if so, fold the tree); fits cleanly within the column.
|
||||
- Timeline: lane structure visible; labels not overlapping.
|
||||
- Coverage matrix: cell colors readable; legend explained.
|
||||
- Architecture figures (if any): consistent style with the taxonomy.
|
||||
|
||||
## Final report format
|
||||
|
||||
When all gates pass:
|
||||
|
||||
```
|
||||
Survey ready: output/literature-survey/<slug>/survey_paper/main.pdf
|
||||
|
||||
Stats:
|
||||
Pages: 9
|
||||
Bibliography: 112 entries (all url-backed)
|
||||
\cite{} total: 78 across 7 sections
|
||||
Figures: 8 (1 taxonomy, 1 timeline, 1 coverage matrix, 3 architecture, 2 trend)
|
||||
Tables: 2 (capability comparison, paradigm summary)
|
||||
Compile: 0 undefined citations, 0 undefined refs, 0 overfull boxes
|
||||
Coverage: recency-led; 41/58 entries in 2026 window (posture honoured)
|
||||
|
||||
Quality gate: PASSED (G1–G8 hard incl. G2.6 temporal coverage, S1–S5 soft).
|
||||
```
|
||||
|
||||
If you cannot pass G1–G4 without compromising honesty, **say so explicitly**:
|
||||
|
||||
> "Bibliography stalled at 52 entries — could not find more high-quality citations on this niche topic via WebFetch. Page count 5. The survey is delivery-quality at this scope; raising to 100 cites would require padding with low-relevance entries that hurt readability."
|
||||
|
||||
That's a legitimate stop. Fabricating entries to clear the gate is not.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] G1 compile clean; 0 overfull boxes (remediate, don't tolerate)
|
||||
- [ ] G2 bibliography ≥ 60 (100+ recommended); every entry has `url=`
|
||||
- [ ] G2.6 temporal coverage matches the §0 posture (recency-led topic → real entries from the named year)
|
||||
- [ ] G3 PDF ≥ 6 pages
|
||||
- [ ] G4 `\cite{}` total ≥ 60; per-section minimums hit
|
||||
- [ ] G5 ≥ 6 figures; taxonomy + timeline + coverage matrix all present
|
||||
- [ ] G6 ≥ 1 comparison table
|
||||
- [ ] G7 `\thanks` carries human-review clause; no simulated clause
|
||||
- [ ] G8 every label is referenced
|
||||
- [ ] S1–S5 soft gates reviewed
|
||||
- [ ] Visual sanity check on the rendered PDF
|
||||
- [ ] Honest report — don't pad to clear gates
|
||||
Reference in new issue
Block a user