Files
Z User e45b68bb8b feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)
Real IMAP connection via imapflow library:
- Connects to IMAP server (IMAPFlow, TLS on port 993)
- Fetches recent messages (configurable limit, default 20)
- Extracts: from, fromEmail, subject, date, snippet (200 chars)
- Thread-to-contact matching (PRD §6.1):
  1. Match sender email against persons table
  2. If no person match, extract domain and match against company.website
  3. Returns matchedCompany + matchedPerson for each email
- Audit-logged (action: crm.mailsync:imap)
- Zod validated (host, port, user, password, folder, limit)
- 502 on IMAP connection failure with detail
- 400 on validation failure

Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
2026-10-27 22:50:50 +00:00

321 lines
20 KiB
Markdown
Executable File
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Survey Figures Playbook
## Why this exists
A research paper's figures tend to be experimental: tables of numbers, training curves, confusion matrices. A survey's figures are *organisational*: how is the field structured, when did things happen, who works on what. They are how a reader navigates the survey. Without good figures a survey is a long list of references; with them it becomes a map.
**Hard targets:** **6–10 figures** spanning these families:
- **At least 1 taxonomy / classification diagram** (TikZ tree or hierarchy)
- **At least 1 chronological timeline** of major works
- **At least 1 area / capability matrix** (coverage heatmap)
- **1–2 representative architecture / mechanism diagrams** (TikZ)
- **1–2 quantitative trend plots** (matplotlib publication style)
- Optional: paradigm comparison, citation-co-occurrence map, geographic / institutional map
## Design figures for THIS research — aim at the Nature/Science bar
Two real failures this playbook used to cause. Read both before you draw anything.
**1. Template leakage (the worst over-fitting).** The worked examples below were written about *time-series Transformers*. A real "open-source LLM" survey produced from this playbook shipped a taxonomy whose leaves were `Informer`, `Autoformer`, `PatchTST`, `iTransformer`, `Moirai` — time-series forecasting models, completely off-topic — because the agent copied the *sample's content*, not just its shape. **Every code block here is a worked example of a principle, never content to reuse.** Replace 100% of the labels, branches, palette, and composition with what *your* topic and *your* gathered papers actually require. If a reader could guess which sample you started from, you over-templated. The figure should look like it was designed for this survey and no other.
**2. Generic look.** Default LaTeX/matplotlib output (Computer Modern *serif*, boxed axes, rainbow colours, one chart per figure) reads as a class assignment, not a journal. The target is **Nature / Science**: figures that are *designed*, not decorated.
### What "Nature/Science" concretely means (not vibes)
- **A figure is a composition, not a lone chart.** Top journals compose multi-panel figures (**a**, **b**, **c** …), each panel one sub-claim, together telling one story. Use `subfigure` / matplotlib `subplots`, with **bold lowercase panel labels in the top-left corner** of each panel.
- **Sans-serif type.** Helvetica/Arial (both installed here), ~7 pt floor — never Computer Modern serif. matplotlib: `font.family:"sans-serif"`. TikZ/LaTeX diagrams: `\usepackage[scaled]{helvet}\renewcommand{\familydefault}{\sfdefault}` (or set the figure's nodes to `\sffamily`).
- **Restrained, colour-blind-safe colour that *encodes something*.** Use the Wong palette (the one *Nature* recommends): `#000000 #E69F00 #56B4E9 #009E73 #F0E442 #0072B2 #D55E00 #CC79A7`. For ordinal data use a single-hue sequential map (`Blues`), never rainbow/`jet`. ≤ ~5 hues per figure; colour must mean something, not decorate.
- **Maximise data-ink.** Despine top + right; ticks short and outward; axis lines ~0.6 pt; no gridlines (or very faint); **direct-label lines** instead of a legend box where possible. Banned: 3-D, pie charts, drop shadows, gradients, raster screenshots.
- **Exact sizing.** *Nature* column width = **89 mm** single (≈ 3.5 in), **183 mm** double (≈ 7.2 in). Set matplotlib `figsize` to these in inches; in this single-column A4 template include at `width=\linewidth`.
- **The caption states a finding.** Bold "Figure N." + one declarative sentence giving the takeaway, then detail. Not "Taxonomy of methods." but "Open-source LLM research splits into four lineages that diverge after the 2024 MoE shift."
### Nature-style matplotlib preamble — use this everywhere instead of the old serif block
```python
import matplotlib as mpl
WONG = ["#000000","#E69F00","#56B4E9","#009E73","#F0E442","#0072B2","#D55E00","#CC79A7"]
mpl.rcParams.update({
"font.family":"sans-serif", "font.sans-serif":["Helvetica","Arial","DejaVu Sans"],
"font.size":7, "axes.labelsize":7, "axes.titlesize":7,
"xtick.labelsize":6.5, "ytick.labelsize":6.5, "legend.fontsize":6.5,
"axes.linewidth":0.6, "axes.spines.top":False, "axes.spines.right":False,
"xtick.direction":"out", "ytick.direction":"out",
"xtick.major.size":2.5, "ytick.major.size":2.5,
"xtick.major.width":0.6, "ytick.major.width":0.6,
"axes.grid":False, "lines.linewidth":1.2, "lines.markersize":3.5,
"savefig.dpi":400, "savefig.bbox":"tight", "pdf.fonttype":42, "ps.fonttype":42,
"axes.prop_cycle": mpl.cycler(color=WONG[1:]),
})
# figsize: single column = (89/25.4, h); double column / 2-panel = (183/25.4, h)
```
### The design process — this is where flexibility lives
Templates are rigid because they answer "what figure exists?" Design answers "what does *this* survey need to show?" For each figure, decide from scratch:
1. **What claim in this section must the reader *see* to believe it?** No claim → no figure (cut it; decoration fails the Nature bar).
2. **What is the shape of that evidence?** hierarchy → tree (`forest`); change over time → timeline / line; coverage or comparison → matrix / small-multiples; mechanism → schematic; distribution → strip/box. Let the data shape pick the form — don't force every survey into taxonomy+timeline+matrix.
3. **What is the most reductive form that still makes the point?** One well-composed panel beats a busy one; merge weak figures, split overloaded ones.
4. **Compose** with the Nature language above, then apply the mechanical floor below (vector, `forest`/`\resizebox`, zero overflow/overlap).
Steps 1–3 change with every topic — that is the flexible craft. The mechanical floor (next section) never changes.
## Fitting & overflow — the mechanical floor (read BEFORE drawing any figure)
Most "ugly survey PDF" problems are not bad taste, they are **figures that overrun the column or overlap themselves** because the source used hand-tuned absolute sizes (`sibling distance=42mm`, `figsize=(7,…)`) that happen not to fit. The fix is to never hand-tune for fit — let the figure *adapt* to the available width.
**Know your column width first.** This template is **single-column** `article` on A4 with 1-inch margins → text width ≈ **159 mm (≈ 6.2 in)**. (`\documentclass{article}` is single-column; `figure*` does **not** widen anything here — it only spans columns in a genuinely two-column class such as IEEEtran. Don't reach for `figure*` to "get more room"; you don't have a second column.)
**There are two different failure modes, and they need two different fixes. Confusing them is why the old taxonomy sample both ran off the page _and_ overlapped.**
### Failure A — figure wider than the column (runs off the right margin)
Fix: wrap the picture so its *rendered width* is clamped.
- **TikZ:** wrap in `\resizebox{\linewidth}{!}{ … }`.
- **Includes:** always `\includegraphics[width=\linewidth]{…}` (or `0.85–0.95\linewidth`), never a bare `\includegraphics{file}`. matplotlib `figsize` then only sets aspect ratio; keep its width ≤ 6.2 in so in-figure fonts aren't double-shrunk.
```latex
\begin{figure}[!t]
\centering
\resizebox{\linewidth}{!}{%
\begin{tikzpicture}[ ... ] ... \end{tikzpicture}%
}
\caption{...}\label{fig:...}
\end{figure}
```
### Failure B — nodes overlapping each other (the real "ugly" problem)
**`\resizebox` does NOT fix overlap.** It scales the *whole* picture uniformly, so two boxes that overlap at natural size still overlap after shrinking — you just get a smaller tangle. Overlap comes from hand-picked spacing (`sibling distance=42mm`) where adjacent sub-trees are wider than the gap between them. The fixes, in order of preference:
1. **Use a layout engine that auto-spaces — `\usepackage{forest}`.** Forest computes sibling positions so children never collide, for any branch/leaf count. This is the default for taxonomies now (see Family 1). It removes the whole class of "magic mm that happens to overlap".
2. **Pick orientation by size.** Few leaves (≲ 12) → top-down reads well. Many leaves → grow the tree **horizontally** (`grow'=east`) so leaves stack down the page (vertical space is plentiful, horizontal isn't).
3. **Fold.** If one figure still can't hold it legibly, split a branch into its own sub-figure.
Then apply Failure-A's `\resizebox` on top as a width backstop. Order matters: forest (or real spacing) removes overlap; resizebox only then trims width.
The quality gate (`05-quality-gate.md` G1 + S5) treats residual `Overfull \hbox` as a remediation loop, and reminds you that a figure shrunk to illegibility is *not* a pass — fold it instead.
## File layout
All figures land under `output/literature-survey/<slug>/survey_paper/figures/`. Use vector PDF for LaTeX (`.pdf`); keep a PNG sibling only for previewing. TikZ figures live inline inside the relevant `sections/*.tex` (no separate file needed). matplotlib / seaborn figures get a `make_fig_NN_<slug>.py` script alongside the PDF.
Naming: `fig_<NN>_<short-slug>.pdf`. Numbers correspond to citation order.
## Family 1 — Taxonomy / classification diagrams (TikZ)
The single most important survey figure. A taxonomy figure says: *here is how I cut the field*. Readers will refer back to it constantly.
### Setup
In `survey_paper/main.tex`:
```latex
\usepackage{tikz}
\usetikzlibrary{positioning, shapes.geometric, arrows.meta, fit, backgrounds, calc, trees}
\usepackage{forest} % auto-spaced trees — taxonomies use this, NOT hand-built child{} trees
```
### Tree-style taxonomy — use `forest`, not hand-tuned `child{}` distances
> **Do not** build the taxonomy from a `tikzpicture` with `level/.style={sibling distance=42mm}` and nested `child{}` (the old sample here did exactly that). With more than ~3 leaves per branch, adjacent sub-trees are wider than the fixed gap and **the leaves overlap** — and because `\resizebox` scales the overlap too, you can't shrink your way out of it. `forest` computes sibling spacing automatically, so children never collide regardless of count.
**Flat, sans-serif, journal-style top-down (default; good up to ~12 leaves).** Define two colours once in `main.tex` (`\definecolor{nblue}{HTML}{0072B2}` and `\definecolor{nfill}{HTML}{EAF2FB}`, `\definecolor{ngrey}{HTML}{4D4D4D}`); thin uniform grey strokes + one accent + sans labels is the Nature diagram look — not the heavy blue/orange boxes the old sample used. **Replace every label with your own taxonomy.**
```latex
\begin{figure}[!t]
\centering
\resizebox{\linewidth}{!}{%
\begin{forest}
for tree={font=\sffamily\small, draw=ngrey, line width=0.4pt, rounded corners=1pt,
inner sep=3pt, align=center, edge={draw=ngrey, line width=0.4pt},
l sep=7mm, s sep=3mm, parent anchor=south, child anchor=north}
[Open-source LLMs, draw=nblue, line width=0.7pt, text=nblue
[Architecture, fill=nfill, draw=nblue [Llama 3][Mistral][Qwen2]]
[Sparse / MoE, fill=nfill, draw=nblue [DeepSeek][Mixtral][OLMoE]]
[Alignment, fill=nfill, draw=nblue [DPO][ORPO][KTO]]
[Efficiency, fill=nfill, draw=nblue [GPTQ][AWQ][QLoRA]]
]
\end{forest}%
}
\caption{\textbf{Figure 1.} Open-source LLM research organises into four lineages. Each leaf cites its primary source; full coverage in Section~\ref{sec:methods}.}
\label{fig:taxonomy}
\end{figure}
```
(`\sffamily` gives Helvetica when `main.tex` loads `\usepackage[scaled]{helvet}` — which the template now does — while body text stays serif, the standard journal split.)
**Horizontal / folder (use when there are many leaves — leaves stack down the page where there is room):** add `grow'=east, parent anchor=east, child anchor=west, anchor=west` to the `for tree={…}` options and switch `align=center`→`align=left`. This keeps labels full-size instead of shrinking them to fit one row. (Verified: a 5-branch / 20-leaf tree renders with zero overlap and zero overfull in both orientations; the same content as a hand-built `child{}` tree overflowed the page by 270 pt and merged half its leaves.)
### Layered / hierarchy diagram (when tree doesn't fit)
When the field has a layered structure (e.g., infrastructure → models → applications), use horizontal layers with `\node[fit=...]` to draw boundary boxes.
### Taste
- Color: one family per branch; orange for the row that contains your contribution if you have one.
- Leaves: 8–15 reads best top-down. With `forest` there is no hard cap (it won't overlap), but past ~15 switch to the horizontal orientation so labels stay full-size rather than shrinking; past ~24 fold into sub-figures.
- Always cite each leaf in the prose; the leaf names alone are not the survey, the citations are.
## Family 2 — Timeline of major works (matplotlib)
A survey timeline shows how the field evolved. Place key papers on a horizontal time axis with publication-year ticks.
```python
"""Generate fig_02_timeline.pdf — chronology of major works."""
import os
import matplotlib.pyplot as plt
# Apply the Nature-style preamble from the top of this reference (sans-serif,
# Wong palette, despined, thin axes). Do NOT use font.family:"serif".
events = [
# (year, lane, label, color)
(2017, 0, "Transformer\n(Vaswani)", "#1f77b4"),
(2019, 0, "LogSparse", "#1f77b4"),
(2020, 0, "Reformer", "#1f77b4"),
(2021, 1, "Informer\nAutoformer", "#ff7f0e"),
(2022, 1, "FEDformer\nPyraformer", "#ff7f0e"),
(2022, 2, "RevIN", "#2ca02c"),
(2023, 1, "PatchTST\nDLinear", "#ff7f0e"),
(2023, 3, "TimesNet\nTSMixer", "#9467bd"),
(2024, 1, "iTransformer\nTimeMixer", "#ff7f0e"),
(2024, 4, "MOMENT, Moirai\nTimeGPT, Chronos", "#d62728"),
]
LANE_LABELS = {
0: "Foundations",
1: "Time-domain Trans.",
2: "Normalisation",
3: "MLP / 2D",
4: "Foundation models",
}
fig, ax = plt.subplots(figsize=(6.0, 3.4)) # width ≤ column (6.2in); include with width=\linewidth
years = [e[0] for e in events]
ax.set_xlim(min(years) - 0.5, max(years) + 0.5)
ax.set_ylim(-0.7, max(LANE_LABELS) + 0.7)
for year, lane, label, color in events:
ax.scatter([year], [lane], s=120, color=color, edgecolor="black", linewidth=0.5, zorder=3)
ax.annotate(label, (year, lane), xytext=(0, 12), textcoords="offset points",
ha="center", fontsize=7.5, color="black")
ax.set_yticks(list(LANE_LABELS.keys()))
ax.set_yticklabels([LANE_LABELS[k] for k in sorted(LANE_LABELS)])
ax.set_xlabel("Year")
ax.grid(True, axis="x", alpha=0.25)
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
plt.tight_layout(pad=0.4)
out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "fig_02_timeline.pdf")
plt.savefig(out)
plt.close(fig)
print(f"Wrote {out}")
```
### Taste
- Lanes carve the timeline into sub-areas (rows of the figure) — the same sub-areas as the taxonomy (Fig~\ref{fig:taxonomy}). Consistency helps readers connect the two.
- Annotate ~10–15 key works; more is clutter. Pick the works that defined the year, not every paper.
- Color matches taxonomy color so the reader can cross-reference.
## Family 3 — Coverage matrix / capability heatmap (seaborn)
A survey often makes claims like "method X covers capabilities A, B but not C". A binary or three-level matrix makes that scannable.
```python
import os
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
methods = ["DLinear", "Informer", "Autoformer", "FEDformer",
"PatchTST", "iTransformer", "TimeMixer",
"MOMENT", "Moirai", "Chronos"]
caps = ["Long horizon", "Multivariate", "Probabilistic",
"Zero-shot", "Efficient", "Interpretable"]
# 0 = no, 1 = partial, 2 = yes (illustrative — derive from your survey reading)
cov = np.array([
[2, 1, 0, 0, 2, 1], # DLinear
[2, 1, 1, 0, 1, 0], # Informer
[2, 2, 1, 0, 1, 1], # Autoformer
[2, 2, 1, 0, 1, 1], # FEDformer
[2, 2, 0, 0, 2, 1], # PatchTST
[2, 2, 0, 0, 2, 0], # iTransformer
[2, 2, 0, 0, 2, 0], # TimeMixer
[2, 2, 1, 2, 1, 0], # MOMENT
[2, 2, 2, 2, 1, 0], # Moirai
[2, 2, 2, 2, 1, 0], # Chronos
])
# Apply the Nature-style preamble from the top of this reference (sans-serif, despined).
fig, ax = plt.subplots(figsize=(5.6, 3.6))
sns.heatmap(cov, annot=False, cmap="Blues", vmin=0, vmax=2,
xticklabels=caps, yticklabels=methods,
linewidths=0.4, linecolor="white",
cbar_kws={"label": "Coverage", "ticks": [0, 1, 2], "shrink": 0.6},
ax=ax)
ax.set_xticklabels(ax.get_xticklabels(), rotation=30, ha="right")
ax.set_yticklabels(ax.get_yticklabels(), rotation=0)
plt.tight_layout(pad=0.4)
plt.savefig("fig_03_capability_matrix.pdf")
plt.close(fig)
```
### Taste
- 0/1/2 (no/partial/yes) is more readable than continuous numbers when the underlying judgement is qualitative.
- Annotate cells only if there are ≤ 60.
- Cmap: greens for "more is better" (coverage); avoid traffic-light reds in surveys.
## Family 4 — Architecture / mechanism diagrams (TikZ)
When a sub-area shares a common architectural pattern (e.g., "all decomposition Transformers do X then Y"), a single TikZ figure illustrating the canonical pipeline saves several pages of prose. See the `paper-writer` skill's `references/02-figures-publication-grade.md` § Family 1 for the full template.
For a survey, prefer **abstracted** architectural diagrams that show the pattern shared by a family of methods, not one specific method.
## Family 5 — Quantitative trends (matplotlib)
Examples that work in a survey:
- Number of papers per year (publication trend)
- Reported MSE on a single benchmark over time (the SOTA curve)
- Compute requirements over time (size/cost trend)
These earn their place when they reveal something the prose can't. A "papers per year" plot that just goes up linearly is decoration; if it reveals a plateau, an inflection, or a divergence between sub-areas, it's content.
## Family 6 — Optional advanced
- **Citation co-occurrence network**: dot/network plot showing which papers cite each other (use NetworkX → matplotlib). Powerful when the field has clear clusters.
- **Geographic / institutional map**: world map with author affiliations marked. Niche but illuminating for cross-cultural fields.
- **Paradigm comparison**: side-by-side architectural sketches (multi-panel TikZ).
## No default figures
This skill has no default figure (unlike experiment-suite). All figures are agent-generated.
## Quick checklist
- [ ] **Designed for THIS topic** — zero labels/leaves/data carried over from the worked examples (no `Informer`/`PatchTST`/time-series content in a non-time-series survey)
- [ ] Each figure earns its place by making a specific claim visible (design process step 1)
- [ ] Nature/Science look: sans-serif labels, Wong palette / single-hue sequential, despined axes, direct labels, multi-panel with **a**/**b** where it tells one story
- [ ] matplotlib uses the sans-serif Nature preamble (not `font.family:"serif"`); `figsize` at 89 mm / 183 mm
- [ ] Caption leads with a one-sentence finding, not a noun phrase
- [ ] Taxonomy built with `forest` (auto-spaced), NOT hand-tuned `sibling distance` + `child{}` — no overlapping leaves
- [ ] Every TikZ/forest figure wrapped in `\resizebox{\linewidth}{!}{…}` as a width backstop (after spacing is correct, not instead of it)
- [ ] Every `\includegraphics` has `width=…\linewidth`; matplotlib `figsize` width ≤ 6.2 in
- [ ] No `figure*` used to "gain width" (single-column template); many-leaf trees go horizontal / fold, not squashed
- [ ] At least 1 taxonomy diagram; cited from §Methods or §Background
- [ ] At least 1 timeline; cited from §Introduction
- [ ] At least 1 capability / coverage matrix; cited from §Discussion
- [ ] 6–10 figures total
- [ ] All figure files are vector PDF; fonts embedded (`pdf.fonttype=42`)
- [ ] Each figure referenced in prose with `\ref{fig:...}`
- [ ] Each caption stands alone (a reader who only sees the figure understands it)
- [ ] No 3-D bars, no pie charts, no rainbow palettes, no raster screenshots