feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed
1 parent
7acc0c5451
commit
e45b68bb8b
1089 files changed
+260614
No files matched your search
@@ -0,0 +1,173 @@
|
||||
# Bibliography Expansion for a Survey — 60+ Real Entries (100+ recommended) via WebFetch
|
||||
|
||||
## Why this exists
|
||||
|
||||
A survey paper is largely *defined* by its bibliography: the breadth of coverage, how recent the most-recent works are, and how foundational the oldest works are determine whether the survey is useful or junk. There is no Python-side fetching: the entire bibliography is built by the orchestrating agent using the **WebSearch** and **WebFetch** tools, in one continuous pass before drafting begins.
|
||||
|
||||
**Hard minimum:** **60 unique, real, properly-attributed entries** before drafting prose; **100+ recommended**. Aim for ~120 as headroom for organic citation needs.
|
||||
|
||||
## The non-negotiable rule
|
||||
|
||||
> **Every BibTeX entry must originate from a URL the agent fetched in this session.** No entries from training-data memory. No "I know this paper exists" shortcuts. If it's not backed by a fetched URL, it does not go in the bib.
|
||||
|
||||
For surveys this is especially important: a survey is judged on how completely and accurately it maps a field. A fabricated entry (wrong author, wrong year, hallucinated paper) ends the survey's usefulness instantly.
|
||||
|
||||
## Tools and when
|
||||
|
||||
| Tool | Use for | Output |
|
||||
|---|---|---|
|
||||
| **WebSearch** | Discovering papers per query angle | List of titles + URLs + snippets |
|
||||
| **WebFetch** | Pulling canonical metadata from a discovered URL | Page content; you parse title/authors/year/venue |
|
||||
|
||||
Pattern: WebSearch first to find candidates, then WebFetch the most promising URLs.
|
||||
|
||||
Reliable URL templates:
|
||||
- arXiv abstract: `https://arxiv.org/abs/<id>`
|
||||
- arXiv API (bulk): `http://export.arxiv.org/api/query?search_query=...&max_results=N` (Atom XML)
|
||||
- Semantic Scholar: `https://www.semanticscholar.org/paper/<slug>/<id>` and the `https://api.semanticscholar.org/graph/v1/paper/...` API (JSON)
|
||||
- OpenReview: `https://openreview.net/forum?id=<id>`
|
||||
- NeurIPS proceedings: `https://papers.nips.cc/paper_files/paper/<year>/...`
|
||||
|
||||
## Process
|
||||
|
||||
### 0. Read the topic's scope & temporal intent FIRST (adaptive search)
|
||||
|
||||
Before planning a single query, read the user's topic the way a librarian would and pick a **search posture**. This is the step that was missing when a topic like "OpenSource LLM **2026**" came back with zero 2026 papers — the year in the topic *is* an instruction, not decoration. There is no fixed rule here; you infer the posture from the signals, then let it shape every later query.
|
||||
|
||||
| Signal in the topic / request | Posture | What it means for search |
|
||||
|---|---|---|
|
||||
| Names a specific recent/current/future year ("… 2026"), or says "latest / recent / emerging / state of the art / this year / past 12 months" | **Recency-led** | The named timeframe is a **hard filter**. Most queries (≳ 70%) carry that year/range and lead with date-sorted sources (see §3.5). Canon/foundational works appear only as *supporting context* (≤ ~30%), explicitly framed as "prior to the surveyed window". **Do not draft until the bib actually contains real entries from the named window.** |
|
||||
| Says "history / evolution / development / foundations / comprehensive / from X to Y" | **Timeline-spanning** | Span earliest seminal work → newest. The "cover the canon" guidance below applies in full. |
|
||||
| No temporal signal at all | **Balanced (default)** | Mix foundational + recent, but always include the **most recent 12 months** so the survey isn't stale. |
|
||||
|
||||
Write the chosen posture and the parsed timeframe as the first line of `.bib_progress.txt`, e.g. `POSTURE: recency-led | window: 2025-09 .. now | canon cap: 30%`. Every later query-planning and gate decision refers back to it.
|
||||
|
||||
> **Why the old behaviour failed:** the model reaches for the famous papers it already knows (all 2–3 years old), and the "span the timeline / a bib that's all-2023+ is a snapshot" guidance below *reinforced* that pull. Under a recency-led posture that guidance is **suspended** — there, an all-old bib is the failure, not the goal.
|
||||
|
||||
### 1. Start with an empty bibliography
|
||||
|
||||
`$RUN/bibliography.bib` starts empty after Step 2 of the skill. Build it up to ≥ 60 entries (100+ recommended) below before drafting any prose. Optionally also maintain a `$RUN/papers.json` index alongside.
|
||||
|
||||
### 2. Plan a query expansion (12–20 angles)
|
||||
|
||||
Decompose the topic into angles spanning multiple axes simultaneously. For a survey, breadth matters more than for a research paper.
|
||||
|
||||
Recommended angle types (pick 12–20 across these):
|
||||
|
||||
- **Foundational classics**: pre-2015 / pre-2010 papers that defined the field.
|
||||
- **Method families**: each named architectural / technical family (one query per family).
|
||||
- **Application domains**: each major domain (one query per domain).
|
||||
- **Adjacent paradigms**: rival or sibling families (e.g., for a Transformer survey: linear models, MLP-only, state-space).
|
||||
- **Empirical & benchmark studies**: dataset and benchmark papers, evaluation studies.
|
||||
- **Theoretical analyses**: convergence, expressivity, identifiability, scaling laws.
|
||||
- **Surveys**: prior surveys / reviews on the same or adjacent topics.
|
||||
- **Recent advances**: queries carrying the **explicit year/window from §0** (not a placeholder) to capture the frontier. Under a *recency-led* posture this is the majority of your angles, not one of twenty.
|
||||
- **Open problems / critiques**: papers that argue against received wisdom.
|
||||
- **Tooling / libraries**: framework papers (when relevant to the field).
|
||||
|
||||
Track the plan in `.bib_progress.txt`.
|
||||
|
||||
### 3. Per query: WebSearch → triage → WebFetch → extract → append
|
||||
|
||||
For each query in the plan:
|
||||
|
||||
1. **WebSearch** with the query string. Result: list of links + titles + snippets.
|
||||
2. **Triage** — for each candidate, decide whether to keep:
|
||||
- Keep peer-reviewed venue papers and authoritative arXiv preprints.
|
||||
- Skip blog posts, marketing pages, vendor pages, Medium articles, lecture slides.
|
||||
- Deduplicate against entries already in `bibliography.bib`.
|
||||
3. **WebFetch** the abstract URL to obtain canonical metadata (title, full author list, year, venue).
|
||||
4. **Append** the BibTeX entry to `bibliography.bib` immediately, and append the query to `.bib_progress.txt`.
|
||||
|
||||
For arXiv specifically, the export.arxiv.org Atom endpoint returns 10–30 entries from one fetch — preferred for bulk loading.
|
||||
|
||||
### 3.5. Date-filtered search (use whenever recency matters)
|
||||
|
||||
A plain WebSearch ranks by relevance/popularity, which skews toward *older, more-cited* papers — exactly why a "2026" topic returns 2023–2024 results. When the posture from §0 is **recency-led** (or balanced, for the recent-12-months slice), don't rely on the model's prior knowledge; **fetch a date-sorted feed and read what's actually new**:
|
||||
|
||||
- **arXiv, sorted by date (primary recency tool).** The Atom API takes `sortBy=submittedDate&sortOrder=descending` — the top results are the newest papers, regardless of citation count:
|
||||
|
||||
```
|
||||
https://export.arxiv.org/api/query?search_query=all:%22open%20source%20large%20language%20model%22&sortBy=submittedDate&sortOrder=descending&max_results=40
|
||||
```
|
||||
|
||||
WebFetch this URL (use **https** — the plain-`http` host can return empty through some proxies), then keep entries whose `<published>` year falls inside the §0 window. Repeat per sub-topic (architectures, alignment, evaluation, …) so each theme has fresh coverage. (Verified: a date-sorted query like this returns current-month arXiv papers, exactly the frontier a relevance-ranked search misses.)
|
||||
- **WebSearch with the year embedded and time-scoped phrasing.** Put the explicit year in the query and add words that bias toward new work: `open source LLM 2026 release`, `... announced 2026`, `... arxiv 2026`. Avoid model-name lists (`Llama Mistral Qwen …`) under a recency-led posture — they pull the search back to the known canon.
|
||||
- **Cross-check the frontier you can't know.** Your training data does not contain the newest papers. Treat any "I already know the key 2026 papers" impulse as a red flag: if it's not from a URL fetched this session via a date-sorted query, it isn't recent enough to trust.
|
||||
|
||||
Record in `.bib_progress.txt` which queries were date-sorted, so the §0 window is auditable.
|
||||
|
||||
### 4. BibTeX entry format
|
||||
|
||||
```bibtex
|
||||
@inproceedings{lastname2023key,
|
||||
title={Exact title from the source page},
|
||||
author={Lastname, Firstname and Other, Author},
|
||||
booktitle={Venue Name},
|
||||
year={2023},
|
||||
url={https://arxiv.org/abs/2305.12345},
|
||||
}
|
||||
|
||||
@article{lastname2024arxiv,
|
||||
title={Exact title from the source page},
|
||||
author={Lastname, Firstname},
|
||||
journal={arXiv preprint arXiv:2401.12345},
|
||||
year={2024},
|
||||
url={https://arxiv.org/abs/2401.12345},
|
||||
}
|
||||
```
|
||||
|
||||
Always include `url=` so a reviewer can audit. Use `@inproceedings` with `booktitle={...}` for conferences, `@article` with `journal={...}` for journals. arXiv preprints are `@article` with `journal={arXiv preprint arXiv:<id>}`.
|
||||
|
||||
**bib_key convention:** `<firstauthorlastname><year><firstmeaningfulword>`, lowercase, alphanumeric. Disambiguate collisions with a single suffix letter.
|
||||
|
||||
**Acronyms in titles**: wrap with `{}` to preserve case — `{T}ransformers`, `{FED}former`, `{N-BEATS}`. Escape `&`, `%`, `$`, `_`, `#`.
|
||||
|
||||
### 5. Sync to `papers.json` (recommended)
|
||||
|
||||
Append a JSON record per BibTeX entry to `papers.json` so the manifest stays consistent. Minimum fields: `title`, `authors`, `year`, `bib_key`, `url`, `dimension` (which query angle discovered it).
|
||||
|
||||
The `dimension` field powers the literature table later: papers grouped by sub-area become a classified inventory the survey reader can use as a starting point.
|
||||
|
||||
### 6. Validate before drafting
|
||||
|
||||
```bash
|
||||
cd output/literature-survey/<slug>/survey_paper
|
||||
grep -c "^@" bibliography.bib # should be ≥ 60 (100+ recommended)
|
||||
```
|
||||
|
||||
Run a `pdflatex + bibtex + pdflatex × 2` pass on the template `main.tex` with the new bib. If `main.bbl` doesn't list everything you expect, the bib has a syntax problem; fix before drafting prose.
|
||||
|
||||
### 7. As you draft, audit cite resolution
|
||||
|
||||
After every section, recompile and grep `main.log` for `Citation .* undefined`. Each undefined citation is either a typo or a missing bib entry. Fix immediately.
|
||||
|
||||
## Survey-specific bib quality
|
||||
|
||||
A survey bibliography should also satisfy (read these **through the §0 posture** — the timeline rule below is for *timeline-spanning / balanced* surveys; a *recency-led* survey deliberately concentrates in its window):
|
||||
|
||||
- **Span the timeline** *(timeline-spanning / balanced postures)*. Papers should range from the earliest seminal work to the last 6 months. A bib that's all 2023+ is a snapshot, not a survey. **Under a recency-led posture this is inverted:** the bib should concentrate in the named window, with older works cited only as the lineage that leads up to it.
|
||||
- **Cover the canon.** Every named technique in your taxonomy should be cited via its primary source — the paper that introduced it.
|
||||
- **Include critiques.** If the field has a debate (e.g., "are X effective?"), cite both sides; a one-sided survey is a position paper.
|
||||
- **Cite prior surveys.** Every prior survey on the same / adjacent topic must be cited, and its scope contrasted with yours in the introduction.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Memory-sourced entries.** Forbidden. WebFetch first or omit.
|
||||
- **Fabricated entries.** Catastrophic. Real or weaker claim, never fake reference.
|
||||
- **One paper, 30 cites.** Each `\cite{}` should advance the argument. Density comes from prose density, not paper repetition.
|
||||
- **Citation dump.** `\cite{a, b, c, d, e, f, g, h}` covering 8 papers at once means you didn't read them. Group thematically and cite 2–4 per claim with prose distinguishing them.
|
||||
- **Stopping at the 60 floor.** 60 is the minimum, not the target — aim for 100+.
|
||||
- **Skipping `url=`.** Without the URL the entry is unverifiable.
|
||||
|
||||
## Quick checklist
|
||||
|
||||
- [ ] Picked a search posture from §0 (recency-led / timeline-spanning / balanced) and wrote it + the parsed window to `.bib_progress.txt`
|
||||
- [ ] If recency matters: ran date-sorted arXiv queries (§3.5), not just relevance-ranked WebSearch
|
||||
- [ ] Planned query angles spanning timeline / family / domain / paradigm / theory / surveys / recent / critiques — weighted by the posture
|
||||
- [ ] Each entry backed by a URL fetched in this session (not from memory)
|
||||
- [ ] `grep -c "^@" bibliography.bib` ≥ 60 (100+ recommended)
|
||||
- [ ] Every entry has `title`, `author`, `year`, venue, `url`
|
||||
- [ ] `papers.json` records `dimension` per entry for the classified table
|
||||
- [ ] `.bib_progress.txt` records executed queries
|
||||
- [ ] First test compile shows zero "Citation undefined" warnings against the template
|
||||
Reference in new issue
Block a user