AOS/skills/literature-survey/references/01-bibliography-expansion.md

Bibliography Expansion for a Survey — 60+ Real Entries (100+ recommended) via WebFetch

Why this exists

A survey paper is largely defined by its bibliography: the breadth of coverage, how recent the most-recent works are, and how foundational the oldest works are determine whether the survey is useful or junk. There is no Python-side fetching: the entire bibliography is built by the orchestrating agent using the WebSearch and WebFetch tools, in one continuous pass before drafting begins.

Hard minimum: 60 unique, real, properly-attributed entries before drafting prose; 100+ recommended. Aim for ~120 as headroom for organic citation needs.

The non-negotiable rule

Every BibTeX entry must originate from a URL the agent fetched in this session. No entries from training-data memory. No "I know this paper exists" shortcuts. If it's not backed by a fetched URL, it does not go in the bib.

For surveys this is especially important: a survey is judged on how completely and accurately it maps a field. A fabricated entry (wrong author, wrong year, hallucinated paper) ends the survey's usefulness instantly.

Tools and when

Tool	Use for	Output
WebSearch	Discovering papers per query angle	List of titles + URLs + snippets
WebFetch	Pulling canonical metadata from a discovered URL	Page content; you parse title/authors/year/venue

Pattern: WebSearch first to find candidates, then WebFetch the most promising URLs.

Reliable URL templates:

• arXiv abstract: https://arxiv.org/abs/<id>
• arXiv API (bulk): http://export.arxiv.org/api/query?search_query=...&max_results=N (Atom XML)
• Semantic Scholar: https://www.semanticscholar.org/paper/<slug>/<id> and the https://api.semanticscholar.org/graph/v1/paper/... API (JSON)
• OpenReview: https://openreview.net/forum?id=<id>
• NeurIPS proceedings: https://papers.nips.cc/paper_files/paper/<year>/...

Process

0. Read the topic's scope & temporal intent FIRST (adaptive search)

Before planning a single query, read the user's topic the way a librarian would and pick a search posture. This is the step that was missing when a topic like "OpenSource LLM 2026" came back with zero 2026 papers — the year in the topic is an instruction, not decoration. There is no fixed rule here; you infer the posture from the signals, then let it shape every later query.

Signal in the topic / request	Posture	What it means for search
Names a specific recent/current/future year ("… 2026"), or says "latest / recent / emerging / state of the art / this year / past 12 months"	Recency-led	The named timeframe is a hard filter. Most queries (≳ 70%) carry that year/range and lead with date-sorted sources (see §3.5). Canon/foundational works appear only as supporting context (≤ ~30%), explicitly framed as "prior to the surveyed window". Do not draft until the bib actually contains real entries from the named window.
Says "history / evolution / development / foundations / comprehensive / from X to Y"	Timeline-spanning	Span earliest seminal work → newest. The "cover the canon" guidance below applies in full.
No temporal signal at all	Balanced (default)	Mix foundational + recent, but always include the most recent 12 months so the survey isn't stale.

Write the chosen posture and the parsed timeframe as the first line of .bib_progress.txt, e.g. POSTURE: recency-led | window: 2025-09 .. now | canon cap: 30%. Every later query-planning and gate decision refers back to it.

Why the old behaviour failed: the model reaches for the famous papers it already knows (all 2–3 years old), and the "span the timeline / a bib that's all-2023+ is a snapshot" guidance below reinforced that pull. Under a recency-led posture that guidance is suspended — there, an all-old bib is the failure, not the goal.

1. Start with an empty bibliography

$RUN/bibliography.bib starts empty after Step 2 of the skill. Build it up to ≥ 60 entries (100+ recommended) below before drafting any prose. Optionally also maintain a $RUN/papers.json index alongside.

2. Plan a query expansion (12–20 angles)

Decompose the topic into angles spanning multiple axes simultaneously. For a survey, breadth matters more than for a research paper.

Recommended angle types (pick 12–20 across these):

• Foundational classics: pre-2015 / pre-2010 papers that defined the field.
• Method families: each named architectural / technical family (one query per family).
• Application domains: each major domain (one query per domain).
• Adjacent paradigms: rival or sibling families (e.g., for a Transformer survey: linear models, MLP-only, state-space).
• Empirical & benchmark studies: dataset and benchmark papers, evaluation studies.
• Theoretical analyses: convergence, expressivity, identifiability, scaling laws.
• Surveys: prior surveys / reviews on the same or adjacent topics.
• Recent advances: queries carrying the explicit year/window from §0 (not a placeholder) to capture the frontier. Under a recency-led posture this is the majority of your angles, not one of twenty.
• Open problems / critiques: papers that argue against received wisdom.
• Tooling / libraries: framework papers (when relevant to the field).

Track the plan in .bib_progress.txt.

3. Per query: WebSearch → triage → WebFetch → extract → append

For each query in the plan:

1. WebSearch with the query string. Result: list of links + titles + snippets.
2. Triage — for each candidate, decide whether to keep:
   • Keep peer-reviewed venue papers and authoritative arXiv preprints.
   • Skip blog posts, marketing pages, vendor pages, Medium articles, lecture slides.
   • Deduplicate against entries already in bibliography.bib.
3. WebFetch the abstract URL to obtain canonical metadata (title, full author list, year, venue).
4. Append the BibTeX entry to bibliography.bib immediately, and append the query to .bib_progress.txt.

For arXiv specifically, the export.arxiv.org Atom endpoint returns 10–30 entries from one fetch — preferred for bulk loading.

3.5. Date-filtered search (use whenever recency matters)

A plain WebSearch ranks by relevance/popularity, which skews toward older, more-cited papers — exactly why a "2026" topic returns 2023–2024 results. When the posture from §0 is recency-led (or balanced, for the recent-12-months slice), don't rely on the model's prior knowledge; fetch a date-sorted feed and read what's actually new:

• arXiv, sorted by date (primary recency tool). The Atom API takes sortBy=submittedDate&sortOrder=descending — the top results are the newest papers, regardless of citation count:

  https://export.arxiv.org/api/query?search_query=all:%22open%20source%20large%20language%20model%22&sortBy=submittedDate&sortOrder=descending&max_results=40
  

  WebFetch this URL (use https — the plain-http host can return empty through some proxies), then keep entries whose <published> year falls inside the §0 window. Repeat per sub-topic (architectures, alignment, evaluation, …) so each theme has fresh coverage. (Verified: a date-sorted query like this returns current-month arXiv papers, exactly the frontier a relevance-ranked search misses.)

• WebSearch with the year embedded and time-scoped phrasing. Put the explicit year in the query and add words that bias toward new work: open source LLM 2026 release, ... announced 2026, ... arxiv 2026. Avoid model-name lists (Llama Mistral Qwen …) under a recency-led posture — they pull the search back to the known canon.

• Cross-check the frontier you can't know. Your training data does not contain the newest papers. Treat any "I already know the key 2026 papers" impulse as a red flag: if it's not from a URL fetched this session via a date-sorted query, it isn't recent enough to trust.

Record in .bib_progress.txt which queries were date-sorted, so the §0 window is auditable.

4. BibTeX entry format

@inproceedings{lastname2023key,
  title={Exact title from the source page},
  author={Lastname, Firstname and Other, Author},
  booktitle={Venue Name},
  year={2023},
  url={https://arxiv.org/abs/2305.12345},
}

@article{lastname2024arxiv,
  title={Exact title from the source page},
  author={Lastname, Firstname},
  journal={arXiv preprint arXiv:2401.12345},
  year={2024},
  url={https://arxiv.org/abs/2401.12345},
}

Always include url= so a reviewer can audit. Use @inproceedings with booktitle={...} for conferences, @article with journal={...} for journals. arXiv preprints are @article with journal={arXiv preprint arXiv:<id>}.

bib_key convention: <firstauthorlastname><year><firstmeaningfulword>, lowercase, alphanumeric. Disambiguate collisions with a single suffix letter.

Acronyms in titles: wrap with {} to preserve case — {T}ransformers, {FED}former, {N-BEATS}. Escape &, %, $, _, #.

5. Sync to papers.json (recommended)

Append a JSON record per BibTeX entry to papers.json so the manifest stays consistent. Minimum fields: title, authors, year, bib_key, url, dimension (which query angle discovered it).

The dimension field powers the literature table later: papers grouped by sub-area become a classified inventory the survey reader can use as a starting point.

6. Validate before drafting

cd output/literature-survey/<slug>/survey_paper
grep -c "^@" bibliography.bib    # should be ≥ 60 (100+ recommended)

Run a pdflatex + bibtex + pdflatex × 2 pass on the template main.tex with the new bib. If main.bbl doesn't list everything you expect, the bib has a syntax problem; fix before drafting prose.

7. As you draft, audit cite resolution

After every section, recompile and grep main.log for Citation .* undefined. Each undefined citation is either a typo or a missing bib entry. Fix immediately.

Survey-specific bib quality

A survey bibliography should also satisfy (read these through the §0 posture — the timeline rule below is for timeline-spanning / balanced surveys; a recency-led survey deliberately concentrates in its window):

• Span the timeline (timeline-spanning / balanced postures). Papers should range from the earliest seminal work to the last 6 months. A bib that's all 2023+ is a snapshot, not a survey. Under a recency-led posture this is inverted: the bib should concentrate in the named window, with older works cited only as the lineage that leads up to it.
• Cover the canon. Every named technique in your taxonomy should be cited via its primary source — the paper that introduced it.
• Include critiques. If the field has a debate (e.g., "are X effective?"), cite both sides; a one-sided survey is a position paper.
• Cite prior surveys. Every prior survey on the same / adjacent topic must be cited, and its scope contrasted with yours in the introduction.

Anti-patterns

• Memory-sourced entries. Forbidden. WebFetch first or omit.
• Fabricated entries. Catastrophic. Real or weaker claim, never fake reference.
• One paper, 30 cites. Each \cite{} should advance the argument. Density comes from prose density, not paper repetition.
• Citation dump. \cite{a, b, c, d, e, f, g, h} covering 8 papers at once means you didn't read them. Group thematically and cite 2–4 per claim with prose distinguishing them.
• Stopping at the 60 floor. 60 is the minimum, not the target — aim for 100+.
• Skipping url=. Without the URL the entry is unverifiable.

Quick checklist

• Picked a search posture from §0 (recency-led / timeline-spanning / balanced) and wrote it + the parsed window to .bib_progress.txt
• If recency matters: ran date-sorted arXiv queries (§3.5), not just relevance-ranked WebSearch
• Planned query angles spanning timeline / family / domain / paradigm / theory / surveys / recent / critiques — weighted by the posture
• Each entry backed by a URL fetched in this session (not from memory)
• grep -c "^@" bibliography.bib ≥ 60 (100+ recommended)
• Every entry has title, author, year, venue, url
• papers.json records dimension per entry for the classified table
• .bib_progress.txt records executed queries
• First test compile shows zero "Citation undefined" warnings against the template