feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)
Real IMAP connection via imapflow library: - Connects to IMAP server (IMAPFlow, TLS on port 993) - Fetches recent messages (configurable limit, default 20) - Extracts: from, fromEmail, subject, date, snippet (200 chars) - Thread-to-contact matching (PRD §6.1): 1. Match sender email against persons table 2. If no person match, extract domain and match against company.website 3. Returns matchedCompany + matchedPerson for each email - Audit-logged (action: crm.mailsync:imap) - Zod validated (host, port, user, password, folder, limit) - 502 on IMAP connection failure with detail - 400 on validation failure Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed
1 parent
7acc0c5451
commit
e45b68bb8b
1089 files changed
+260614
No files matched your search
Executable
+3
@@ -0,0 +1,3 @@
|
||||
outputs/
|
||||
__pycache__/
|
||||
.pytest_cache/
|
||||
Executable
+111
@@ -0,0 +1,111 @@
|
||||
---
|
||||
name: aminer-deep-search
|
||||
version: 2.0.0
|
||||
author: AMiner
|
||||
contact: report@aminer.cn
|
||||
description: >
|
||||
Activate this skill when the user wants deep, multi-round academic paper collection for a survey or literature review.
|
||||
The host model (the model running this skill) drives the loop itself: it expands queries, judges relevance, snowballs backward references, and decides when to stop.
|
||||
The bundled scripts are pure tool commands that call documented AMiner Open Platform endpoints and print JSON tool results only — no extra LLM configuration is needed.
|
||||
Use this skill for broad topic exploration, survey bibliography construction, and collecting hundreds of candidate papers with AMiner IDs and titles.
|
||||
Not intended for single-paper lookup or lightweight recommendations; use aminer-free-academic or aminer-daily-paper for those.
|
||||
metadata:
|
||||
{
|
||||
"openclaw":
|
||||
{
|
||||
"requires": {
|
||||
"bins": ["python3"],
|
||||
"env": ["AMINER_API_KEY"]
|
||||
},
|
||||
"primaryEnv": "AMINER_API_KEY"
|
||||
}
|
||||
}
|
||||
---
|
||||
|
||||
# AMiner Deep Search
|
||||
|
||||
Host-model-driven survey paper collection. You (the model reading this) are the controller: run the tool scripts, read their JSON output, judge relevance yourself, and iterate until the collection target is met.
|
||||
|
||||
## Scope
|
||||
|
||||
- Use for: survey bibliography collection (hundreds of papers), keyword expansion, backward-citation snowballing.
|
||||
- Do not use for: single-paper lookup or Q&A (route to `aminer-free-academic`), personalized recommendations (route to `aminer-daily-paper`).
|
||||
|
||||
## Pre-flight
|
||||
|
||||
1. Check the key without printing it:
|
||||
|
||||
```bash
|
||||
[ -z "${AMINER_API_KEY:-}" ] && echo "AMINER_API_KEY missing" || echo "AMINER_API_KEY exists"
|
||||
```
|
||||
|
||||
If missing, stop and ask the user to set `AMINER_API_KEY` (console: https://open.aminer.cn/open/board?tab=control). Never print the key.
|
||||
|
||||
2. Confirm the `topic` and the `target-size` (default 400). If your round plan is estimated to cost ¥5 or more, tell the user the estimate and get confirmation before starting.
|
||||
|
||||
## Tools
|
||||
|
||||
Both scripts live in `scripts/` under this skill directory. They print exactly one JSON document to stdout (the tool result); diagnostics and a `[cost]` line go to stderr. They never score relevance — that is your job.
|
||||
|
||||
### `scripts/aminer_api.py` — AMiner API calls
|
||||
|
||||
| Subcommand | Endpoint | Price |
|
||||
|---|---|---|
|
||||
| `search --query Q [--size 20] [--year YYYY] [--order n_citation\|year] [--max-pages 3]` | GET `/api/paper/search/pro` + free `paper/info` enrichment | ¥0.01/page |
|
||||
| `qa-search [--query "natural language question"] [--topic-high '[["termA","termB"],["termC"]]'] [--size 20] [--year-from Y] [--year-to Y] [--citation-sort]` | POST `/api/paper/qa/search` (always `use_topic=true`; the backend ignores `query` when `use_topic=false`) + free enrichment | ¥0.05/call |
|
||||
| `info --ids id1 id2 ...` | POST `/api/paper/info` (batched ≤100 ids) | Free |
|
||||
| `references --ids id1 id2 ... [--per-seed 20]` | GET `/api/paper/relation` per seed + free enrichment | ¥0.10/seed |
|
||||
|
||||
Output shape: `search`/`qa-search`/`info` print `[{id, title, year?, venue?, abstract_slice?}]`; `references` additionally includes `source_paper_ids` (which seeds cited the paper). Seeds themselves are excluded from `references` output.
|
||||
|
||||
### `scripts/paper_set.py` — cross-round state file (no network)
|
||||
|
||||
State file defaults to `outputs/paper_set.json` relative to the working directory.
|
||||
|
||||
```bash
|
||||
# Merge kept results (pipe the filtered JSON array in), dedupe by id
|
||||
python3 scripts/aminer_api.py search --query "..." | python3 scripts/paper_set.py add
|
||||
# → {"added": N, "duplicates": M, "total": T}
|
||||
|
||||
python3 scripts/paper_set.py stats # totals, expanded_seeds, by_year
|
||||
python3 scripts/paper_set.py mark-expanded --ids id1 id2 # record snowballed seeds
|
||||
python3 scripts/paper_set.py export -o outputs/final_papers.json
|
||||
```
|
||||
|
||||
`add` also accepts `--ids id1 id2 ...` for bare IDs. Items carrying `source_paper_ids` (from `references`) automatically mark those seeds as expanded.
|
||||
|
||||
If you want to filter before adding, read the search output first, then pipe only the kept items:
|
||||
|
||||
```bash
|
||||
printf '%s' '[{"id":"...","title":"..."}]' | python3 scripts/paper_set.py add
|
||||
```
|
||||
|
||||
## Round Protocol (core)
|
||||
|
||||
### Round 0 — plan
|
||||
|
||||
- Derive 4–8 seed queries from the topic: synonyms, subfields, method names, datasets/benchmarks, common English abbreviations.
|
||||
- Estimate rounds and cost (searches ≈ ¥0.01–0.05 each, references ≈ ¥0.10/seed). If the estimate is ≥¥5, confirm with the user first.
|
||||
|
||||
### Each round (default budget: 12 rounds), five fixed steps
|
||||
|
||||
1. **Search**: run 1–4 `search` / `qa-search` calls from the pending query queue. Prefer `search` (cheaper); use `qa-search` when the query is a natural-language question.
|
||||
2. **Filter & add**: read the stdout results, judge relevance to the topic yourself, and pipe only the kept items into `paper_set.py add`. Never add papers you consider off-topic.
|
||||
3. **Check**: run `stats` to see the total and this round's increment.
|
||||
4. **Snowball**: from this round's relevant additions pick ≤5 strong seeds (highly relevant, ranked high under `--order n_citation`, not in `expanded_seeds`) and run `references --ids ...`. Filter the output for relevance, then add it. Run `mark-expanded` for seeds that yielded nothing addable.
|
||||
5. **Decide**: choose the next move —
|
||||
- a search returned <5 results or poor quality → replace it with a reformulated query (max 2 variants per direction, then switch to snowballing);
|
||||
- references are yielding many relevant papers → keep snowballing from fresh seeds;
|
||||
- reached `target-size`, or results are exhausted, or 2 consecutive rounds added <5 papers → terminate.
|
||||
|
||||
### Wrap-up
|
||||
|
||||
Run `export`, then report: final paper count, total cost (sum the `[cost]` stderr lines), and the output path.
|
||||
|
||||
## Rules
|
||||
|
||||
1. Never fabricate paper IDs or titles; only cite data actually returned by the tools.
|
||||
2. Free first: metadata always comes from the free `paper/info` (the scripts already do this); never call the paid `paper/detail` for bulk metadata.
|
||||
3. Keep the raw tool output out of your final answer; report counts and the exported file path instead.
|
||||
4. Never print or log `AMINER_API_KEY`.
|
||||
5. If AMiner returns fewer papers than the target, report the real count instead of inventing papers.
|
||||
Executable
+111
@@ -0,0 +1,111 @@
|
||||
---
|
||||
name: aminer-deep-search
|
||||
version: 2.0.0
|
||||
author: AMiner
|
||||
contact: report@aminer.cn
|
||||
description: >
|
||||
当用户需要为综述或文献回顾做深度、多轮的学术论文收集时激活本 skill。
|
||||
由宿主模型（正在运行本 skill 的模型）亲自驱动循环：扩展查询、判断相关性、做引用滚雪球、决定何时终止。
|
||||
附带脚本只是纯工具命令，调用 AMiner 开放平台的正规接口并只输出 JSON tool-result，无需配置任何额外 LLM。
|
||||
适用于大范围主题探索、综述参考文献构建、收集数百篇带 AMiner ID 与标题的候选论文。
|
||||
不适用于单篇论文查询或轻量推荐，请分别使用 aminer-free-academic 或 aminer-daily-paper。
|
||||
metadata:
|
||||
{
|
||||
"openclaw":
|
||||
{
|
||||
"requires": {
|
||||
"bins": ["python3"],
|
||||
"env": ["AMINER_API_KEY"]
|
||||
},
|
||||
"primaryEnv": "AMINER_API_KEY"
|
||||
}
|
||||
}
|
||||
---
|
||||
|
||||
# AMiner 深度收集
|
||||
|
||||
宿主模型直驱的综述文献收集。你（正在读这份文档的模型）就是控制器：运行工具脚本、阅读 JSON 输出、自己判断相关性、迭代直到达成收集目标。
|
||||
|
||||
## 适用边界
|
||||
|
||||
- 适用：综述级文献收集（数百篇）、关键词扩展、后向引用滚雪球。
|
||||
- 不适用：单篇查询或问答（走 `aminer-free-academic`）、个性化推荐（走 `aminer-daily-paper`）。
|
||||
|
||||
## Pre-flight
|
||||
|
||||
1. 检查 key（不打印值）：
|
||||
|
||||
```bash
|
||||
[ -z "${AMINER_API_KEY:-}" ] && echo "AMINER_API_KEY missing" || echo "AMINER_API_KEY exists"
|
||||
```
|
||||
|
||||
缺失时停止并让用户设置 `AMINER_API_KEY`（控制台：https://open.aminer.cn/open/board?tab=control）。绝不打印 key。
|
||||
|
||||
2. 确认 `topic` 和 `target-size`（默认 400）。若轮次计划预估费用 ≥¥5，先告知用户并获得确认再开始。
|
||||
|
||||
## 工具
|
||||
|
||||
两个脚本都在本 skill 目录的 `scripts/` 下。stdout 只输出一个 JSON 文档（即 tool-result）；诊断信息与 `[cost]` 费用行走 stderr。脚本不做任何相关性打分——那是你的工作。
|
||||
|
||||
### `scripts/aminer_api.py` — AMiner 接口调用
|
||||
|
||||
| 子命令 | 端点 | 价格 |
|
||||
|---|---|---|
|
||||
| `search --query Q [--size 20] [--year YYYY] [--order n_citation\|year] [--max-pages 3]` | GET `/api/paper/search/pro` + 免费 `paper/info` 补全 | ¥0.01/页 |
|
||||
| `qa-search [--query "自然语言问题"] [--topic-high '[["词A","词B"],["词C"]]'] [--size 20] [--year-from Y] [--year-to Y] [--citation-sort]` | POST `/api/paper/qa/search`（固定 `use_topic=true`：`use_topic=false` 时后端忽略 `query`）+ 免费补全 | ¥0.05/次 |
|
||||
| `info --ids id1 id2 ...` | POST `/api/paper/info`（≤100 个 id 分批） | 免费 |
|
||||
| `references --ids id1 id2 ... [--per-seed 20]` | 每个 seed 调 GET `/api/paper/relation` + 免费补全 | ¥0.10/篇 seed |
|
||||
|
||||
输出形状：`search`/`qa-search`/`info` 输出 `[{id, title, year?, venue?, abstract_slice?}]`；`references` 额外带 `source_paper_ids`（哪些 seed 引用了该论文），且结果中排除 seed 本身。
|
||||
|
||||
### `scripts/paper_set.py` — 跨轮状态文件（无网络）
|
||||
|
||||
状态文件默认是工作目录下的 `outputs/paper_set.json`。
|
||||
|
||||
```bash
|
||||
# 把保留的结果管道进去（可直接接在搜索后面），按 id 去重
|
||||
python3 scripts/aminer_api.py search --query "..." | python3 scripts/paper_set.py add
|
||||
# → {"added": N, "duplicates": M, "total": T}
|
||||
|
||||
python3 scripts/paper_set.py stats # 总量、expanded_seeds、按年分布
|
||||
python3 scripts/paper_set.py mark-expanded --ids id1 id2 # 记录已滚雪球的 seed
|
||||
python3 scripts/paper_set.py export -o outputs/final_papers.json
|
||||
```
|
||||
|
||||
`add` 也接受 `--ids id1 id2 ...` 直接加裸 id。带 `source_paper_ids` 的条目（来自 `references`）会自动把对应 seed 记为已扩展。
|
||||
|
||||
如需先筛选再入库，先阅读搜索输出，再只把保留的条目管道进去：
|
||||
|
||||
```bash
|
||||
printf '%s' '[{"id":"...","title":"..."}]' | python3 scripts/paper_set.py add
|
||||
```
|
||||
|
||||
## 每轮协议（核心）
|
||||
|
||||
### 第 0 轮 — 规划
|
||||
|
||||
- 从 topic 派生 4–8 个种子查询：同义词、子领域、方法名、数据集/基准、常用英文缩写。
|
||||
- 预估轮数与费用（搜索约 ¥0.01–0.05/次，references 约 ¥0.10/seed）。预估 ≥¥5 时先向用户确认。
|
||||
|
||||
### 每一轮（默认最多 12 轮），固定五步
|
||||
|
||||
1. **搜索**：对待查队列执行 1–4 个 `search` / `qa-search`。优先用 `search`（更便宜）；查询是自然语言问题时用 `qa-search`。
|
||||
2. **筛选入库**：阅读 stdout 结果，自己判断与主题的相关性，只把保留的条目管道进 `paper_set.py add`。绝不入库你认为跑题的论文。
|
||||
3. **查看进度**：跑 `stats` 查看总量与本轮增量。
|
||||
4. **滚雪球**：从本轮新增的相关论文里挑 ≤5 个强种子（高相关、`--order n_citation` 排前、不在 `expanded_seeds` 中），跑 `references --ids ...`。对输出再做相关性筛选后入库。没有可入库产出的 seed 用 `mark-expanded` 记录。
|
||||
5. **决策下一轮**：
|
||||
- 某个搜索结果 <5 条或质量差 → 换一个改写后的查询（每个方向最多试 2 个变体，之后转滚雪球）；
|
||||
- references 持续产出大量相关论文 → 继续从新 seed 滚雪球；
|
||||
- 达到 `target-size`、结果枯竭、或连续 2 轮新增 <5 篇 → 终止。
|
||||
|
||||
### 收尾
|
||||
|
||||
跑 `export`，然后报告：最终篇数、总费用（累加 stderr 的 `[cost]` 行）、输出文件路径。
|
||||
|
||||
## 规则
|
||||
|
||||
1. 绝不编造论文 ID 或标题；只引用工具实际返回的数据。
|
||||
2. 免费优先：元数据一律来自免费的 `paper/info`（脚本已内置）；绝不为批量元数据调付费的 `paper/detail`。
|
||||
3. 不要把原始工具输出塞进最终回答；只报告数量与导出文件路径。
|
||||
4. 绝不打印或记录 `AMINER_API_KEY`。
|
||||
5. 如果 AMiner 返回的论文少于目标，如实报告实际数量，不要编造。
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
description: AMiner deep multi-round paper collection for survey references
|
||||
argument-hint: "[research topic | topic: ... target-size: 400 max-rounds: 12]"
|
||||
allowed-tools: Read, Bash, Glob, Grep
|
||||
---
|
||||
|
||||
# /aminer-deep-search - AMiner Deep Search
|
||||
|
||||
User invoked the AMiner deep paper collection skill with the following arguments:
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
## Your task
|
||||
|
||||
Follow `${CLAUDE_PLUGIN_ROOT}/SKILL.md`. You are the controller: run the tool scripts, read their JSON output, judge relevance yourself, and iterate. There is no external LLM mode and nothing to configure beyond `AMINER_API_KEY`.
|
||||
|
||||
Use this command only for deep survey-style paper collection, not for simple paper lookup or lightweight recommendations.
|
||||
|
||||
### 1. Parse `$ARGUMENTS`
|
||||
|
||||
- `topic`: required research topic. Preserve the user's wording. If absent or too vague, ask for a concrete topic.
|
||||
- `target-size`: optional final paper target, default 400.
|
||||
- `max-rounds`: optional round budget, default 12.
|
||||
|
||||
### 2. Pre-flight
|
||||
|
||||
```bash
|
||||
[ -z "${AMINER_API_KEY:-}" ] && echo "AMINER_API_KEY missing" || echo "AMINER_API_KEY exists"
|
||||
```
|
||||
|
||||
If missing, stop and tell the user to set `AMINER_API_KEY`. Never print the key. The scripts are pure stdlib — no dependency installation is needed.
|
||||
|
||||
### 3. Run the round protocol
|
||||
|
||||
Execute the Round Protocol from `${CLAUDE_PLUGIN_ROOT}/SKILL.md`:
|
||||
|
||||
- Round 0: derive 4–8 seed queries, estimate cost; confirm with the user if the estimate is ≥¥5.
|
||||
- Each round (max `max-rounds`): search via
|
||||
`python3 "${CLAUDE_PLUGIN_ROOT}/scripts/aminer_api.py" search --query "..." --order n_citation`,
|
||||
filter results for relevance yourself, add the kept items via
|
||||
`python3 "${CLAUDE_PLUGIN_ROOT}/scripts/paper_set.py" add`,
|
||||
snowball with `references --ids ...` on ≤5 strong unexpanded seeds, and check `stats`.
|
||||
- Stop when `target-size` is reached, results are exhausted, or 2 consecutive rounds add <5 papers.
|
||||
|
||||
Keep the state file and exports under the current working directory (`outputs/`).
|
||||
|
||||
### 4. Present the result
|
||||
|
||||
Run `export` and report the final paper count, the total cost (sum of `[cost]` stderr lines), and the output path. If the run fails due to missing configuration or API errors, show the actionable error without exposing secrets. Never fabricate papers.
|
||||
+410
@@ -0,0 +1,410 @@
|
||||
#!/usr/bin/env python3
|
||||
"""AMiner Open Platform tool CLI for aminer-deep-search.
|
||||
|
||||
Pure-stdlib tool commands driven by the host model. Every subcommand prints a
|
||||
single JSON document to stdout (the tool result); diagnostics and the cost
|
||||
summary go to stderr. No relevance scoring or loop control happens here — the
|
||||
host model decides what to search, what to keep, and when to stop.
|
||||
|
||||
Endpoints (base: https://publicapi.chatglm.cn/chatglm_public/skill/aminer):
|
||||
GET /api/paper/search/pro keyword search, ¥0.01/page (subcommand: search)
|
||||
POST /api/paper/qa/search natural-language search, ¥0.05/call (qa-search)
|
||||
POST /api/paper/info batch metadata, free, <=100 ids (info, and enrichment)
|
||||
GET /api/paper/relation backward references, ¥0.10/seed (references)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
BASE_URL = "https://publicapi.chatglm.cn/chatglm_public/skill/aminer"
|
||||
CONSOLE_URL = "https://open.aminer.cn/open/board?tab=control"
|
||||
TIMEOUT_SECONDS = 30.0
|
||||
MAX_RETRIES = 3
|
||||
RETRYABLE_STATUS = frozenset({408, 429, 500, 502, 503, 504})
|
||||
PAPER_INFO_BATCH_LIMIT = 100
|
||||
SEARCH_PRO_PAGE_SIZE = 20
|
||||
ABSTRACT_SLICE_LIMIT = 300
|
||||
|
||||
PRICE_CNY = {
|
||||
"search_pro": 0.01,
|
||||
"qa_search": 0.05,
|
||||
"paper_info": 0.0,
|
||||
"relation": 0.10,
|
||||
}
|
||||
|
||||
_call_counts: dict[str, int] = {}
|
||||
|
||||
|
||||
class ToolError(Exception):
|
||||
"""Fatal error carrying a structured JSON payload for stdout."""
|
||||
|
||||
def __init__(self, payload: dict[str, Any], exit_code: int = 1) -> None:
|
||||
super().__init__(payload.get("message", "tool error"))
|
||||
self.payload = payload
|
||||
self.exit_code = exit_code
|
||||
|
||||
|
||||
def _get_token() -> str:
|
||||
token = (os.getenv("AMINER_API_KEY") or "").strip()
|
||||
if not token:
|
||||
raise ToolError(
|
||||
{
|
||||
"ok": False,
|
||||
"error": "missing_aminer_api_key",
|
||||
"message": "Set the AMINER_API_KEY environment variable before calling AMiner APIs.",
|
||||
"console": CONSOLE_URL,
|
||||
},
|
||||
exit_code=2,
|
||||
)
|
||||
return token
|
||||
|
||||
|
||||
def _redact(value: Any, token: str) -> Any:
|
||||
if isinstance(value, str):
|
||||
return value.replace(token, "[REDACTED]") if token else value
|
||||
if isinstance(value, list):
|
||||
return [_redact(item, token) for item in value]
|
||||
if isinstance(value, dict):
|
||||
return {key: _redact(item, token) for key, item in value.items()}
|
||||
return value
|
||||
|
||||
|
||||
def _request(api_name: str, method: str, path: str,
|
||||
params: dict[str, Any] | None = None,
|
||||
body: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Call one endpoint with retries; return the parsed JSON envelope."""
|
||||
token = _get_token()
|
||||
url = BASE_URL + path
|
||||
if method == "GET" and params:
|
||||
query = urllib.parse.urlencode(
|
||||
{k: v for k, v in params.items() if v is not None}
|
||||
)
|
||||
url = f"{url}?{query}"
|
||||
data = json.dumps(body, ensure_ascii=False).encode("utf-8") if body is not None else None
|
||||
headers = {"Authorization": token, "X-Platform": "openclaw"}
|
||||
if data is not None:
|
||||
headers["Content-Type"] = "application/json;charset=utf-8"
|
||||
|
||||
_call_counts[api_name] = _call_counts.get(api_name, 0) + 1
|
||||
|
||||
last_error: dict[str, Any] | None = None
|
||||
for attempt in range(1, MAX_RETRIES + 1):
|
||||
request = urllib.request.Request(url, data=data, headers=headers, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response:
|
||||
raw = response.read().decode("utf-8", errors="replace")
|
||||
envelope = json.loads(raw)
|
||||
if not isinstance(envelope, dict):
|
||||
return {"data": envelope}
|
||||
# AMiner can wrap errors in an HTTP-200 response, e.g. code 403
|
||||
# for insufficient balance/permission. Surface those instead of
|
||||
# letting them look like an empty result set.
|
||||
code = envelope.get("code")
|
||||
if isinstance(code, int) and code not in (0, 200):
|
||||
raise ToolError({
|
||||
"ok": False,
|
||||
"error": "api_error",
|
||||
"api": api_name,
|
||||
"code": code,
|
||||
"message": _redact(str(envelope.get("msg") or ""), token)[:300],
|
||||
})
|
||||
return envelope
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = _redact(exc.read().decode("utf-8", errors="replace")[:500], token)
|
||||
last_error = {
|
||||
"ok": False,
|
||||
"error": "http_error",
|
||||
"api": api_name,
|
||||
"status": exc.code,
|
||||
"detail": detail,
|
||||
}
|
||||
if exc.code not in RETRYABLE_STATUS:
|
||||
break
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||
last_error = {
|
||||
"ok": False,
|
||||
"error": "network_error",
|
||||
"api": api_name,
|
||||
"detail": _redact(str(getattr(exc, "reason", exc)), token)[:500],
|
||||
}
|
||||
except json.JSONDecodeError as exc:
|
||||
last_error = {
|
||||
"ok": False,
|
||||
"error": "invalid_response",
|
||||
"api": api_name,
|
||||
"detail": str(exc)[:200],
|
||||
}
|
||||
break
|
||||
if attempt < MAX_RETRIES:
|
||||
backoff = (2 ** (attempt - 1)) + random.uniform(0, 0.3)
|
||||
print(f"[retry] {api_name} attempt={attempt}/{MAX_RETRIES} wait={backoff:.1f}s",
|
||||
file=sys.stderr)
|
||||
time.sleep(backoff)
|
||||
|
||||
raise ToolError(last_error or {"ok": False, "error": "request_failed", "api": api_name})
|
||||
|
||||
|
||||
def _envelope_items(envelope: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
data = envelope.get("data")
|
||||
if isinstance(data, dict):
|
||||
data = data.get("data") or data.get("items") or []
|
||||
if not isinstance(data, list):
|
||||
return []
|
||||
return [item for item in data if isinstance(item, dict)]
|
||||
|
||||
|
||||
def _dedupe(items: list[str]) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
result: list[str] = []
|
||||
for item in items:
|
||||
cleaned = str(item or "").strip()
|
||||
if cleaned and cleaned not in seen:
|
||||
seen.add(cleaned)
|
||||
result.append(cleaned)
|
||||
return result
|
||||
|
||||
|
||||
def _paper_id(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or item.get("_id") or "").strip()
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# paper/info enrichment (free)
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _fetch_paper_info(ids: list[str]) -> dict[str, dict[str, Any]]:
|
||||
"""Batch-fetch lightweight cards; returns a map of id -> raw info entry."""
|
||||
info_by_id: dict[str, dict[str, Any]] = {}
|
||||
unique_ids = _dedupe(ids)
|
||||
for start in range(0, len(unique_ids), PAPER_INFO_BATCH_LIMIT):
|
||||
chunk = unique_ids[start:start + PAPER_INFO_BATCH_LIMIT]
|
||||
envelope = _request("paper_info", "POST", "/api/paper/info", body={"ids": chunk})
|
||||
for entry in _envelope_items(envelope):
|
||||
entry_id = _paper_id(entry)
|
||||
if entry_id:
|
||||
info_by_id[entry_id] = entry
|
||||
return info_by_id
|
||||
|
||||
|
||||
def _venue_text(entry: dict[str, Any]) -> str:
|
||||
venue = entry.get("venue")
|
||||
if isinstance(venue, dict):
|
||||
return str(venue.get("raw") or venue.get("name") or "")
|
||||
return str(venue or entry.get("raw") or entry.get("venue_name") or "")
|
||||
|
||||
|
||||
def _compact_paper(base: dict[str, Any], info: dict[str, Any] | None) -> dict[str, Any]:
|
||||
merged: dict[str, Any] = dict(info or {})
|
||||
item: dict[str, Any] = {
|
||||
"id": _paper_id(base) or _paper_id(merged),
|
||||
"title": str(base.get("title") or merged.get("title")
|
||||
or base.get("title_zh") or merged.get("title_zh") or "").strip(),
|
||||
}
|
||||
year = merged.get("year") or base.get("year")
|
||||
if year:
|
||||
item["year"] = year
|
||||
venue = _venue_text(merged) or _venue_text(base)
|
||||
if venue:
|
||||
item["venue"] = venue
|
||||
abstract = str(merged.get("abstract_slice") or merged.get("abstract") or "").strip()
|
||||
if abstract:
|
||||
item["abstract_slice"] = abstract[:ABSTRACT_SLICE_LIMIT]
|
||||
return item
|
||||
|
||||
|
||||
def _enrich(base_items: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Attach year/venue/abstract_slice from the free paper/info endpoint."""
|
||||
ids = [_paper_id(item) for item in base_items if _paper_id(item)]
|
||||
info_by_id = _fetch_paper_info(ids) if ids else {}
|
||||
papers: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for item in base_items:
|
||||
paper_id = _paper_id(item)
|
||||
if not paper_id or paper_id in seen:
|
||||
continue
|
||||
seen.add(paper_id)
|
||||
compact = _compact_paper(item, info_by_id.get(paper_id))
|
||||
if compact["id"] and compact["title"]:
|
||||
papers.append(compact)
|
||||
return papers
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Subcommands
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def cmd_search(args: argparse.Namespace) -> list[dict[str, Any]]:
|
||||
target = max(1, args.size)
|
||||
max_year = args.year
|
||||
collected: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for page in range(max(1, args.max_pages)):
|
||||
params: dict[str, Any] = {
|
||||
"keyword": args.query,
|
||||
"page": page,
|
||||
"size": SEARCH_PRO_PAGE_SIZE,
|
||||
}
|
||||
if args.order:
|
||||
params["order"] = args.order
|
||||
envelope = _request("search_pro", "GET", "/api/paper/search/pro", params=params)
|
||||
page_items = _envelope_items(envelope)
|
||||
if not page_items:
|
||||
break
|
||||
enriched = _enrich(page_items)
|
||||
for paper in enriched:
|
||||
if paper["id"] in seen:
|
||||
continue
|
||||
if max_year and paper.get("year") and int(paper["year"]) > max_year:
|
||||
continue
|
||||
seen.add(paper["id"])
|
||||
collected.append(paper)
|
||||
if len(collected) >= target or len(page_items) < SEARCH_PRO_PAGE_SIZE:
|
||||
break
|
||||
return collected[:target]
|
||||
|
||||
|
||||
def cmd_qa_search(args: argparse.Namespace) -> list[dict[str, Any]]:
|
||||
# The backend only uses the `query` keyword-extraction result when
|
||||
# use_topic=true; with use_topic=false it reads `title` only, so a
|
||||
# query-only request ends up with empty terms and returns envelope 403.
|
||||
# Therefore always send use_topic=true.
|
||||
body: dict[str, Any] = {
|
||||
"use_topic": True,
|
||||
"size": max(1, args.size),
|
||||
}
|
||||
if args.query:
|
||||
body["query"] = args.query
|
||||
if args.topic_high:
|
||||
body["topic_high"] = args.topic_high
|
||||
if not args.query and not args.topic_high:
|
||||
raise ToolError({
|
||||
"ok": False,
|
||||
"error": "invalid_request",
|
||||
"message": "qa-search requires --query and/or --topic-high",
|
||||
}, exit_code=2)
|
||||
if args.year_from or args.year_to:
|
||||
year_from = args.year_from or 1900
|
||||
year_to = args.year_to or time.gmtime().tm_year
|
||||
body["year"] = list(range(year_from, year_to + 1))
|
||||
if args.citation_sort:
|
||||
body["force_citation_sort"] = True
|
||||
envelope = _request("qa_search", "POST", "/api/paper/qa/search", body=body)
|
||||
return _enrich(_envelope_items(envelope))
|
||||
|
||||
|
||||
def cmd_info(args: argparse.Namespace) -> list[dict[str, Any]]:
|
||||
info_by_id = _fetch_paper_info(args.ids)
|
||||
return [_compact_paper(entry, None) for entry in info_by_id.values()]
|
||||
|
||||
|
||||
def cmd_references(args: argparse.Namespace) -> list[dict[str, Any]]:
|
||||
seed_ids = _dedupe(args.ids)
|
||||
per_seed = max(1, args.per_seed)
|
||||
sources_by_id: dict[str, list[str]] = {}
|
||||
ordered_ids: list[str] = []
|
||||
seed_set = set(seed_ids)
|
||||
for seed_id in seed_ids:
|
||||
envelope = _request("relation", "GET", "/api/paper/relation",
|
||||
params={"id": seed_id})
|
||||
cited_ids: list[str] = []
|
||||
for item in _envelope_items(envelope):
|
||||
cited = item.get("cited") or []
|
||||
if not isinstance(cited, list):
|
||||
continue
|
||||
for cited_item in cited:
|
||||
cited_id = _paper_id(cited_item) if isinstance(cited_item, dict) else str(cited_item or "").strip()
|
||||
if cited_id:
|
||||
cited_ids.append(cited_id)
|
||||
for cited_id in _dedupe(cited_ids)[:per_seed]:
|
||||
if cited_id in seed_set:
|
||||
continue
|
||||
if cited_id not in sources_by_id:
|
||||
sources_by_id[cited_id] = []
|
||||
ordered_ids.append(cited_id)
|
||||
sources_by_id[cited_id].append(seed_id)
|
||||
|
||||
papers = _enrich([{"id": paper_id} for paper_id in ordered_ids])
|
||||
for paper in papers:
|
||||
paper["source_paper_ids"] = sources_by_id.get(paper["id"], [])
|
||||
return papers
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# CLI
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="AMiner Open Platform tool commands (stdout: JSON only; diagnostics: stderr)."
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_search = sub.add_parser("search", help="Keyword search via paper/search/pro (¥0.01/page).")
|
||||
p_search.add_argument("--query", required=True)
|
||||
p_search.add_argument("--size", type=int, default=20, help="Papers to return (default 20).")
|
||||
p_search.add_argument("--year", type=int, default=None,
|
||||
help="Inclusive publication end year (client-side filter).")
|
||||
p_search.add_argument("--order", choices=["n_citation", "year"], default=None)
|
||||
p_search.add_argument("--max-pages", type=int, default=3,
|
||||
help="Max result pages to fetch (default 3).")
|
||||
p_search.set_defaults(func=cmd_search)
|
||||
|
||||
p_qa = sub.add_parser("qa-search", help="Natural-language search via paper/qa/search (¥0.05/call).")
|
||||
p_qa.add_argument("--query", default=None, help="Natural-language question.")
|
||||
p_qa.add_argument("--topic-high", default=None,
|
||||
help='Structured keywords, JSON string like [["termA","termB"],["termC"]] '
|
||||
"(outer AND, inner OR).")
|
||||
p_qa.add_argument("--size", type=int, default=20)
|
||||
p_qa.add_argument("--year-from", type=int, default=None)
|
||||
p_qa.add_argument("--year-to", type=int, default=None)
|
||||
p_qa.add_argument("--citation-sort", action="store_true",
|
||||
help="Sort entirely by citation count.")
|
||||
p_qa.set_defaults(func=cmd_qa_search)
|
||||
|
||||
p_info = sub.add_parser("info", help="Batch metadata via paper/info (free, <=100 ids/batch).")
|
||||
p_info.add_argument("--ids", nargs="+", required=True)
|
||||
p_info.set_defaults(func=cmd_info)
|
||||
|
||||
p_refs = sub.add_parser("references",
|
||||
help="Backward-reference expansion via paper/relation (¥0.10/seed).")
|
||||
p_refs.add_argument("--ids", nargs="+", required=True, help="Seed AMiner paper IDs.")
|
||||
p_refs.add_argument("--per-seed", type=int, default=20,
|
||||
help="Max references kept per seed (default 20).")
|
||||
p_refs.set_defaults(func=cmd_references)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def _print_cost_summary() -> None:
|
||||
if not _call_counts:
|
||||
return
|
||||
total = sum(PRICE_CNY.get(name, 0.0) * count for name, count in _call_counts.items())
|
||||
parts = " + ".join(f"{name} x{count}" for name, count in sorted(_call_counts.items()))
|
||||
print(f"[cost] {parts} = ¥{total:.2f}", file=sys.stderr)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
result = args.func(args)
|
||||
except ToolError as exc:
|
||||
print(json.dumps(exc.payload, ensure_ascii=False), file=sys.stdout)
|
||||
_print_cost_summary()
|
||||
return exc.exit_code
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
_print_cost_summary()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cross-round paper-set state file for aminer-deep-search (no network access).
|
||||
|
||||
Hundreds of collected papers do not fit in the host model's context, so this
|
||||
CLI keeps them in a JSON state file and deduplicates by AMiner paper ID.
|
||||
|
||||
Subcommands (stdout is always a single JSON document):
|
||||
add merge papers from stdin (JSON array) or --ids into the state file
|
||||
stats totals, expanded seeds, and year distribution
|
||||
export write the final [{"id","title"}] list and report path + count
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
DEFAULT_STATE_FILE = "outputs/paper_set.json"
|
||||
|
||||
|
||||
def _load_state(path: Path) -> dict[str, Any]:
|
||||
if not path.exists():
|
||||
return {"papers": {}, "expanded_seeds": []}
|
||||
with path.open("r", encoding="utf-8") as file:
|
||||
state = json.load(file)
|
||||
state.setdefault("papers", {})
|
||||
state.setdefault("expanded_seeds", [])
|
||||
return state
|
||||
|
||||
|
||||
def _save_state(path: Path, state: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as file:
|
||||
json.dump(state, file, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def _normalize_items(raw_items: Any) -> list[dict[str, Any]]:
|
||||
if not isinstance(raw_items, list):
|
||||
raise ValueError("input must be a JSON array of papers")
|
||||
items: list[dict[str, Any]] = []
|
||||
for raw in raw_items:
|
||||
if isinstance(raw, str):
|
||||
items.append({"id": raw.strip()})
|
||||
elif isinstance(raw, dict):
|
||||
items.append(raw)
|
||||
return items
|
||||
|
||||
|
||||
def cmd_add(args: argparse.Namespace) -> dict[str, Any]:
|
||||
if args.ids:
|
||||
items = [{"id": paper_id} for paper_id in args.ids]
|
||||
else:
|
||||
stdin_text = sys.stdin.read().strip()
|
||||
if not stdin_text:
|
||||
raise ValueError("no input: pipe a JSON array to stdin or pass --ids")
|
||||
items = _normalize_items(json.loads(stdin_text))
|
||||
|
||||
state_path = Path(args.file)
|
||||
state = _load_state(state_path)
|
||||
papers: dict[str, dict[str, Any]] = state["papers"]
|
||||
expanded: set[str] = set(state["expanded_seeds"])
|
||||
|
||||
added = 0
|
||||
duplicates = 0
|
||||
for item in items:
|
||||
paper_id = str(item.get("id") or item.get("_id") or "").strip()
|
||||
if not paper_id:
|
||||
continue
|
||||
# source_paper_ids on an item means those seeds have been expanded
|
||||
for seed_id in item.get("source_paper_ids") or []:
|
||||
expanded.add(str(seed_id))
|
||||
if paper_id in papers:
|
||||
duplicates += 1
|
||||
existing = papers[paper_id]
|
||||
for key, value in item.items():
|
||||
if key != "source_paper_ids" and value not in (None, "", []):
|
||||
existing.setdefault(key, value)
|
||||
else:
|
||||
added += 1
|
||||
papers[paper_id] = {
|
||||
key: value for key, value in item.items()
|
||||
if key != "source_paper_ids" and value not in (None, "", [])
|
||||
}
|
||||
|
||||
state["expanded_seeds"] = sorted(expanded)
|
||||
_save_state(state_path, state)
|
||||
return {"added": added, "duplicates": duplicates, "total": len(papers)}
|
||||
|
||||
|
||||
def cmd_mark_expanded(args: argparse.Namespace) -> dict[str, Any]:
|
||||
state_path = Path(args.file)
|
||||
state = _load_state(state_path)
|
||||
expanded = set(state["expanded_seeds"])
|
||||
expanded.update(str(paper_id).strip() for paper_id in args.ids if str(paper_id).strip())
|
||||
state["expanded_seeds"] = sorted(expanded)
|
||||
_save_state(state_path, state)
|
||||
return {"expanded_seeds": state["expanded_seeds"]}
|
||||
|
||||
|
||||
def cmd_stats(args: argparse.Namespace) -> dict[str, Any]:
|
||||
state = _load_state(Path(args.file))
|
||||
papers = state["papers"]
|
||||
by_year: dict[str, int] = {}
|
||||
missing_title = 0
|
||||
for paper in papers.values():
|
||||
year = paper.get("year")
|
||||
key = str(year) if year else "unknown"
|
||||
by_year[key] = by_year.get(key, 0) + 1
|
||||
if not paper.get("title"):
|
||||
missing_title += 1
|
||||
return {
|
||||
"total": len(papers),
|
||||
"expanded_seeds": state["expanded_seeds"],
|
||||
"missing_title": missing_title,
|
||||
"by_year": dict(sorted(by_year.items(), reverse=True)),
|
||||
}
|
||||
|
||||
|
||||
def cmd_export(args: argparse.Namespace) -> dict[str, Any]:
|
||||
state = _load_state(Path(args.file))
|
||||
final = [
|
||||
{"id": paper_id, "title": paper.get("title", "")}
|
||||
for paper_id, paper in state["papers"].items()
|
||||
if paper.get("title")
|
||||
]
|
||||
out_path = Path(args.output)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with out_path.open("w", encoding="utf-8") as file:
|
||||
json.dump(final, file, ensure_ascii=False, indent=2)
|
||||
return {"count": len(final), "path": str(out_path)}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Deduplicated paper-set state file (stdout: JSON only, no network)."
|
||||
)
|
||||
parser.add_argument("--file", default=DEFAULT_STATE_FILE,
|
||||
help=f"State file path (default {DEFAULT_STATE_FILE}).")
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_add = sub.add_parser("add", help="Merge papers from stdin JSON array or --ids.")
|
||||
p_add.add_argument("--ids", nargs="*", default=None,
|
||||
help="Add bare paper IDs instead of reading stdin.")
|
||||
p_add.set_defaults(func=cmd_add)
|
||||
|
||||
p_mark = sub.add_parser("mark-expanded", help="Record seed IDs whose references were expanded.")
|
||||
p_mark.add_argument("--ids", nargs="+", required=True)
|
||||
p_mark.set_defaults(func=cmd_mark_expanded)
|
||||
|
||||
p_stats = sub.add_parser("stats", help="Show totals, expanded seeds, and year distribution.")
|
||||
p_stats.set_defaults(func=cmd_stats)
|
||||
|
||||
p_export = sub.add_parser("export", help="Write the final [{id,title}] list.")
|
||||
p_export.add_argument("-o", "--output", default="outputs/final_papers.json")
|
||||
p_export.set_defaults(func=cmd_export)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
result = args.func(args)
|
||||
except (ValueError, json.JSONDecodeError) as exc:
|
||||
print(json.dumps({"ok": False, "error": "invalid_input", "message": str(exc)},
|
||||
ensure_ascii=False))
|
||||
return 2
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+235
@@ -0,0 +1,235 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
SKILL_DIR = Path(__file__).resolve().parents[1]
|
||||
MODULE_PATH = SKILL_DIR / "scripts" / "aminer_api.py"
|
||||
SPEC = importlib.util.spec_from_file_location("aminer_deep_search_api", MODULE_PATH)
|
||||
assert SPEC and SPEC.loader
|
||||
api = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules[SPEC.name] = api
|
||||
SPEC.loader.exec_module(api)
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, payload, status=200):
|
||||
self.payload = payload
|
||||
self.status = status
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return json.dumps(self.payload).encode("utf-8")
|
||||
|
||||
|
||||
def run_main(argv, urlopen):
|
||||
"""Run the CLI with a fake transport; return (exit_code, stdout, stderr)."""
|
||||
stdout, stderr = io.StringIO(), io.StringIO()
|
||||
env = {"AMINER_API_KEY": "test-token"}
|
||||
api._call_counts.clear()
|
||||
with mock.patch.dict(os.environ, env, clear=True), \
|
||||
mock.patch.object(api.urllib.request, "urlopen", urlopen), \
|
||||
contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
|
||||
code = api.main(argv)
|
||||
return code, stdout.getvalue(), stderr.getvalue()
|
||||
|
||||
|
||||
def paper_info_response(request):
|
||||
body = json.loads(request.data.decode("utf-8"))
|
||||
return FakeResponse({
|
||||
"code": 200, "success": True,
|
||||
"data": [
|
||||
{"id": pid, "title": f"Title {pid}",
|
||||
"year": (2000 + int(pid)) if pid.isdigit() else 2015,
|
||||
"venue": {"raw": "Venue X"}, "abstract_slice": "abs " * 200}
|
||||
for pid in body["ids"]
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
class SearchTests(unittest.TestCase):
|
||||
def test_search_paginates_filters_year_and_enriches(self):
|
||||
search_pages = {
|
||||
0: [{"id": "1", "title": "P1"}, {"id": "2", "title": "P2"}] +
|
||||
[{"id": str(i), "title": f"P{i}"} for i in range(3, 21)],
|
||||
1: [{"id": "21", "title": "P21"}],
|
||||
}
|
||||
calls = {"search": [], "info": 0}
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
url = request.full_url
|
||||
if "/api/paper/search/pro" in url:
|
||||
self.assertIn("keyword=graph", url)
|
||||
self.assertIn("order=n_citation", url)
|
||||
page = int(url.split("page=")[1].split("&")[0])
|
||||
calls["search"].append(page)
|
||||
return FakeResponse({"code": 200, "success": True,
|
||||
"data": search_pages.get(page, [])})
|
||||
if "/api/paper/info" in url:
|
||||
calls["info"] += 1
|
||||
return paper_info_response(request)
|
||||
raise AssertionError(f"unexpected url: {url}")
|
||||
|
||||
code, out, err = run_main(
|
||||
["search", "--query", "graph", "--size", "30",
|
||||
"--year", "2010", "--order", "n_citation", "--max-pages", "3"],
|
||||
fake_urlopen,
|
||||
)
|
||||
self.assertEqual(code, 0)
|
||||
papers = json.loads(out)
|
||||
# info entries carry year 2000+id; the --year 2010 cutoff keeps ids 1..10 and none from page 1 (id 21)
|
||||
self.assertEqual([p["id"] for p in papers], [str(i) for i in range(1, 11)])
|
||||
self.assertEqual(papers[0]["venue"], "Venue X")
|
||||
self.assertLessEqual(len(papers[0]["abstract_slice"]), api.ABSTRACT_SLICE_LIMIT)
|
||||
self.assertEqual(calls["search"], [0, 1]) # stops after exhausting pages
|
||||
self.assertIn("[cost]", err)
|
||||
self.assertNotIn("[cost]", out) # stdout stays pure JSON
|
||||
|
||||
def test_missing_key_returns_structured_error(self):
|
||||
stdout = io.StringIO()
|
||||
api._call_counts.clear()
|
||||
with mock.patch.dict(os.environ, {}, clear=True), \
|
||||
contextlib.redirect_stdout(stdout):
|
||||
code = api.main(["search", "--query", "x"])
|
||||
self.assertEqual(code, 2)
|
||||
payload = json.loads(stdout.getvalue())
|
||||
self.assertEqual(payload["error"], "missing_aminer_api_key")
|
||||
self.assertNotIn("test-token", stdout.getvalue())
|
||||
|
||||
|
||||
class InfoTests(unittest.TestCase):
|
||||
def test_info_batches_at_most_100_ids(self):
|
||||
batch_sizes = []
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
self.assertIn("/api/paper/info", request.full_url)
|
||||
body = json.loads(request.data.decode("utf-8"))
|
||||
batch_sizes.append(len(body["ids"]))
|
||||
return FakeResponse({"code": 200, "success": True,
|
||||
"data": [{"id": pid, "title": f"T{pid}"} for pid in body["ids"]]})
|
||||
|
||||
ids = [str(i) for i in range(150)]
|
||||
code, out, _ = run_main(["info", "--ids", *ids], fake_urlopen)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(batch_sizes, [100, 50])
|
||||
self.assertEqual(len(json.loads(out)), 150)
|
||||
|
||||
|
||||
class ReferencesTests(unittest.TestCase):
|
||||
def test_references_extracts_cited_excludes_seeds_and_tracks_sources(self):
|
||||
def fake_urlopen(request, timeout):
|
||||
url = request.full_url
|
||||
if "/api/paper/relation" in url:
|
||||
seed = url.split("id=")[1]
|
||||
cited = {
|
||||
"s1": [{"_id": "a"}, {"_id": "b"}, {"_id": "s2"}],
|
||||
"s2": [{"_id": "b"}, {"_id": "c"}],
|
||||
}[seed]
|
||||
return FakeResponse({"code": 200, "success": True,
|
||||
"data": [{"_id": seed, "cited": cited}]})
|
||||
if "/api/paper/info" in url:
|
||||
return paper_info_response(request)
|
||||
raise AssertionError(f"unexpected url: {url}")
|
||||
|
||||
code, out, _ = run_main(["references", "--ids", "s1", "s2"], fake_urlopen)
|
||||
self.assertEqual(code, 0)
|
||||
papers = {p["id"]: p for p in json.loads(out)}
|
||||
self.assertEqual(set(papers), {"a", "b", "c"}) # seed s2 excluded
|
||||
self.assertEqual(papers["a"]["source_paper_ids"], ["s1"])
|
||||
self.assertEqual(papers["b"]["source_paper_ids"], ["s1", "s2"])
|
||||
|
||||
def test_per_seed_limit(self):
|
||||
def fake_urlopen(request, timeout):
|
||||
url = request.full_url
|
||||
if "/api/paper/relation" in url:
|
||||
cited = [{"_id": f"r{i}"} for i in range(30)]
|
||||
return FakeResponse({"code": 200, "success": True,
|
||||
"data": [{"_id": "s1", "cited": cited}]})
|
||||
if "/api/paper/info" in url:
|
||||
return paper_info_response(request)
|
||||
raise AssertionError(f"unexpected url: {url}")
|
||||
|
||||
code, out, _ = run_main(["references", "--ids", "s1", "--per-seed", "5"], fake_urlopen)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(len(json.loads(out)), 5)
|
||||
|
||||
|
||||
class QaSearchTests(unittest.TestCase):
|
||||
def test_qa_search_always_uses_topic_mode(self):
|
||||
captured = {}
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
url = request.full_url
|
||||
if "/api/paper/qa/search" in url:
|
||||
captured["body"] = json.loads(request.data.decode("utf-8"))
|
||||
return FakeResponse({"code": 200, "success": True,
|
||||
"data": [{"id": "1", "title": "P1"}]})
|
||||
if "/api/paper/info" in url:
|
||||
return paper_info_response(request)
|
||||
raise AssertionError(f"unexpected url: {url}")
|
||||
|
||||
code, out, _ = run_main(
|
||||
["qa-search", "--query", "GNN scalability",
|
||||
"--topic-high", '[["graph neural network"]]', "--size", "5"],
|
||||
fake_urlopen,
|
||||
)
|
||||
self.assertEqual(code, 0)
|
||||
# with use_topic=false the backend ignores `query` (only reads `title`),
|
||||
# yielding empty terms and envelope 403 — so the script must always send true
|
||||
self.assertIs(captured["body"]["use_topic"], True)
|
||||
self.assertEqual(captured["body"]["query"], "GNN scalability")
|
||||
self.assertEqual(captured["body"]["topic_high"], '[["graph neural network"]]')
|
||||
self.assertEqual(json.loads(out)[0]["id"], "1")
|
||||
|
||||
def test_qa_search_requires_query_or_topic(self):
|
||||
code, out, _ = run_main(["qa-search"], lambda *a, **k: None)
|
||||
self.assertEqual(code, 2)
|
||||
self.assertEqual(json.loads(out)["error"], "invalid_request")
|
||||
|
||||
|
||||
class TransportTests(unittest.TestCase):
|
||||
def test_http_200_with_error_code_in_envelope_is_reported(self):
|
||||
def fake_urlopen(request, timeout):
|
||||
return FakeResponse({"code": 403, "success": True, "msg": "no data", "data": None})
|
||||
|
||||
code, out, _ = run_main(
|
||||
["qa-search", "--query", "scalable GNN approaches"], fake_urlopen,
|
||||
)
|
||||
self.assertEqual(code, 1)
|
||||
payload = json.loads(out)
|
||||
self.assertEqual(payload["error"], "api_error")
|
||||
self.assertEqual(payload["code"], 403)
|
||||
|
||||
|
||||
def test_non_retryable_http_error_fails_fast(self):
|
||||
attempts = []
|
||||
|
||||
def fake_urlopen(request, timeout):
|
||||
attempts.append(1)
|
||||
raise api.urllib.error.HTTPError(
|
||||
request.full_url, 401, "Unauthorized", {},
|
||||
io.BytesIO(b'{"msg":"bad token"}'),
|
||||
)
|
||||
|
||||
code, out, _ = run_main(["info", "--ids", "x"], fake_urlopen)
|
||||
self.assertEqual(code, 1)
|
||||
payload = json.loads(out)
|
||||
self.assertEqual(payload["status"], 401)
|
||||
self.assertEqual(len(attempts), 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
SKILL_DIR = Path(__file__).resolve().parents[1]
|
||||
MODULE_PATH = SKILL_DIR / "scripts" / "paper_set.py"
|
||||
SPEC = importlib.util.spec_from_file_location("aminer_deep_search_paper_set", MODULE_PATH)
|
||||
assert SPEC and SPEC.loader
|
||||
paper_set = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules[SPEC.name] = paper_set
|
||||
SPEC.loader.exec_module(paper_set)
|
||||
|
||||
|
||||
def run_main(argv, stdin_text=None):
|
||||
stdout = io.StringIO()
|
||||
with contextlib.redirect_stdout(stdout):
|
||||
if stdin_text is not None:
|
||||
with mock.patch.object(sys, "stdin", io.StringIO(stdin_text)):
|
||||
code = paper_set.main(argv)
|
||||
else:
|
||||
code = paper_set.main(argv)
|
||||
return code, stdout.getvalue()
|
||||
|
||||
|
||||
class PaperSetTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.state = str(Path(self.tmp.name) / "paper_set.json")
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_add_from_stdin_dedupes_and_counts(self):
|
||||
items = [
|
||||
{"id": "a", "title": "Paper A", "year": 2020},
|
||||
{"id": "b", "title": "Paper B"},
|
||||
]
|
||||
code, out = run_main(["--file", self.state, "add"], json.dumps(items))
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(json.loads(out), {"added": 2, "duplicates": 0, "total": 2})
|
||||
|
||||
again = [{"id": "a", "title": "Paper A"}, {"id": "c", "title": "Paper C"}]
|
||||
code, out = run_main(["--file", self.state, "add"], json.dumps(again))
|
||||
self.assertEqual(json.loads(out), {"added": 1, "duplicates": 1, "total": 3})
|
||||
|
||||
def test_add_bare_ids(self):
|
||||
code, out = run_main(["--file", self.state, "add", "--ids", "x", "y", "x"])
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(json.loads(out), {"added": 2, "duplicates": 1, "total": 2})
|
||||
|
||||
def test_source_paper_ids_mark_seeds_expanded(self):
|
||||
items = [{"id": "a", "title": "A", "source_paper_ids": ["seed1", "seed2"]}]
|
||||
run_main(["--file", self.state, "add"], json.dumps(items))
|
||||
code, out = run_main(["--file", self.state, "stats"])
|
||||
stats = json.loads(out)
|
||||
self.assertEqual(stats["expanded_seeds"], ["seed1", "seed2"])
|
||||
# source_paper_ids must not be persisted onto the paper entry
|
||||
state = json.loads(Path(self.state).read_text(encoding="utf-8"))
|
||||
self.assertNotIn("source_paper_ids", state["papers"]["a"])
|
||||
|
||||
def test_mark_expanded(self):
|
||||
run_main(["--file", self.state, "mark-expanded", "--ids", "s1"])
|
||||
code, out = run_main(["--file", self.state, "stats"])
|
||||
self.assertEqual(json.loads(out)["expanded_seeds"], ["s1"])
|
||||
|
||||
def test_stats_by_year(self):
|
||||
items = [
|
||||
{"id": "a", "title": "A", "year": 2021},
|
||||
{"id": "b", "title": "B", "year": 2021},
|
||||
{"id": "c", "title": "C"},
|
||||
]
|
||||
run_main(["--file", self.state, "add"], json.dumps(items))
|
||||
_, out = run_main(["--file", self.state, "stats"])
|
||||
stats = json.loads(out)
|
||||
self.assertEqual(stats["total"], 3)
|
||||
self.assertEqual(stats["by_year"]["2021"], 2)
|
||||
self.assertEqual(stats["by_year"]["unknown"], 1)
|
||||
|
||||
def test_export_shape_and_file(self):
|
||||
items = [
|
||||
{"id": "a", "title": "Paper A", "year": 2020, "venue": "V"},
|
||||
{"id": "no-title"},
|
||||
]
|
||||
run_main(["--file", self.state, "add"], json.dumps(items))
|
||||
out_path = str(Path(self.tmp.name) / "final.json")
|
||||
code, out = run_main(["--file", self.state, "export", "-o", out_path])
|
||||
self.assertEqual(code, 0)
|
||||
result = json.loads(out)
|
||||
self.assertEqual(result, {"count": 1, "path": out_path})
|
||||
exported = json.loads(Path(out_path).read_text(encoding="utf-8"))
|
||||
self.assertEqual(exported, [{"id": "a", "title": "Paper A"}])
|
||||
|
||||
def test_empty_stdin_is_a_structured_error(self):
|
||||
code, out = run_main(["--file", self.state, "add"], "")
|
||||
self.assertEqual(code, 2)
|
||||
self.assertEqual(json.loads(out)["error"], "invalid_input")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user