feat: IMAP mail sync with thread-to-contact matching (PRD §6.1)

Real IMAP connection via imapflow library:
- Connects to IMAP server (IMAPFlow, TLS on port 993)
- Fetches recent messages (configurable limit, default 20)
- Extracts: from, fromEmail, subject, date, snippet (200 chars)
- Thread-to-contact matching (PRD §6.1):
  1. Match sender email against persons table
  2. If no person match, extract domain and match against company.website
  3. Returns matchedCompany + matchedPerson for each email
- Audit-logged (action: crm.mailsync:imap)
- Zod validated (host, port, user, password, folder, limit)
- 502 on IMAP connection failure with detail
- 400 on validation failure

Tested: invalid creds → 502 'IMAP connection failed', missing host → 400
This commit is contained in:
Z User committed 2026-10-27 22:50:50 +00:00
1 parent 7acc0c5451
commit e45b68bb8b
1089 files changed
+260614

No files matched your search

+512
View File
@@ -0,0 +1,512 @@
---
name: ppt
metadata:
author: Z.AI
version: "1.0"
description: "Presentation creation, editing, and analysis for .pptx files: (1) Creating new presentations, (2) Modifying or editing content, (3) Working with layouts, (4) Adding comments or speaker notes. Academic/paper-based presentations use the embedded Beamer module at end of this file (PDF output only)."
license: Proprietary. LICENSE.txt has complete terms
---
# PPT creation, editing, and analysis
## Overview
A user may ask you to create, edit, or analyze the contents of a .pptx file. A .pptx file is essentially a ZIP archive containing XML files and other resources that you can read or edit. You have different tools and workflows available for different tasks.
## Reading and analyzing content
### Text extraction
To read the text content of a presentation, convert it to markdown:
```bash
python -m markitdown path-to-file.pptx
```
### Raw XML access
For comments, speaker notes, slide layouts, animations, design elements, or complex formatting, unpack the presentation and inspect its raw XML.
#### Unpacking a file
```
python ooxml/scripts/unpack.py <office_file> <output_dir>
```
**Note**: `unpack.py` is at `skills/pptx/ooxml/scripts/unpack.py` relative to the project root. If not found, run `find . -name "unpack.py"` to locate it.
#### Key file structures
| Path | Contents |
|------|----------|
| `ppt/presentation.xml` | Main metadata and slide references |
| `ppt/slides/slide{N}.xml` | Per-slide content |
| `ppt/notesSlides/notesSlide{N}.xml` | Speaker notes |
| `ppt/comments/modernComment_*.xml` | Slide comments |
| `ppt/slideLayouts/` | Layout templates |
| `ppt/slideMasters/` | Master slide templates |
| `ppt/theme/` | Theme and styling |
| `ppt/media/` | Images and other media |
#### Typography and color extraction
**When emulating an existing design**, extract typography and colors before starting:
1. **Theme file** — `ppt/theme/theme1.xml`: colors (`<a:clrScheme>`), fonts (`<a:fontScheme>`)
2. **Slide content** — `ppt/slides/slide1.xml`: actual font usage (`<a:rPr>`) and colors
3. **Global search** — grep for `<a:solidFill>`, `<a:srgbClr>`, and font references across all XML files
---
## Creating a new PowerPoint presentation **without a template**(most case,creating ppt/pptx from scrach)
When user upload pptx and ask for to use it as template ,do not follow this section routine! follow ## Editing an existing PowerPoint presentation(pptx)instead
You are an elite HTML-PPT design agent. You produce beautiful, content-rich, 1280x720 slide decks rendered as standalone HTML pages and committed to disk via the `Write` tool. Use the `Agent` tool to fan out the rendering of body slides to a **ppt sub-agent** (`subagent_type: ppt-expert`, falling back to `general-purpose` if unavailable) so multiple sections render in parallel.
You operate in four strict stages. Do NOT skip or reorder stages.
═══════════════════════════════════════════════════════════════════════
STAGE 1 — CLARIFY (collect mandatory inputs via `AskUserQuestion`)
═══════════════════════════════════════════════════════════════════════
**Step 0 — Reference detection (run BEFORE asking any question).**
Before invoking `AskUserQuestion`, look at the user's upload directory and
the user's prompt for a single reference asset. The case below is
auto-handleable; run the matching extraction script first,
write the result into `<work_dir>/slides/`,the <work_dir> is directly the download directory
then SKIP the palette and typography questions in the list below (the user has implicitly answered
them by handing you the reference). Still ask the remaining questions.
• **HTML reference** (`*.html` containing the desired deck-wide style):
```
python /home/z/my-project/skills/ppt/scripts/extract_html_style.py \
<reference.html> <work_dir>/slides
```
Writes `<work_dir>/slides/global.css` by concatenating every
`<style>...</style>` block from the source HTML verbatim. The
generated slides will reuse those CSS variables and class names
directly. After running this, `slides_brief.json.design` should
record `reference: "imitating <reference.html>"` and the model
should NOT redefine the palette/typography in `global.css`.
If neither reference exists, proceed to the questions below as normal.
**Step 1 — Ask the user.** Before doing anything else (after Step 0),
collect the user's explicit choices on the dimensions below.
**These are mandatory inputs — do NOT silently default any of them**,
even when the user's request hints at one. Explicit confirmation
prevents wrong-style decks and saves whole re-renders later.
Mandatory dimensions — ask between 4-6 questions(not exceed 6 questions) across the 10 items below.
Items marked **★** are strictly mandatory (palette, typography, per-page detail);
the rest should be asked unless the user's original request explicitly rules
them out (e.g. "internal deck, no notes" → skip notes question).
1. **主题与目的 (Topic & purpose)** — the deck's core theme and the main
aspects it should showcase. Ask this FIRST — every downstream choice
(length, layouts, data) hinges on it, so do not let it be implicit.
2. **目标受众 (Audience & tone)** — who is this for, what register.
3. **篇幅 (Deck length)** — tight range. like 1–8 / 8–12 / 12以上 slides,you must give user short slides choice.
4. **风格与基调 (Style & tone)** — the deck's overall voice as ONE preset:
商务 (business) / 简约 (minimal) / 正式汇报 (formal report). This is the
high-level register; the palette and typography questions below refine
it into concrete colors and fonts.
5. **Visual reference** — a recognizable style anchor (one or two named
reference decks / websites the user wants this to feel like).
6. **★ Palette (调色盘)** — ONE question that collects **all three** of
`background` / `primary` / `accent` together (as a unified palette
choice). Do NOT split into three separate questions. The options the
user picks from must be generated at runtime — do not hard-code
specific colors or hex codes here in the skill.
7. **Speaker notes (备注讲稿)** — none / short bullet hints / verbatim full script.
8. **已有素材 / 数据来源 (Existing materials / data sourcing)** — 用户是否
已提供素材或数据;是否需要在最终页面标注数据来源 (是 / 否)。
9. **Must include data**
10. **Need image search**
Important constraints when constructing the actual `AskUserQuestion` calls:
• **Think carefully before asking.** Before each `AskUserQuestion` call,
spend real reasoning on the user's domain, audience, and visual
reference (Q1 / Q4). The point of asking is to surface CHOICES — so
the choices you offer must themselves be a creative, well-considered
design proposal. Generic "Light / Dark / Other" is a failure mode.
• **Each option must be a concrete creative direction**, not a category
label. For the palette question that means each option names a coherent
trio (background + primary + accent) tuned to the user's topic — e.g.
a finance deck and a children's-book pitch should NEVER see the same
three palette options. Same rule for typography: each option proposes
a specific heading + body pairing that fits the deck's voice.
• **Mark your recommendation.** In every question, pick ONE option that
you genuinely believe is the strongest fit and put it FIRST, with the
suffix `(Recommended)` appended to its `label`. Use this for palette,
typography, per-page detail level, and visual reference — anywhere the
answer materially shapes the deck. Do NOT mark more than one option as
recommended per question.
• The palette question is a SINGLE question whose options each describe
a coherent (background + primary + accent) trio. The user picks one
trio, or chooses "Other" to provide their own three colors.
How to dispatch the questions:
• `AskUserQuestion` carries 4-6 questions per call.
• Each question: a short `header` (≤12 chars), the `question` itself, and
3–4 `options` (each with `label` + short `description`). The user can
always pick "Other" for free-text custom input.
• Do NOT call any other tool in the same turn as `AskUserQuestion`.
• Do NOT proceed to Stage 2 until every mandatory dimension has been
answered. Once answered, the palette / typography / detail-level
choices flow directly into `slides_brief.json.design` and the per-slide
`task_brief` in Stage 3 — they MUST match exactly.
═══════════════════════════════════════════════════════════════════════
STAGE 2 — RESEARCH (parallel web search + page visits + parallel image search)
═══════════════════════════════════════════════════════════════════════
Gather facts and visual assets.
Text research:
• Use `web_search` function for up-to-date facts(if the user do not provided). Batch 2-3 queries across parallel calls in one turn.
```bash
# Simple search query
z-ai function --name "web_search" --args '{"query": "artificial intelligence","num": 3}'
# Using short options
z-ai function -n web_search -a '{"query": "cryptocurrency news", "num": 3, "recency_days": 7}'
```
Image assets (THIS IS IMPORTANT):
• For EVERY slide that needs a photo or illustration, run the `z-ai image-search` CLI (provided by `z-ai-web-dev-sdk`). Prerequisite: `npm install -g z-ai-web-dev-sdk` (or `bun add -g z-ai-web-dev-sdk`), which installs the `z-ai` binary.
• Basic invocation:
z-ai image-search --query "<natural-language sentence>" --count 3
The CLI prints the JSON response (pretty-formatted) to stdout(--count control the image number return,use 5 by default). The query MUST be a natural-language sentence (NOT keywords). The response contains `results[].original_url` (OSS-hosted, embeddable) and `results[].caption`.
• **Output handling — print-to-stdout only.** pick the best `original_url`, and **bake that URL directly into the slide's `task_brief`** in `slides_brief.json` (see Stage 3). The sub-agent that renders the slide has no access to the search results, so the URL MUST be inlined into `task_brief` — that is the only carrier.
• Very important!!! if you want to use local image, YOU MUST use RELATIVE path rather than absolute path
• You MAY emit several `Bash` calls in the SAME assistant turn — they will run concurrently. Prefer batching 3–4 image searches per turn to save wall time.
• HARD LIMIT: invoke the image-search CLI at most 6 times total across the whole trajectory. Plan your image needs up front and reuse returned URLs across multiple slides where appropriate.
• If anything goes wrong (e.g. `Unknown command "image-search"`, auth errors, empty results, region tuning, `--no-rank` for speed) consult the image-search skill — it documents every flag, the full response schema, and the standard troubleshooting matrix.
═══════════════════════════════════════════════════════════════════════
STAGE 3 — PLAN (commit design as files on disk before fan-out)
═══════════════════════════════════════════════════════════════════════
After research, commit the deck design(global css) and slide brief(slides_brief.json) to disk so the build stage and every sub-agent share the SAME source of truth.
**Step 3.0 — After `Reasearch` succeeds**, in a SINGLE assistant turn, use `Write` to create:
**`<work_dir>` definition (MANDATORY)**: `<work_dir>` IS the user's `download/` directory itself. Do NOT create an intermediate themed/topic-named subdirectory (e.g. `download/theme_ppt/`, `download/<topic>/`). The slides folder must end up at `download/slides/`, NOT `download/<anything>/slides/`. Every `<work_dir>/...` path in this skill resolves under `download/` directly.
You should first create a directory '<work_dir>/slides' to keep the produced files
**Directory layout (MANDATORY)**: `global.css`, `slides_brief.json`, AND every `slide_NN.html` MUST all live under the SAME single directory `<work_dir>/slides/` (i.e. `download/slides/`). Do NOT place `global.css` or `slides_brief.json` at `<work_dir>/` root, do NOT split slides into per-section subfolders, and do NOT create an `images/` subfolder — image-search results are NEVER written to disk; the picked `original_url` is inlined directly into each slide's `task_brief` (see Stage 2). The flat-under-`slides/` layout is what every sub-agent assumes when it reads the brief and resolves the `<link rel="stylesheet">` href.
Final on-disk layout after Stage 3:
```
<work_dir>/slides/
├── global.css
├── slides_brief.json
├── slide_01.html ← written in Stage 4 by sub-agents
├── slide_02.html
└── ...
```
1. `<work_dir>/slides/global.css` — the deck-wide stylesheet that EVERY slide will `<link>` into its `<head>`. Bake in:
• You must use the font size /font style/color in the global.css, do not redefine it in each html
• CSS variables for the chosen palette: `--bg`, `--primary`, `--accent`, plus any tonal stops you need.
• `@import` lines for the chosen Google Fonts and a `:root` typography scale (title / subtitle / body / footnote sizes — see SLIDE QUALITY BAR).
• Reusable utility classes for the slide canvas (e.g. `.slide { width:1280px; min-height:720px; overflow:hidden; padding:64px; background:var(--bg); }`), card surfaces, and accent emphasis.
• NO per-slide layout CSS — that lives inside each slide's HTML. Keep this file under ~150 lines.
• Avoid CSS class collisions with Tailwind utilities
When the slide loads Tailwind via <script src="https://cdn.tailwindcss.com">, the CDN injects its utility styles into <head> after your <link
rel="stylesheet">. Custom classes that share a name with a Tailwind utility lose the cascade (same specificity, Tailwind comes later) and get silently
overridden.
Critical example: Do not name heading classes .h-1 / .h-2 / .h-3. In Tailwind these are height utilities (.h-1 = 4px, .h-2 = 8px, .h-3 = 12px). Applied to
a heading, the box collapses to a few pixels while the text still renders at full size — the heading visibly overlaps the element below it.
• Because `global.css` and the slide HTMLs are siblings inside `<work_dir>/slides/`, slides may link it as `<link rel="stylesheet" href="global.css">` (relative).
• keep these in mind!!! Global CSS scope — typography tokens only, no typography classes.Define typography design tokens (--font-heading, --font-body, --font-cn, --font-num, --fs-display, --fs-h1, --fs-h2, --fs-h3, --fs-body, --fs-small, --fs-micro) on :root in
global.css Do not ship pre-baked typography classes like .h-display / .h-1 / .h-2 / .h-3 / .cn-sub that bundle font-family, font-size, line-height,font-weight, and letter-spacing together. (This rule applies only to font family / size — palette, spacing, layout primitives, and other non-typography utilities can stilllive in global.css as before.) keep these in mind!!!
2. `<work_dir>/slides/slides_brief.json` — the dispatch manifest. Schema:
```json
{
"design": {
"title":"(ppt title)",
"style_name": "...",
"palette": {"background": "#...", "primary": "#...", "accent": "#..."},
"typography": {"heading": "xx", "body": "xx", "numeric": "xx"},
"reference": "Apple keynote / The Verge / ..."
},
"global_css_path": "<absolute path to <work_dir>/slides/global.css>",
"slides_dir": "<absolute path to <work_dir>/slides>",
"language": "zh|en|bilingual",
"speaker_notes": "none | short | full",
"slides": [
{
"title": "...",
"layout": "cover",
"output_path": "<absolute path to <work_dir>/slides/slide_XX.html>",
"task_brief": "Self-contained brief — see rules below."
},
...
]
}
```
• Slide ORDER is the order of entries in the `slides` array — no `position` / index field. To reorder a deck, reorder the list.
• Every slide MUST have `title`, `layout`, `output_path`, `task_brief`.
• `output_path` is the ABSOLUTE path the slide HTML will be written to. Filenames are arbitrary stable identifiers (e.g. `slide_01.html`, `slide_02.html` at first creation) — they live in the same directory as `global.css` and `slides_brief.json`, but their alphabetical order does NOT define slide order. Once assigned, do NOT rename a slide's file when reordering or inserting; keep the filename stable and just move its entry in the list.
• `layout` MUST be chosen per page from a coherent catalog (cover, section header, key message, bento grid, split text+image, timeline, stats, comparison, quote, closing, etc.). Diversify layouts across pages — do NOT reuse the same layout for consecutive slides.
• Section structure for longer decks: once the deck gets long (roughly **≥10 pages**), split the body into 2–5 named chapters with a `section header` page at the start of each. A reasonable rhythm is 3–6 content pages per chapter; if a stretch goes past ~7 pages without a break, that's a hint another section header would help. Section-header `task_brief` typically carries: chapter number ("01 / 03"), chapter title, one short tagline; no body bullets, no images. For shorter decks, only add a section header if the content has a strong narrative break.
• `task_brief` is the SOLE input the sub-agent will see for that slide — it MUST be self-contained for the renderer.
– Keep it concise and to the point: carry exactly what the renderer needs, no filler, no restating the design block, no meta reminders.
– Restate the slide's exact text content: headline, body copy, data points, stats, quotes, key phrases. Copy verbatim — do not paraphrase loosely.
– Resolve every "Hero photo of …" / "icon of …" reference into the FULL image URL (https://…) from your `z-ai image-search` results. The sub-agent has NO access to search results — if a URL isn't in the brief (either inlined here or read back from a saved `images/<slot>.json` manifest), the slide can't display the image.
– If the user asked for speaker notes in Stage 1, append a `Speaker notes:` block at the END of the brief. write the notes request. do NOT write notes verbatim here. The top-level `speaker_notes` field (`none` / `short` / `full`) tells the sub-agent whether and how deeply to generate notes from the rendered slide. The sub-agent will inject them into the slide HTML as `<aside data-notes>…</aside>` (visually hidden). If the user did NOT request notes, omit this block — do not invent notes.
– Add any layout-specific payload the sub-agent needs (chart data values, ordered timeline events, comparison columns, exact quote attribution, etc.). The committed `design` block (palette, typography, reference) is already in `slides_brief.json`, so do NOT repeat it inside each `task_brief`.
– Do NOT carry meta / quality-bar reminders inside the brief: word-count targets, "diversify layouts", "no photos", "verify contrast", etc. Those live ONCE in the sub-agent's self-audit checklist (Stage 4) — repeating them per slide is noise. If the slide is image-free, simply do not include any image URL; the sub-agent will not invent one.
═══════════════════════════════════════════════════════════════════════
STAGE 4 — BUILD ( FAN OUT `Agent` sub-agents to actually write the slides)
═══════════════════════════════════════════════════════════════════════
every slide, cover / TOC / body / closing, is committed by a `ppt` sub-agent that calls the `Write` tool itself. The main agent NEVER calls `Write` to commit slide HTML. This keeps rendering uniformly parallel and removes any temptation to serialize work onto the main turn.
• `Agent(subagent_type="ppt-expert", description=..., prompt=...)` — spawn a sub-agent that writes a CONTIGUOUS RANGE of slides. Use for every slide in the deck — cover, intro, data, cases, narrative chapters, closing, thank-you. Single-slide groups are fine when a section (or the cover) stands alone.
1. Immediately after Stage 3's files are on disk, emit ALL `Agent` calls in ONE assistant turn. Everything in that turn runs concurrently — this is ~3–5x faster than sequential iteration.
2. How to group the slides:
• A group is a CONTIGUOUS RANGE of entries in the `slides` list (e.g. indices 0..4 in JSON order). The sub-agent reads the briefs for those entries from `slides_brief.json` and writes each slide's HTML to its `output_path` in list order — locking in a consistent layout/component language across a section.
• Slides in DIFFERENT groups run IN PARALLEL.
• Keep groups small (5–7 slides). Good groups follow narrative sections: cover / intro / data / cases / closing.
• HARD CAP: at most **3 `Agent` calls per deck total**. If the body has more sections than that, MERGE adjacent sections into a single group rather than spawning a 6th. The cap is real — more groups means more sub-agent spin-up overhead AND weaker visual consistency across the deck.
3. Sub-agent dispatch prompt — every `Agent` call's `prompt` MUST explicitly carry these three things at the top, in this exact order:
a) the index range (0-based, half-open) into `slides_brief.json.slides[]` the sub-agent is responsible for (`start..end`),
b) the ABSOLUTE path to `slides_brief.json` (`<work_dir>/slides/slides_brief.json`),
c) the ABSOLUTE path to `global.css` (`<work_dir>/slides/global.css`). directly use it in every html
If any of the three is missing or relative, the sub-agent will fail to render. Do NOT hand-wave these as "see the brief" — write the literal absolute paths and the literal index numbers into every dispatch prompt.
Use this exact shape for every `Agent` call's `prompt` field (substitute the bracketed values per group):
```
You are a slide-rendering sub-agent. Render ONLY slides `slides[{start}:{end}]` (0-based, half-open) — do NOT touch other slides.
First Read these absolute paths:
• Brief manifest: {absolute path to <work_dir>/slides/slides_brief.json}
• Global stylesheet: {absolute path to <work_dir>/slides/global.css}
Both live in the SAME directory ({absolute path to <work_dir>/slides/}); write each slide HTML there as a sibling.
For each entry in `slides[{start}:{end}]`, in list order, produce one 1280x720 standalone HTML page that:
- starts with `<!DOCTYPE html>`, ends with `</html>`, and links `global.css` via `<link rel="stylesheet" href="global.css">` in `<head>`
- uses ONLY the palette/typography in `design`, and renders EVERY fact/headline/bullet/stat/quote/image URL in `task_brief` verbatim
- obeys every constraint in the SLIDE QUALITY BAR (canvas, contrast, layout discipline)
- speaker notes: if `speaker_notes` is `short`/`full`, generate notes (`short` = 3–5 bullet hints, `full` = ~80–150 word script; match deck language; cover any `notes_must_include:` points) embedded as the LAST `<body>` child `<aside data-notes class="hidden">…</aside>`. If `none`, omit it.
Commit each slide with `Write(file_path=<output_path from brief>, content=<full HTML>)`.
Update the worklog only once, when finished — one sentence summary per slide.
```
Pass `subagent_type: ppt-expert`. If the harness reports it as unavailable, fall back to `subagent_type: general-purpose` and prepend the rendering rules from this skill into the prompt.
4. The `Agent` tool result returns ONLY the sub-agent's textual summary(in one sentence) — it does NOT echo the HTML source. To inspect a generated slide, call `Read(file_path=<output_path>)`.
5. Sub-agent slide HTML rules: must start with `<!DOCTYPE html>`, end with `</html>`, link `global.css`, follow the SAME design palette / typography committed in `slides_brief.json`, and obey every constraint in the SLIDE QUALITY BAR. The sub-agent must NOT echo the HTML as plain text — only the `Write` call commits the slide.
═══════════════════════════════════════════════════════════════════════
Only when user requested a pptx file,if user not request,directly give a summary and end the task — how to EXPORT TO PPTX:
═══════════════════════════════════════════════════════════════════════
Convert `<work_dir>/slides/` into ONE `.pptx` via `batch_html2pptx.js`.
The converter walks sorted filename order. Right before invoking it,
ensure filenames sort alphabetically into the same order as
`slides_brief.json.slides[]`. If editing rounds left them out of order,
rename to `slide_{NN:02d}.html` matching list index AND update each
entry's `output_path`. This is the ONLY step that renames files.
You can directly run the files,do not try to modify it
Invoke:
```
cd /home/z/my-project && NODE_PATH=/usr/local/lib/node_modules nohup node /home/z/my-project/skills/ppt/batch_html2pptx.js /home/z/my-project/download/slides /home/z/my-project/download/xx.pptx
```
• Arg 1: slides directory. Arg 2 (optional): output path; defaults to
`<work_dir>/<basename(work_dir)>.pptx`.
• `NODE_PATH` is required so `pptxgenjs` / `playwright` / `sharp` resolve.
• Per-slide warnings (`🚨 CRITICAL OVERFLOW`, `⚠ BOUNDS/FONT/OVERLAP/LAYOUT`)
are non-fatal, but treat any `🚨 CRITICAL` as a real bug — fix the
source HTML and re-run (the pptx is overwritten in place).
After export: `ls -lh <path>` to confirm, then report the `.pptx` path
(and source slide HTML paths) to the user.
═══════════════════════════════════════════════════════════════════════
MULTI-ROUND HTML EDITING — keep `slides_brief.json` in sync
═══════════════════════════════════════════════════════════════════════
When you only need to editing severals pages,directly read and editing them,when you need to change the entire design,try to edit global.css first
Very inportant!: Keep only one version of html slides directly in "slides/" directory,do not keep mutiple versions of html. and there is only one version of slides.
Very inportant!: when deleting or adding pages,first editing slides_brief.json first
`slides_brief.json` is the single source of truth for the deck's
structure. Slide ORDER is the order of entries in `slides[]` — there is
no `position` / index field, so adding / removing / reordering NEVER
requires renumbering. Whenever a follow-up edit ADDS or REMOVES a slide,
update the brief in the SAME turn — never let on-disk slides and the
brief diverge.
• Add a page → create the HTML file (any stable filename like
`slide_NEW1.html`) AND insert a matching entry into
`slides_brief.json.slides[]` at the desired position in the list
(`title`, `layout`, `output_path`, self-contained `task_brief`).
• Remove a page → delete the file AND remove its entry from the list.
• Reorder → just move entries within `slides[]`. Do NOT rename files.
In-place content tweaks only need a brief update if the change is large
enough that the old `task_brief` no longer describes the slide
(different headline, new image URL, layout switched). Typo fixes do not.
═══════════════════════════════════════════════════════════════════════
SLIDE QUALITY BAR (applies to every page)
═══════════════════════════════════════════════════════════════════════
• Exact canvas: 1280 × 720, `overflow-hidden`. No scroll.
• Real content density: every page carries 60-120 words of substantive text — no placeholders, no lorem ipsum, no "insert data here".
• Exactly the chosen palette on every page. Background/primary/accent must match `slides_brief.json.design.palette`.
• Every text color on every background must pass WCAG AA (4.5:1 contrast).
• Use TailwindCSS via CDN (`<script src="https://cdn.tailwindcss.com"></script>`) and link the deck-wide `global.css`. Use Google Fonts for typography.
• Diversify layouts across slides — vary between cover, section header, split text+image, bento grid, stats, timeline, comparison, quote, closing, etc. Never repeat the same layout back-to-back.
• Never fabricate image URLs. Only use URLs that came back from the `z-ai image-search` CLI.
• Include icons (Material Icons via `<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">` and `<i class="material-icons">name</i>`). No `<script>` icon loaders.
• Numbers, dates, and technical terms render in the proper numeric font per typography guide.
• No graphical timelines, no SVG/connector flowcharts, no code-drawn maps, no base64 images, no Reveal.js.
• Headers and footers are OPTIONAL, not mandatory. Add a page header/footer (deck title, page number, brand mark, footnote bar) only when user asked !!!
═══════════════════════════════════════════════════════════════════════
GENERAL RULES
═══════════════════════════════════════════════════════════════════════
• Batch aggressively. Prefer 1 turn with 5 tool calls over 5 turns with 1 tool call each.
• Parallelism is free — `WebSearch`, `WebFetch`, `Bash` (image-search), and the fan-out of `Agent` sub-agents all run concurrently when emitted in the same turn; use it.
• Always respond in the user's language.
## Editing an existing PowerPoint presentation(pptx)
Pick the approach by what you're editing:
- **Approach A — `python-pptx` script** — preferred for text replacement, deleting/reordering slides, and any edit that should preserve fonts/colors/layout. Simpler and safer than raw XML for content swaps.
- **Approach B — raw OOXML** — required for animations, transitions, comments, speaker notes XML, theme tweaks, custom layout edits — anything `python-pptx` can't reach.
### Approach A — `python-pptx` text replacement (preferred for text edits)
**Workflow**
1. **Inventory the deck** — walk every slide, recurse into GROUP shapes (`shape_type == 6`), `print(repr(para.text))`. Use the inventory as the source of truth for replacement keys; rendered text often contains hidden chars that won't survive copy-paste.
2. **Helpers** — keep the build script short:
```python
from pptx import Presentation
from pptx.enum.text import MSO_AUTO_SIZE
from pptx.oxml.ns import qn
from pptx.util import Emu, Pt
def iter_text_frames(shapes):
for s in shapes:
if s.shape_type == 6: # GROUP → recurse
yield from iter_text_frames(s.shapes)
elif s.has_text_frame:
yield s, s.text_frame
def _norm(s): # strip soft breaks before matching
return s.replace("\x0b", "").replace("\r", "").strip()
def replace_in_paragraph(p, new_text): # first-run replace preserves formatting
runs = p.runs
if not runs:
p.add_run().text = new_text; return
runs[0].text = new_text
for r in runs[1:]:
r._r.getparent().remove(r._r)
def apply_replacements(tf, mapping): # full-frame match, then per-paragraph
m = {_norm(k): v for k, v in mapping.items()}
full = "\n".join(p.text for p in tf.paragraphs)
if _norm(full) in m:
parts = m[_norm(full)].split("\n")
for i, p in enumerate(tf.paragraphs):
replace_in_paragraph(p, parts[i] if i < len(parts) else "")
return
for p in tf.paragraphs:
if _norm(p.text) in m:
replace_in_paragraph(p, m[_norm(p.text)])
def delete_slide(prs, idx): # call high-index first
sld = list(prs.slides._sldIdLst)[idx]
prs.part.drop_rel(sld.get(qn("r:id")))
prs.slides._sldIdLst.remove(sld)
```
**Budget every shape BEFORE generating replacement text (do this first)**
Most overflow bugs come from generating copy without knowing the target box's capacity. Before drafting any replacement, walk the deck once and emit a capacity manifest — then feed it to the content step as a hard constraint.
For each text-bearing shape collect: `slide_idx, shape_id, w_cm, h_cm, font_pt, orig_text, orig_len`. Derive:
- `chars_per_line ≈ w_cm / (font_pt × 0.014)` for CJK; multiply by ~2 for Latin. Mixed text: weight by character class.
- `lines ≈ h_cm / (font_pt × 0.0185)` (line-height ≈ 1.3).
- `budget = min(chars_per_line × lines, orig_len × 1.1)`. The template designer already tuned `orig_len` for readability — treat it as the ceiling, not a starting point.
- `role = "label"` if `h_cm < 1.5` OR `orig_len ≤ 8` OR `font_pt ≥ 20`; else `"body"`.
Rules the generation step MUST obey:
- **Label boxes**: short phrase only. No full sentences, no trailing punctuation, no "term + explanation" expansion. Hard cap = `max(orig_len, 8)`. SWOT tiles, timeline tags, KPI labels all fall here.
- **Body boxes**: stay within `budget`. Font size is inherited from the template; shrinking is a last resort, not plan A.
- If the content is genuinely longer and the layout permits, **grow the box itself** (`widen_to_fit(shape, Emu(...))` — see below) rather than shrinking the font. Check first that `left + width` won't collide with the next shape.
**Handling long replacement / unwanted wrapping after replacement**
When a longer replacement wraps to a new line, apply remedies in this order (cheapest first):
```python
def widen_to_fit(shape, max_grow_emu=Emu(0)):
"""Let PowerPoint size the shape to its text. Pass max_grow_emu>0 to also
grow the explicit width (centered on the original position) before sizing."""
if max_grow_emu:
shape.left -= max_grow_emu // 2
shape.width += max_grow_emu
shape.text_frame.word_wrap = True
shape.text_frame.auto_size = MSO_AUTO_SIZE.SHAPE_TO_FIT_TEXT
def shrink_text_to_fit(shape):
"""Keep the box fixed; let PowerPoint shrink the font to fit."""
shape.text_frame.word_wrap = True
shape.text_frame.auto_size = MSO_AUTO_SIZE.TEXT_TO_FIT_SHAPE
```
1. **Budget first (preferred).** Check `shape.width` × `font_size` from inventory and trim the replacement so it fits the original visual budget. Numeric badges / small label boxes (`width ≤ 0.7"`, `font_size ≥ 16pt`) hold ~3–4 chars max.
2. **Widen the shape** with `widen_to_fit(shape, Emu(...))` when the content is genuinely longer and there's free space next to it. Always check the shape isn't going to collide with a neighbor first (compare `left+width` against the next shape's `left`).
3. **Shrink the font** with `shrink_text_to_fit(shape)` only for tight-layout boxes (table cells, numeric badges) where widening would break the grid. Last resort — it visibly breaks the typographic rhythm.
Skip `word_wrap = False`: it makes text overflow the box invisibly in PowerPoint and looks broken when exported.
**Critical gotchas**
- **Soft line breaks (`\x0b`)** silently break exact-match. Always `_norm()` both keys and lookups.
- **GROUP shapes** (`shape_type == 6`) hide text frames — recurse.
- **First-run replace** preserves formatting; `paragraph.text = ...` destroys it.
- **Short tokens collide.** `"01"`, `"%"`, `"18"` recur across slides — keep identity mappings or scope per slide index, never global cross-mappings like `"18": "12"`.
- **Delete slides high-index first** — deleting index 5 first shifts every later index down by one.
## Code Style Guidelines
**IMPORTANT**: When generating code for PPTX operations:
- Write concise code
- Avoid verbose variable names and redundant operations
- Avoid unnecessary print statements
## Dependencies
Required dependencies (should already be installed):
- **markitdown**: `pip install "markitdown[pptx]"` (text extraction)
- **pptxgenjs**: `npm install -g pptxgenjs` (creating presentations)
- **playwright**: `npm install -g playwright@1.50.0` (HTML rendering)
- **react-icons**: `npm install -g react-icons react react-dom` (icons)
- **sharp**: `npm install -g sharp` (SVG rasterization and image processing)
- **LibreOffice**: `sudo apt-get install libreoffice` (PDF conversion)
- **Poppler**: `sudo apt-get install poppler-utils` (pdftoppm)
- **defusedxml**: `pip install defusedxml` (secure XML parsing)
+224
View File
@@ -0,0 +1,224 @@
// batch_html2pptx.js
// Convert slide_*.html files into a single .pptx using html2pptx1.js.
//
// Two modes:
//
// 1) Single-directory mode (default — most common after Stage 4):
// node batch_html2pptx.js <slides_dir> [output.pptx]
// Converts every slide_*.html under <slides_dir> (sorted) into ONE pptx.
// If [output.pptx] is omitted, the file is saved next to <slides_dir> as
// <parent_basename>.pptx (e.g. .../my_deck/slides → .../my_deck/my_deck.pptx).
//
// 2) Batch mode (legacy — walks <base>/ppt_*/download/*/slides):
// node batch_html2pptx.js --batch <base_dir>
// Each slides/ folder becomes one .pptx, saved in the parent <name>/ folder.
//
// NODE_PATH=/share/hc/utils/superz/p-p-p-x/ppt0403/node_modules \
// node batch_html2pptx.js <slides_dir>
const path = require('path');
const fs = require('fs');
const pptxgen = require('pptxgenjs');
const html2pptx = require(path.join(__dirname, 'html2pptx1.js'));
function safeReaddir(dir) {
if (!fs.existsSync(dir)) return [];
return fs.readdirSync(dir, { withFileTypes: true });
}
function findSlidesDirs(baseDir) {
// Match: <baseDir>/ppt_*/download/*/slides
const results = [];
for (const pptDir of safeReaddir(baseDir)) {
if (!/^ppt_/.test(pptDir.name) || !pptDir.isDirectory()) continue;
const downloadDir = path.join(baseDir, pptDir.name, 'download');
for (const sub of safeReaddir(downloadDir)) {
if (!sub.isDirectory()) continue;
const slidesDir = path.join(downloadDir, sub.name, 'slides');
if (fs.existsSync(slidesDir) && fs.statSync(slidesDir).isDirectory()) {
results.push(slidesDir);
}
}
}
return results;
}
function collectHtmlFiles(slidesDir) {
return fs.readdirSync(slidesDir)
.filter(f => /^slide.*\.html$/i.test(f))
.sort()
.map(f => path.join(slidesDir, f));
}
async function buildPptxFromDir(slidesDir, outputPath, options = {}) {
const { strict = false, fontConfig = null, concurrency = 4 } = options;
const htmlFiles = collectHtmlFiles(slidesDir);
if (htmlFiles.length === 0) {
console.log(` [skip] no slide_*.html in ${slidesDir}`);
return null;
}
console.log(` ${htmlFiles.length} slides → ${outputPath} (concurrency=${concurrency})`);
const pptx = new pptxgen();
pptx.layout = 'LAYOUT_16x9';
// Pre-allocate slides in HTML-file order so final slide order is deterministic
// regardless of parallel completion order. html2pptx will populate each
// pre-allocated slide via the `slide` option instead of calling addSlide().
const tasks = htmlFiles.map((htmlFile) => ({
htmlFile,
slide: pptx.addSlide(),
}));
const failures = [];
const mustFix = [];
let nextIndex = 0;
let doneCount = 0;
const total = tasks.length;
async function worker() {
while (true) {
const i = nextIndex++;
if (i >= total) return;
const { htmlFile, slide } = tasks[i];
const base = path.basename(htmlFile);
console.log(` [start ${i + 1}/${total}] ${base}`);
try {
const { warnings } = await html2pptx(htmlFile, pptx, { fontConfig, slide });
if (warnings && warnings.length > 0) {
warnings.forEach(w => {
// Severe overflow is a must-fix signal — surface it loudly but do
// NOT block output (pptx still writes, exit code stays 0).
if (w.includes('MUST FIX')) {
mustFix.push({ file: base, message: w });
console.error(` ${base}: ${w}`);
} else {
console.warn(` ${base}: ${w}`);
}
});
}
} catch (err) {
console.error(` [error] ${base}: ${err.message}`);
failures.push({ file: htmlFile, error: err });
if (strict) throw err;
} finally {
doneCount++;
console.log(` [done ${doneCount}/${total}] ${base}`);
}
}
}
const workerCount = Math.max(1, Math.min(concurrency, total));
await Promise.all(Array.from({ length: workerCount }, () => worker()));
if (failures.length > 0) {
console.warn(` ${failures.length} slide(s) failed during conversion.`);
}
// Severe-overflow slides do not block output, but they MUST be fixed — print a
// consolidated, hard-to-miss list so the caller knows exactly what to re-render.
if (mustFix.length > 0) {
console.error(`\n 🚨🚨 ${mustFix.length} slide(s) with SEVERE overflow — MUST FIX and re-run:`);
mustFix.forEach(({ file, message }) => console.error(` - ${file}: ${message}`));
console.error('');
}
await pptx.writeFile({ fileName: outputPath });
console.log(` saved: ${outputPath}`);
return outputPath;
}
function defaultOutputPath(slidesDir) {
// Save the pptx next to the slides/ folder (one folder up),
// named after the parent directory.
const parentDir = path.dirname(path.resolve(slidesDir));
return path.join(parentDir, `${path.basename(parentDir)}.pptx`);
}
async function runSingle(slidesDir, outputPath, options = {}) {
if (!fs.existsSync(slidesDir) || !fs.statSync(slidesDir).isDirectory()) {
throw new Error(`slides_dir not found or not a directory: ${slidesDir}`);
}
const out = outputPath || defaultOutputPath(slidesDir);
await buildPptxFromDir(slidesDir, out, options);
}
async function runBatch(baseDir, options = {}) {
const slidesDirs = findSlidesDirs(baseDir);
console.log(`Found ${slidesDirs.length} slides directories under ${baseDir}`);
for (const slidesDir of slidesDirs) {
const label = path.relative(baseDir, slidesDir);
console.log(`\nProcessing: ${label}`);
try {
const parentDir = path.dirname(slidesDir);
const outputPath = path.join(parentDir, `${path.basename(parentDir)}.pptx`);
await buildPptxFromDir(slidesDir, outputPath, options);
} catch (err) {
console.error(` [error] ${slidesDir}: ${err.message}`);
if (options.strict) throw err;
}
}
}
function printUsageAndExit(code = 1) {
console.error(
'Usage:\n' +
' node batch_html2pptx.js [--strict] [--concurrency=N] <slides_dir> [output.pptx]\n' +
' node batch_html2pptx.js [--strict] [--concurrency=N] --batch <base_dir>\n'
);
process.exit(code);
}
async function main() {
const args = process.argv.slice(2);
const strict = args.includes('--strict');
let concurrency = 8;
const concIdx = args.findIndex(a => a === '--concurrency' || a.startsWith('--concurrency='));
if (concIdx !== -1) {
const arg = args[concIdx];
if (arg.includes('=')) {
concurrency = parseInt(arg.split('=')[1], 10);
} else {
concurrency = parseInt(args[concIdx + 1], 10);
}
if (!Number.isFinite(concurrency) || concurrency < 1) {
console.error(`invalid --concurrency value`);
process.exit(1);
}
}
const filteredArgs = args.filter((arg, i) => {
if (arg === '--strict') return false;
if (arg === '--concurrency' || arg.startsWith('--concurrency=')) return false;
if (concIdx !== -1 && !args[concIdx].includes('=') && i === concIdx + 1) return false;
return true;
});
if (filteredArgs.length === 0) printUsageAndExit();
const runOpts = { strict, concurrency };
if (filteredArgs[0] === '--batch') {
if (!filteredArgs[1]) printUsageAndExit();
await runBatch(filteredArgs[1], runOpts);
} else if (filteredArgs[0] === '--dir') {
// Backwards-compatible alias for batch mode.
if (!filteredArgs[1]) printUsageAndExit();
await runBatch(filteredArgs[1], runOpts);
} else {
const slidesDir = filteredArgs[0];
const outputPath = filteredArgs[1];
await runSingle(slidesDir, outputPath, runOpts);
}
console.log('\nDone.');
}
main().catch(err => {
console.error('Fatal:', err);
process.exit(1);
});
+5046
View File
File diff suppressed because it is too large. Load diff
+159
View File
@@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""
Tool to pack a directory into a .docx, .pptx, or .xlsx file with XML formatting undone.
Example usage:
python pack.py <input_directory> <office_file> [--force]
"""
import argparse
import shutil
import subprocess
import sys
import tempfile
import defusedxml.minidom
import zipfile
from pathlib import Path
def main():
parser = argparse.ArgumentParser(description="Pack a directory into an Office file")
parser.add_argument("input_directory", help="Unpacked Office document directory")
parser.add_argument("output_file", help="Output Office file (.docx/.pptx/.xlsx)")
parser.add_argument("--force", action="store_true", help="Skip validation")
args = parser.parse_args()
try:
success = pack_document(
args.input_directory, args.output_file, validate=not args.force
)
# Show warning if validation was skipped
if args.force:
print("Warning: Skipped validation, file may be corrupt", file=sys.stderr)
# Exit with error if validation failed
elif not success:
print("Contents would produce a corrupt file.", file=sys.stderr)
print("Please validate XML before repacking.", file=sys.stderr)
print("Use --force to skip validation and pack anyway.", file=sys.stderr)
sys.exit(1)
except ValueError as e:
sys.exit(f"Error: {e}")
def pack_document(input_dir, output_file, validate=False):
"""Pack a directory into an Office file (.docx/.pptx/.xlsx).
Args:
input_dir: Path to unpacked Office document directory
output_file: Path to output Office file
validate: If True, validates with soffice (default: False)
Returns:
bool: True if successful, False if validation failed
"""
input_dir = Path(input_dir)
output_file = Path(output_file)
if not input_dir.is_dir():
raise ValueError(f"{input_dir} is not a directory")
if output_file.suffix.lower() not in {".docx", ".pptx", ".xlsx"}:
raise ValueError(f"{output_file} must be a .docx, .pptx, or .xlsx file")
# Work in temporary directory to avoid modifying original
with tempfile.TemporaryDirectory() as temp_dir:
temp_content_dir = Path(temp_dir) / "content"
shutil.copytree(input_dir, temp_content_dir)
# Process XML files to remove pretty-printing whitespace
for pattern in ["*.xml", "*.rels"]:
for xml_file in temp_content_dir.rglob(pattern):
condense_xml(xml_file)
# Create final Office file as zip archive
output_file.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(output_file, "w", zipfile.ZIP_DEFLATED) as zf:
for f in temp_content_dir.rglob("*"):
if f.is_file():
zf.write(f, f.relative_to(temp_content_dir))
# Validate if requested
if validate:
if not validate_document(output_file):
output_file.unlink() # Delete the corrupt file
return False
return True
def validate_document(doc_path):
"""Validate document by converting to HTML with soffice."""
# Determine the correct filter based on file extension
match doc_path.suffix.lower():
case ".docx":
filter_name = "html:HTML"
case ".pptx":
filter_name = "html:impress_html_Export"
case ".xlsx":
filter_name = "html:HTML (StarCalc)"
with tempfile.TemporaryDirectory() as temp_dir:
try:
result = subprocess.run(
[
"soffice",
"--headless",
"--convert-to",
filter_name,
"--outdir",
temp_dir,
str(doc_path),
],
capture_output=True,
timeout=10,
text=True,
)
if not (Path(temp_dir) / f"{doc_path.stem}.html").exists():
error_msg = result.stderr.strip() or "Document validation failed"
print(f"Validation error: {error_msg}", file=sys.stderr)
return False
return True
except FileNotFoundError:
print("Warning: soffice not found. Skipping validation.", file=sys.stderr)
return True
except subprocess.TimeoutExpired:
print("Validation error: Timeout during conversion", file=sys.stderr)
return False
except Exception as e:
print(f"Validation error: {e}", file=sys.stderr)
return False
def condense_xml(xml_file):
"""Strip unnecessary whitespace and remove comments."""
with open(xml_file, "r", encoding="utf-8") as f:
dom = defusedxml.minidom.parse(f)
# Process each element to remove whitespace and comments
for element in dom.getElementsByTagName("*"):
# Skip w:t elements and their processing
if element.tagName.endswith(":t"):
continue
# Remove whitespace-only text nodes and comment nodes
for child in list(element.childNodes):
if (
child.nodeType == child.TEXT_NODE
and child.nodeValue
and child.nodeValue.strip() == ""
) or child.nodeType == child.COMMENT_NODE:
element.removeChild(child)
# Write back the condensed XML
with open(xml_file, "wb") as f:
f.write(dom.toxml(encoding="UTF-8"))
if __name__ == "__main__":
main()
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env python3
"""Unpack and format XML contents of Office files (.docx, .pptx, .xlsx)"""
import argparse
import random
import zipfile
import defusedxml.minidom
from pathlib import Path
def main():
parser = argparse.ArgumentParser(description="Unpack an Office file into a directory")
parser.add_argument("office_file", help="Office file (.docx/.pptx/.xlsx)")
parser.add_argument("output_dir", help="Output directory")
args = parser.parse_args()
unpack_document(args.office_file, args.output_dir)
def unpack_document(input_file, output_dir):
"""Unpack an Office file into a directory and pretty-print all XML files."""
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(input_file) as zf:
zf.extractall(output_path)
for pattern in ["*.xml", "*.rels"]:
for xml_file in output_path.rglob(pattern):
pretty_print_xml(xml_file)
# For .docx files, suggest an RSID for tracked changes
if str(input_file).endswith(".docx"):
suggested_rsid = "".join(random.choices("0123456789ABCDEF", k=8))
print(f"Suggested RSID for edit session: {suggested_rsid}")
def pretty_print_xml(xml_file):
"""Pretty-print a single XML file in place."""
content = xml_file.read_text(encoding="utf-8")
dom = defusedxml.minidom.parseString(content)
xml_file.write_bytes(dom.toprettyxml(indent=" ", encoding="ascii"))
if __name__ == "__main__":
main()
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env python3
"""
Command line tool to validate Office document XML files against XSD schemas and tracked changes.
Usage:
python validate.py <dir> --original <original_file>
"""
import argparse
import sys
from pathlib import Path
from validation import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator
def main():
parser = argparse.ArgumentParser(description="Validate Office document XML files")
parser.add_argument(
"unpacked_dir",
help="Path to unpacked Office document directory",
)
parser.add_argument(
"--original",
required=True,
help="Path to original file (.docx/.pptx/.xlsx)",
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="Enable verbose output",
)
args = parser.parse_args()
# Validate paths
unpacked_dir = Path(args.unpacked_dir)
original_file = Path(args.original)
file_extension = original_file.suffix.lower()
if not unpacked_dir.is_dir():
sys.exit(f"Error: {unpacked_dir} is not a directory")
if not original_file.is_file():
sys.exit(f"Error: {original_file} is not a file")
if file_extension not in [".docx", ".pptx", ".xlsx"]:
sys.exit(f"Error: {original_file} must be a .docx, .pptx, or .xlsx file")
# Run validations
match file_extension:
case ".docx":
validators = [DOCXSchemaValidator, RedliningValidator]
case ".pptx":
validators = [PPTXSchemaValidator]
case _:
print(f"Error: Validation not supported for file type {file_extension}")
sys.exit(1)
# Run validators
success = True
for V in validators:
validator = V(unpacked_dir, original_file, verbose=args.verbose)
if not validator.validate():
success = False
if success:
print("All validations PASSED!")
sys.exit(0 if success else 1)
if __name__ == "__main__":
main()
+15
View File
@@ -0,0 +1,15 @@
"""
Validation modules for Word document processing.
"""
from .base import BaseSchemaValidator
from .docx import DOCXSchemaValidator
from .pptx import PPTXSchemaValidator
from .redlining import RedliningValidator
__all__ = [
"BaseSchemaValidator",
"DOCXSchemaValidator",
"PPTXSchemaValidator",
"RedliningValidator",
]
+948
View File
@@ -0,0 +1,948 @@
"""
Base validator with common validation logic for document files.
"""
import re
import tempfile
import zipfile
from pathlib import Path
import lxml.etree
class BaseSchemaValidator:
"""Base validator with common validation logic for document files."""
# Elements whose 'id' attributes must be unique within their file
# Format: element_name -> (attribute_name, scope)
# scope can be 'file' (unique within file) or 'global' (unique across all files)
UNIQUE_ID_REQUIREMENTS = {
# Word elements
"comment": ("id", "file"), # Comment IDs in comments.xml
"commentrangestart": ("id", "file"), # Must match comment IDs
"commentrangeend": ("id", "file"), # Must match comment IDs
"bookmarkstart": ("id", "file"), # Bookmark start IDs
"bookmarkend": ("id", "file"), # Bookmark end IDs
# Note: ins and del (track changes) can share IDs when part of same revision
# PowerPoint elements
"sldid": ("id", "file"), # Slide IDs in presentation.xml
"sldmasterid": ("id", "global"), # Slide master IDs must be globally unique
"sldlayoutid": ("id", "global"), # Slide layout IDs must be globally unique
"cm": ("authorid", "file"), # Comment author IDs
# Excel elements
"sheet": ("sheetid", "file"), # Sheet IDs in workbook.xml
"definedname": ("id", "file"), # Named range IDs
# Drawing/Shape elements (all formats)
"cxnsp": ("id", "file"), # Connection shape IDs
"sp": ("id", "file"), # Shape IDs
"pic": ("id", "file"), # Picture IDs
"grpsp": ("id", "file"), # Group shape IDs
}
# Mapping of element names to expected relationship types
# Subclasses should override this with format-specific mappings
ELEMENT_RELATIONSHIP_TYPES = {}
# Unified schema mappings for all Office document types
SCHEMA_MAPPINGS = {
# Document type specific schemas
"word": "ISO-IEC29500-4_2016/wml.xsd", # Word documents
"ppt": "ISO-IEC29500-4_2016/pml.xsd", # PowerPoint presentations
"xl": "ISO-IEC29500-4_2016/sml.xsd", # Excel spreadsheets
# Common file types
"[Content_Types].xml": "ecma/fouth-edition/opc-contentTypes.xsd",
"app.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd",
"core.xml": "ecma/fouth-edition/opc-coreProperties.xsd",
"custom.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd",
".rels": "ecma/fouth-edition/opc-relationships.xsd",
# Word-specific files
"people.xml": "microsoft/wml-2012.xsd",
"commentsIds.xml": "microsoft/wml-cid-2016.xsd",
"commentsExtensible.xml": "microsoft/wml-cex-2018.xsd",
"commentsExtended.xml": "microsoft/wml-2012.xsd",
# Chart files (common across document types)
"chart": "ISO-IEC29500-4_2016/dml-chart.xsd",
# Theme files (common across document types)
"theme": "ISO-IEC29500-4_2016/dml-main.xsd",
# Drawing and media files
"drawing": "ISO-IEC29500-4_2016/dml-main.xsd",
}
# Unified namespace constants
MC_NAMESPACE = "http://schemas.openxmlformats.org/markup-compatibility/2006"
XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace"
# Common OOXML namespaces used across validators
PACKAGE_RELATIONSHIPS_NAMESPACE = (
"http://schemas.openxmlformats.org/package/2006/relationships"
)
OFFICE_RELATIONSHIPS_NAMESPACE = (
"http://schemas.openxmlformats.org/officeDocument/2006/relationships"
)
CONTENT_TYPES_NAMESPACE = (
"http://schemas.openxmlformats.org/package/2006/content-types"
)
# Folders where we should clean ignorable namespaces
MAIN_CONTENT_FOLDERS = {"word", "ppt", "xl"}
# All allowed OOXML namespaces (superset of all document types)
OOXML_NAMESPACES = {
"http://schemas.openxmlformats.org/officeDocument/2006/math",
"http://schemas.openxmlformats.org/officeDocument/2006/relationships",
"http://schemas.openxmlformats.org/schemaLibrary/2006/main",
"http://schemas.openxmlformats.org/drawingml/2006/main",
"http://schemas.openxmlformats.org/drawingml/2006/chart",
"http://schemas.openxmlformats.org/drawingml/2006/chartDrawing",
"http://schemas.openxmlformats.org/drawingml/2006/diagram",
"http://schemas.openxmlformats.org/drawingml/2006/picture",
"http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing",
"http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing",
"http://schemas.openxmlformats.org/wordprocessingml/2006/main",
"http://schemas.openxmlformats.org/presentationml/2006/main",
"http://schemas.openxmlformats.org/spreadsheetml/2006/main",
"http://schemas.openxmlformats.org/officeDocument/2006/sharedTypes",
"http://www.w3.org/XML/1998/namespace",
}
def __init__(self, unpacked_dir, original_file, verbose=False):
self.unpacked_dir = Path(unpacked_dir).resolve()
self.original_file = Path(original_file)
self.verbose = verbose
# Set schemas directory
self.schemas_dir = Path(__file__).parent.parent.parent / "schemas"
# Get all XML and .rels files
patterns = ["*.xml", "*.rels"]
self.xml_files = [
f for pattern in patterns for f in self.unpacked_dir.rglob(pattern)
]
if not self.xml_files:
print(f"Warning: No XML files found in {self.unpacked_dir}")
def validate(self):
"""Run all validation checks and return True if all pass."""
raise NotImplementedError("Subclasses must implement the validate method")
def validate_xml(self):
"""Validate that all XML files are well-formed."""
errors = []
for xml_file in self.xml_files:
try:
# Try to parse the XML file
lxml.etree.parse(str(xml_file))
except lxml.etree.XMLSyntaxError as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {e.lineno}: {e.msg}"
)
except Exception as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Unexpected error: {str(e)}"
)
if errors:
print(f"FAILED - Found {len(errors)} XML violations:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - All XML files are well-formed")
return True
def validate_namespaces(self):
"""Validate that namespace prefixes in Ignorable attributes are declared."""
errors = []
for xml_file in self.xml_files:
try:
root = lxml.etree.parse(str(xml_file)).getroot()
declared = set(root.nsmap.keys()) - {None} # Exclude default namespace
for attr_val in [
v for k, v in root.attrib.items() if k.endswith("Ignorable")
]:
undeclared = set(attr_val.split()) - declared
errors.extend(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Namespace '{ns}' in Ignorable but not declared"
for ns in undeclared
)
except lxml.etree.XMLSyntaxError:
continue
if errors:
print(f"FAILED - {len(errors)} namespace issues:")
for error in errors:
print(error)
return False
if self.verbose:
print("PASSED - All namespace prefixes properly declared")
return True
def validate_unique_ids(self):
"""Validate that specific IDs are unique according to OOXML requirements."""
errors = []
global_ids = {} # Track globally unique IDs across all files
for xml_file in self.xml_files:
try:
root = lxml.etree.parse(str(xml_file)).getroot()
file_ids = {} # Track IDs that must be unique within this file
# Remove all mc:AlternateContent elements from the tree
mc_elements = root.xpath(
".//mc:AlternateContent", namespaces={"mc": self.MC_NAMESPACE}
)
for elem in mc_elements:
elem.getparent().remove(elem)
# Now check IDs in the cleaned tree
for elem in root.iter():
# Get the element name without namespace
tag = (
elem.tag.split("}")[-1].lower()
if "}" in elem.tag
else elem.tag.lower()
)
# Check if this element type has ID uniqueness requirements
if tag in self.UNIQUE_ID_REQUIREMENTS:
attr_name, scope = self.UNIQUE_ID_REQUIREMENTS[tag]
# Look for the specified attribute
id_value = None
for attr, value in elem.attrib.items():
attr_local = (
attr.split("}")[-1].lower()
if "}" in attr
else attr.lower()
)
if attr_local == attr_name:
id_value = value
break
if id_value is not None:
if scope == "global":
# Check global uniqueness
if id_value in global_ids:
prev_file, prev_line, prev_tag = global_ids[
id_value
]
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {elem.sourceline}: Global ID '{id_value}' in <{tag}> "
f"already used in {prev_file} at line {prev_line} in <{prev_tag}>"
)
else:
global_ids[id_value] = (
xml_file.relative_to(self.unpacked_dir),
elem.sourceline,
tag,
)
elif scope == "file":
# Check file-level uniqueness
key = (tag, attr_name)
if key not in file_ids:
file_ids[key] = {}
if id_value in file_ids[key]:
prev_line = file_ids[key][id_value]
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {elem.sourceline}: Duplicate {attr_name}='{id_value}' in <{tag}> "
f"(first occurrence at line {prev_line})"
)
else:
file_ids[key][id_value] = elem.sourceline
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print(f"FAILED - Found {len(errors)} ID uniqueness violations:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - All required IDs are unique")
return True
def validate_file_references(self):
"""
Validate that all .rels files properly reference files and that all files are referenced.
"""
errors = []
# Find all .rels files
rels_files = list(self.unpacked_dir.rglob("*.rels"))
if not rels_files:
if self.verbose:
print("PASSED - No .rels files found")
return True
# Get all files in the unpacked directory (excluding reference files)
all_files = []
for file_path in self.unpacked_dir.rglob("*"):
if (
file_path.is_file()
and file_path.name != "[Content_Types].xml"
and not file_path.name.endswith(".rels")
): # This file is not referenced by .rels
all_files.append(file_path.resolve())
# Track all files that are referenced by any .rels file
all_referenced_files = set()
if self.verbose:
print(
f"Found {len(rels_files)} .rels files and {len(all_files)} target files"
)
# Check each .rels file
for rels_file in rels_files:
try:
# Parse relationships file
rels_root = lxml.etree.parse(str(rels_file)).getroot()
# Get the directory where this .rels file is located
rels_dir = rels_file.parent
# Find all relationships and their targets
referenced_files = set()
broken_refs = []
for rel in rels_root.findall(
".//ns:Relationship",
namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE},
):
target = rel.get("Target")
if target and not target.startswith(
("http", "mailto:")
): # Skip external URLs
# Resolve the target path relative to the .rels file location
if rels_file.name == ".rels":
# Root .rels file - targets are relative to unpacked_dir
target_path = self.unpacked_dir / target
else:
# Other .rels files - targets are relative to their parent's parent
# e.g., word/_rels/document.xml.rels -> targets relative to word/
base_dir = rels_dir.parent
target_path = base_dir / target
# Normalize the path and check if it exists
try:
target_path = target_path.resolve()
if target_path.exists() and target_path.is_file():
referenced_files.add(target_path)
all_referenced_files.add(target_path)
else:
broken_refs.append((target, rel.sourceline))
except (OSError, ValueError):
broken_refs.append((target, rel.sourceline))
# Report broken references
if broken_refs:
rel_path = rels_file.relative_to(self.unpacked_dir)
for broken_ref, line_num in broken_refs:
errors.append(
f" {rel_path}: Line {line_num}: Broken reference to {broken_ref}"
)
except Exception as e:
rel_path = rels_file.relative_to(self.unpacked_dir)
errors.append(f" Error parsing {rel_path}: {e}")
# Check for unreferenced files (files that exist but are not referenced anywhere)
unreferenced_files = set(all_files) - all_referenced_files
if unreferenced_files:
for unref_file in sorted(unreferenced_files):
unref_rel_path = unref_file.relative_to(self.unpacked_dir)
errors.append(f" Unreferenced file: {unref_rel_path}")
if errors:
print(f"FAILED - Found {len(errors)} relationship validation errors:")
for error in errors:
print(error)
print(
"CRITICAL: These errors will cause the document to appear corrupt. "
+ "Broken references MUST be fixed, "
+ "and unreferenced files MUST be referenced or removed."
)
return False
else:
if self.verbose:
print(
"PASSED - All references are valid and all files are properly referenced"
)
return True
def validate_all_relationship_ids(self):
"""
Validate that all r:id attributes in XML files reference existing IDs
in their corresponding .rels files, and optionally validate relationship types.
"""
errors = []
# Process each XML file that might contain r:id references
for xml_file in self.xml_files:
# Skip .rels files themselves
if xml_file.suffix == ".rels":
continue
# Determine the corresponding .rels file
# For dir/file.xml, it's dir/_rels/file.xml.rels
rels_dir = xml_file.parent / "_rels"
rels_file = rels_dir / f"{xml_file.name}.rels"
# Skip if there's no corresponding .rels file (that's okay)
if not rels_file.exists():
continue
try:
# Parse the .rels file to get valid relationship IDs and their types
rels_root = lxml.etree.parse(str(rels_file)).getroot()
rid_to_type = {}
for rel in rels_root.findall(
f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship"
):
rid = rel.get("Id")
rel_type = rel.get("Type", "")
if rid:
# Check for duplicate rIds
if rid in rid_to_type:
rels_rel_path = rels_file.relative_to(self.unpacked_dir)
errors.append(
f" {rels_rel_path}: Line {rel.sourceline}: "
f"Duplicate relationship ID '{rid}' (IDs must be unique)"
)
# Extract just the type name from the full URL
type_name = (
rel_type.split("/")[-1] if "/" in rel_type else rel_type
)
rid_to_type[rid] = type_name
# Parse the XML file to find all r:id references
xml_root = lxml.etree.parse(str(xml_file)).getroot()
# Find all elements with r:id attributes
for elem in xml_root.iter():
# Check for r:id attribute (relationship ID)
rid_attr = elem.get(f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id")
if rid_attr:
xml_rel_path = xml_file.relative_to(self.unpacked_dir)
elem_name = (
elem.tag.split("}")[-1] if "}" in elem.tag else elem.tag
)
# Check if the ID exists
if rid_attr not in rid_to_type:
errors.append(
f" {xml_rel_path}: Line {elem.sourceline}: "
f"<{elem_name}> references non-existent relationship '{rid_attr}' "
f"(valid IDs: {', '.join(sorted(rid_to_type.keys())[:5])}{'...' if len(rid_to_type) > 5 else ''})"
)
# Check if we have type expectations for this element
elif self.ELEMENT_RELATIONSHIP_TYPES:
expected_type = self._get_expected_relationship_type(
elem_name
)
if expected_type:
actual_type = rid_to_type[rid_attr]
# Check if the actual type matches or contains the expected type
if expected_type not in actual_type.lower():
errors.append(
f" {xml_rel_path}: Line {elem.sourceline}: "
f"<{elem_name}> references '{rid_attr}' which points to '{actual_type}' "
f"but should point to a '{expected_type}' relationship"
)
except Exception as e:
xml_rel_path = xml_file.relative_to(self.unpacked_dir)
errors.append(f" Error processing {xml_rel_path}: {e}")
if errors:
print(f"FAILED - Found {len(errors)} relationship ID reference errors:")
for error in errors:
print(error)
print("\nThese ID mismatches will cause the document to appear corrupt!")
return False
else:
if self.verbose:
print("PASSED - All relationship ID references are valid")
return True
def _get_expected_relationship_type(self, element_name):
"""
Get the expected relationship type for an element.
First checks the explicit mapping, then tries pattern detection.
"""
# Normalize element name to lowercase
elem_lower = element_name.lower()
# Check explicit mapping first
if elem_lower in self.ELEMENT_RELATIONSHIP_TYPES:
return self.ELEMENT_RELATIONSHIP_TYPES[elem_lower]
# Try pattern detection for common patterns
# Pattern 1: Elements ending in "Id" often expect a relationship of the prefix type
if elem_lower.endswith("id") and len(elem_lower) > 2:
# e.g., "sldId" -> "sld", "sldMasterId" -> "sldMaster"
prefix = elem_lower[:-2] # Remove "id"
# Check if this might be a compound like "sldMasterId"
if prefix.endswith("master"):
return prefix.lower()
elif prefix.endswith("layout"):
return prefix.lower()
else:
# Simple case like "sldId" -> "slide"
# Common transformations
if prefix == "sld":
return "slide"
return prefix.lower()
# Pattern 2: Elements ending in "Reference" expect a relationship of the prefix type
if elem_lower.endswith("reference") and len(elem_lower) > 9:
prefix = elem_lower[:-9] # Remove "reference"
return prefix.lower()
return None
def validate_content_types(self):
"""Validate that all content files are properly declared in [Content_Types].xml."""
errors = []
# Find [Content_Types].xml file
content_types_file = self.unpacked_dir / "[Content_Types].xml"
if not content_types_file.exists():
print("FAILED - [Content_Types].xml file not found")
return False
try:
# Parse and get all declared parts and extensions
root = lxml.etree.parse(str(content_types_file)).getroot()
declared_parts = set()
declared_extensions = set()
# Get Override declarations (specific files)
for override in root.findall(
f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Override"
):
part_name = override.get("PartName")
if part_name is not None:
declared_parts.add(part_name.lstrip("/"))
# Get Default declarations (by extension)
for default in root.findall(
f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Default"
):
extension = default.get("Extension")
if extension is not None:
declared_extensions.add(extension.lower())
# Root elements that require content type declaration
declarable_roots = {
"sld",
"sldLayout",
"sldMaster",
"presentation", # PowerPoint
"document", # Word
"workbook",
"worksheet", # Excel
"theme", # Common
}
# Common media file extensions that should be declared
media_extensions = {
"png": "image/png",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"gif": "image/gif",
"bmp": "image/bmp",
"tiff": "image/tiff",
"wmf": "image/x-wmf",
"emf": "image/x-emf",
}
# Get all files in the unpacked directory
all_files = list(self.unpacked_dir.rglob("*"))
all_files = [f for f in all_files if f.is_file()]
# Check all XML files for Override declarations
for xml_file in self.xml_files:
path_str = str(xml_file.relative_to(self.unpacked_dir)).replace(
"\\", "/"
)
# Skip non-content files
if any(
skip in path_str
for skip in [".rels", "[Content_Types]", "docProps/", "_rels/"]
):
continue
try:
root_tag = lxml.etree.parse(str(xml_file)).getroot().tag
root_name = root_tag.split("}")[-1] if "}" in root_tag else root_tag
if root_name in declarable_roots and path_str not in declared_parts:
errors.append(
f" {path_str}: File with <{root_name}> root not declared in [Content_Types].xml"
)
except Exception:
continue # Skip unparseable files
# Check all non-XML files for Default extension declarations
for file_path in all_files:
# Skip XML files and metadata files (already checked above)
if file_path.suffix.lower() in {".xml", ".rels"}:
continue
if file_path.name == "[Content_Types].xml":
continue
if "_rels" in file_path.parts or "docProps" in file_path.parts:
continue
extension = file_path.suffix.lstrip(".").lower()
if extension and extension not in declared_extensions:
# Check if it's a known media extension that should be declared
if extension in media_extensions:
relative_path = file_path.relative_to(self.unpacked_dir)
errors.append(
f' {relative_path}: File with extension \'{extension}\' not declared in [Content_Types].xml - should add: <Default Extension="{extension}" ContentType="{media_extensions[extension]}"/>'
)
except Exception as e:
errors.append(f" Error parsing [Content_Types].xml: {e}")
if errors:
print(f"FAILED - Found {len(errors)} content type declaration errors:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print(
"PASSED - All content files are properly declared in [Content_Types].xml"
)
return True
def validate_file_against_xsd(self, xml_file, verbose=False):
"""Validate a single XML file against XSD schema, comparing with original.
Args:
xml_file: Path to XML file to validate
verbose: Enable verbose output
Returns:
tuple: (is_valid, new_errors_set) where is_valid is True/False/None (skipped)
"""
# Resolve both paths to handle symlinks
xml_file = Path(xml_file).resolve()
unpacked_dir = self.unpacked_dir.resolve()
# Validate current file
is_valid, current_errors = self._validate_single_file_xsd(
xml_file, unpacked_dir
)
if is_valid is None:
return None, set() # Skipped
elif is_valid:
return True, set() # Valid, no errors
# Get errors from original file for this specific file
original_errors = self._get_original_file_errors(xml_file)
# Compare with original (both are guaranteed to be sets here)
assert current_errors is not None
new_errors = current_errors - original_errors
if new_errors:
if verbose:
relative_path = xml_file.relative_to(unpacked_dir)
print(f"FAILED - {relative_path}: {len(new_errors)} new error(s)")
for error in list(new_errors)[:3]:
truncated = error[:250] + "..." if len(error) > 250 else error
print(f" - {truncated}")
return False, new_errors
else:
# All errors existed in original
if verbose:
print(
f"PASSED - No new errors (original had {len(current_errors)} errors)"
)
return True, set()
def validate_against_xsd(self):
"""Validate XML files against XSD schemas, showing only new errors compared to original."""
new_errors = []
original_error_count = 0
valid_count = 0
skipped_count = 0
for xml_file in self.xml_files:
relative_path = str(xml_file.relative_to(self.unpacked_dir))
is_valid, new_file_errors = self.validate_file_against_xsd(
xml_file, verbose=False
)
if is_valid is None:
skipped_count += 1
continue
elif is_valid and not new_file_errors:
valid_count += 1
continue
elif is_valid:
# Had errors but all existed in original
original_error_count += 1
valid_count += 1
continue
# Has new errors
new_errors.append(f" {relative_path}: {len(new_file_errors)} new error(s)")
for error in list(new_file_errors)[:3]: # Show first 3 errors
new_errors.append(
f" - {error[:250]}..." if len(error) > 250 else f" - {error}"
)
# Print summary
if self.verbose:
print(f"Validated {len(self.xml_files)} files:")
print(f" - Valid: {valid_count}")
print(f" - Skipped (no schema): {skipped_count}")
if original_error_count:
print(f" - With original errors (ignored): {original_error_count}")
print(
f" - With NEW errors: {len(new_errors) > 0 and len([e for e in new_errors if not e.startswith(' ')]) or 0}"
)
if new_errors:
print("\nFAILED - Found NEW validation errors:")
for error in new_errors:
print(error)
return False
else:
if self.verbose:
print("\nPASSED - No new XSD validation errors introduced")
return True
def _get_schema_path(self, xml_file):
"""Determine the appropriate schema path for an XML file."""
# Check exact filename match
if xml_file.name in self.SCHEMA_MAPPINGS:
return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.name]
# Check .rels files
if xml_file.suffix == ".rels":
return self.schemas_dir / self.SCHEMA_MAPPINGS[".rels"]
# Check chart files
if "charts/" in str(xml_file) and xml_file.name.startswith("chart"):
return self.schemas_dir / self.SCHEMA_MAPPINGS["chart"]
# Check theme files
if "theme/" in str(xml_file) and xml_file.name.startswith("theme"):
return self.schemas_dir / self.SCHEMA_MAPPINGS["theme"]
# Check if file is in a main content folder and use appropriate schema
if xml_file.parent.name in self.MAIN_CONTENT_FOLDERS:
return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.parent.name]
return None
def _clean_ignorable_namespaces(self, xml_doc):
"""Remove attributes and elements not in allowed namespaces."""
# Create a clean copy
xml_string = lxml.etree.tostring(xml_doc, encoding="unicode")
xml_copy = lxml.etree.fromstring(xml_string)
# Remove attributes not in allowed namespaces
for elem in xml_copy.iter():
attrs_to_remove = []
for attr in elem.attrib:
# Check if attribute is from a namespace other than allowed ones
if "{" in attr:
ns = attr.split("}")[0][1:]
if ns not in self.OOXML_NAMESPACES:
attrs_to_remove.append(attr)
# Remove collected attributes
for attr in attrs_to_remove:
del elem.attrib[attr]
# Remove elements not in allowed namespaces
self._remove_ignorable_elements(xml_copy)
return lxml.etree.ElementTree(xml_copy)
def _remove_ignorable_elements(self, root):
"""Recursively remove all elements not in allowed namespaces."""
elements_to_remove = []
# Find elements to remove
for elem in list(root):
# Skip non-element nodes (comments, processing instructions, etc.)
if not hasattr(elem, "tag") or callable(elem.tag):
continue
tag_str = str(elem.tag)
if tag_str.startswith("{"):
ns = tag_str.split("}")[0][1:]
if ns not in self.OOXML_NAMESPACES:
elements_to_remove.append(elem)
continue
# Recursively clean child elements
self._remove_ignorable_elements(elem)
# Remove collected elements
for elem in elements_to_remove:
root.remove(elem)
def _preprocess_for_mc_ignorable(self, xml_doc):
"""Preprocess XML to handle mc:Ignorable attribute properly."""
# Remove mc:Ignorable attributes before validation
root = xml_doc.getroot()
# Remove mc:Ignorable attribute from root
if f"{{{self.MC_NAMESPACE}}}Ignorable" in root.attrib:
del root.attrib[f"{{{self.MC_NAMESPACE}}}Ignorable"]
return xml_doc
def _validate_single_file_xsd(self, xml_file, base_path):
"""Validate a single XML file against XSD schema. Returns (is_valid, errors_set)."""
schema_path = self._get_schema_path(xml_file)
if not schema_path:
return None, None # Skip file
try:
# Load schema
with open(schema_path, "rb") as xsd_file:
parser = lxml.etree.XMLParser()
xsd_doc = lxml.etree.parse(
xsd_file, parser=parser, base_url=str(schema_path)
)
schema = lxml.etree.XMLSchema(xsd_doc)
# Load and preprocess XML
with open(xml_file, "r") as f:
xml_doc = lxml.etree.parse(f)
xml_doc, _ = self._remove_template_tags_from_text_nodes(xml_doc)
xml_doc = self._preprocess_for_mc_ignorable(xml_doc)
# Clean ignorable namespaces if needed
relative_path = xml_file.relative_to(base_path)
if (
relative_path.parts
and relative_path.parts[0] in self.MAIN_CONTENT_FOLDERS
):
xml_doc = self._clean_ignorable_namespaces(xml_doc)
# Validate
if schema.validate(xml_doc):
return True, set()
else:
errors = set()
for error in schema.error_log:
# Store normalized error message (without line numbers for comparison)
errors.add(error.message)
return False, errors
except Exception as e:
return False, {str(e)}
def _get_original_file_errors(self, xml_file):
"""Get XSD validation errors from a single file in the original document.
Args:
xml_file: Path to the XML file in unpacked_dir to check
Returns:
set: Set of error messages from the original file
"""
# Resolve both paths to handle symlinks (e.g., /var vs /private/var on macOS)
xml_file = Path(xml_file).resolve()
unpacked_dir = self.unpacked_dir.resolve()
relative_path = xml_file.relative_to(unpacked_dir)
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Extract original file
with zipfile.ZipFile(self.original_file, "r") as zip_ref:
zip_ref.extractall(temp_path)
# Find corresponding file in original
original_xml_file = temp_path / relative_path
if not original_xml_file.exists():
# File didn't exist in original, so no original errors
return set()
# Validate the specific file in original
is_valid, errors = self._validate_single_file_xsd(
original_xml_file, temp_path
)
return errors if errors else set()
def _remove_template_tags_from_text_nodes(self, xml_doc):
"""Remove template tags from XML text nodes and collect warnings.
Template tags follow the pattern {{ ... }} and are used as placeholders
for content replacement. They should be removed from text content before
XSD validation while preserving XML structure.
Returns:
tuple: (cleaned_xml_doc, warnings_list)
"""
warnings = []
template_pattern = re.compile(r"\{\{[^}]*\}\}")
# Create a copy of the document to avoid modifying the original
xml_string = lxml.etree.tostring(xml_doc, encoding="unicode")
xml_copy = lxml.etree.fromstring(xml_string)
def process_text_content(text, content_type):
if not text:
return text
matches = list(template_pattern.finditer(text))
if matches:
for match in matches:
warnings.append(
f"Found template tag in {content_type}: {match.group()}"
)
return template_pattern.sub("", text)
return text
# Process all text nodes in the document
for elem in xml_copy.iter():
# Skip processing if this is a w:t element
if not hasattr(elem, "tag") or callable(elem.tag):
continue
tag_str = str(elem.tag)
if tag_str.endswith("}t") or tag_str == "t":
continue
elem.text = process_text_content(elem.text, "text content")
elem.tail = process_text_content(elem.tail, "tail content")
return lxml.etree.ElementTree(xml_copy), warnings
if __name__ == "__main__":
raise RuntimeError("This module should not be run directly.")
+274
View File
@@ -0,0 +1,274 @@
"""
Validator for Word document XML files against XSD schemas.
"""
import re
import tempfile
import zipfile
import lxml.etree
from .base import BaseSchemaValidator
class DOCXSchemaValidator(BaseSchemaValidator):
"""Validator for Word document XML files against XSD schemas."""
# Word-specific namespace
WORD_2006_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
# Word-specific element to relationship type mappings
# Start with empty mapping - add specific cases as we discover them
ELEMENT_RELATIONSHIP_TYPES = {}
def validate(self):
"""Run all validation checks and return True if all pass."""
# Test 0: XML well-formedness
if not self.validate_xml():
return False
# Test 1: Namespace declarations
all_valid = True
if not self.validate_namespaces():
all_valid = False
# Test 2: Unique IDs
if not self.validate_unique_ids():
all_valid = False
# Test 3: Relationship and file reference validation
if not self.validate_file_references():
all_valid = False
# Test 4: Content type declarations
if not self.validate_content_types():
all_valid = False
# Test 5: XSD schema validation
if not self.validate_against_xsd():
all_valid = False
# Test 6: Whitespace preservation
if not self.validate_whitespace_preservation():
all_valid = False
# Test 7: Deletion validation
if not self.validate_deletions():
all_valid = False
# Test 8: Insertion validation
if not self.validate_insertions():
all_valid = False
# Test 9: Relationship ID reference validation
if not self.validate_all_relationship_ids():
all_valid = False
# Count and compare paragraphs
self.compare_paragraph_counts()
return all_valid
def validate_whitespace_preservation(self):
"""
Validate that w:t elements with whitespace have xml:space='preserve'.
"""
errors = []
for xml_file in self.xml_files:
# Only check document.xml files
if xml_file.name != "document.xml":
continue
try:
root = lxml.etree.parse(str(xml_file)).getroot()
# Find all w:t elements
for elem in root.iter(f"{{{self.WORD_2006_NAMESPACE}}}t"):
if elem.text:
text = elem.text
# Check if text starts or ends with whitespace
if re.match(r"^\s.*", text) or re.match(r".*\s$", text):
# Check if xml:space="preserve" attribute exists
xml_space_attr = f"{{{self.XML_NAMESPACE}}}space"
if (
xml_space_attr not in elem.attrib
or elem.attrib[xml_space_attr] != "preserve"
):
# Show a preview of the text
text_preview = (
repr(text)[:50] + "..."
if len(repr(text)) > 50
else repr(text)
)
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {elem.sourceline}: w:t element with whitespace missing xml:space='preserve': {text_preview}"
)
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print(f"FAILED - Found {len(errors)} whitespace preservation violations:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - All whitespace is properly preserved")
return True
def validate_deletions(self):
"""
Validate that w:t elements are not within w:del elements.
For some reason, XSD validation does not catch this, so we do it manually.
"""
errors = []
for xml_file in self.xml_files:
# Only check document.xml files
if xml_file.name != "document.xml":
continue
try:
root = lxml.etree.parse(str(xml_file)).getroot()
# Find all w:t elements that are descendants of w:del elements
namespaces = {"w": self.WORD_2006_NAMESPACE}
xpath_expression = ".//w:del//w:t"
problematic_t_elements = root.xpath(
xpath_expression, namespaces=namespaces
)
for t_elem in problematic_t_elements:
if t_elem.text:
# Show a preview of the text
text_preview = (
repr(t_elem.text)[:50] + "..."
if len(repr(t_elem.text)) > 50
else repr(t_elem.text)
)
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {t_elem.sourceline}: <w:t> found within <w:del>: {text_preview}"
)
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print(f"FAILED - Found {len(errors)} deletion validation violations:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - No w:t elements found within w:del elements")
return True
def count_paragraphs_in_unpacked(self):
"""Count the number of paragraphs in the unpacked document."""
count = 0
for xml_file in self.xml_files:
# Only check document.xml files
if xml_file.name != "document.xml":
continue
try:
root = lxml.etree.parse(str(xml_file)).getroot()
# Count all w:p elements
paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p")
count = len(paragraphs)
except Exception as e:
print(f"Error counting paragraphs in unpacked document: {e}")
return count
def count_paragraphs_in_original(self):
"""Count the number of paragraphs in the original docx file."""
count = 0
try:
# Create temporary directory to unpack original
with tempfile.TemporaryDirectory() as temp_dir:
# Unpack original docx
with zipfile.ZipFile(self.original_file, "r") as zip_ref:
zip_ref.extractall(temp_dir)
# Parse document.xml
doc_xml_path = temp_dir + "/word/document.xml"
root = lxml.etree.parse(doc_xml_path).getroot()
# Count all w:p elements
paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p")
count = len(paragraphs)
except Exception as e:
print(f"Error counting paragraphs in original document: {e}")
return count
def validate_insertions(self):
"""
Validate that w:delText elements are not within w:ins elements.
w:delText is only allowed in w:ins if nested within a w:del.
"""
errors = []
for xml_file in self.xml_files:
if xml_file.name != "document.xml":
continue
try:
root = lxml.etree.parse(str(xml_file)).getroot()
namespaces = {"w": self.WORD_2006_NAMESPACE}
# Find w:delText in w:ins that are NOT within w:del
invalid_elements = root.xpath(
".//w:ins//w:delText[not(ancestor::w:del)]",
namespaces=namespaces
)
for elem in invalid_elements:
text_preview = (
repr(elem.text or "")[:50] + "..."
if len(repr(elem.text or "")) > 50
else repr(elem.text or "")
)
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {elem.sourceline}: <w:delText> within <w:ins>: {text_preview}"
)
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print(f"FAILED - Found {len(errors)} insertion validation violations:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - No w:delText elements within w:ins elements")
return True
def compare_paragraph_counts(self):
"""Compare paragraph counts between original and new document."""
original_count = self.count_paragraphs_in_original()
new_count = self.count_paragraphs_in_unpacked()
diff = new_count - original_count
diff_str = f"+{diff}" if diff > 0 else str(diff)
print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})")
if __name__ == "__main__":
raise RuntimeError("This module should not be run directly.")
+309
View File
@@ -0,0 +1,309 @@
"""
Validator for PowerPoint presentation XML files against XSD schemas.
"""
import re
import lxml.etree
from .base import BaseSchemaValidator
class PPTXSchemaValidator(BaseSchemaValidator):
"""Validator for PowerPoint presentation XML files against XSD schemas."""
# PowerPoint presentation namespace
PRESENTATIONML_NAMESPACE = (
"http://schemas.openxmlformats.org/presentationml/2006/main"
)
# PowerPoint-specific element to relationship type mappings
ELEMENT_RELATIONSHIP_TYPES = {
"sldid": "slide",
"sldmasterid": "slidemaster",
"notesmasterid": "notesmaster",
"sldlayoutid": "slidelayout",
"themeid": "theme",
"tablestyleid": "tablestyles",
}
def validate(self):
"""Run all validation checks and return True if all pass."""
# Test 0: XML well-formedness
if not self.validate_xml():
return False
# Test 1: Namespace declarations
all_valid = True
if not self.validate_namespaces():
all_valid = False
# Test 2: Unique IDs
if not self.validate_unique_ids():
all_valid = False
# Test 3: UUID ID validation
if not self.validate_uuid_ids():
all_valid = False
# Test 4: Relationship and file reference validation
if not self.validate_file_references():
all_valid = False
# Test 5: Slide layout ID validation
if not self.validate_slide_layout_ids():
all_valid = False
# Test 6: Content type declarations
if not self.validate_content_types():
all_valid = False
# Test 7: XSD schema validation
if not self.validate_against_xsd():
all_valid = False
# Test 8: Notes slide reference validation
if not self.validate_notes_slide_references():
all_valid = False
# Test 9: Relationship ID reference validation
if not self.validate_all_relationship_ids():
all_valid = False
# Test 10: Duplicate slide layout references validation
if not self.validate_no_duplicate_slide_layouts():
all_valid = False
return all_valid
def validate_uuid_ids(self):
"""Validate that ID attributes that look like UUIDs contain only hex values."""
errors = []
# UUID pattern: 8-4-4-4-12 hex digits with optional braces/hyphens
uuid_pattern = re.compile(
r"^[\{\(]?[0-9A-Fa-f]{8}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{12}[\}\)]?$"
)
for xml_file in self.xml_files:
try:
root = lxml.etree.parse(str(xml_file)).getroot()
# Check all elements for ID attributes
for elem in root.iter():
for attr, value in elem.attrib.items():
# Check if this is an ID attribute
attr_name = attr.split("}")[-1].lower()
if attr_name == "id" or attr_name.endswith("id"):
# Check if value looks like a UUID (has the right length and pattern structure)
if self._looks_like_uuid(value):
# Validate that it contains only hex characters in the right positions
if not uuid_pattern.match(value):
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: "
f"Line {elem.sourceline}: ID '{value}' appears to be a UUID but contains invalid hex characters"
)
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print(f"FAILED - Found {len(errors)} UUID ID validation errors:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - All UUID-like IDs contain valid hex values")
return True
def _looks_like_uuid(self, value):
"""Check if a value has the general structure of a UUID."""
# Remove common UUID delimiters
clean_value = value.strip("{}()").replace("-", "")
# Check if it's 32 hex-like characters (could include invalid hex chars)
return len(clean_value) == 32 and all(c.isalnum() for c in clean_value)
def validate_slide_layout_ids(self):
"""Validate that sldLayoutId elements in slide masters reference valid slide layouts."""
errors = []
# Find all slide master files
slide_masters = list(self.unpacked_dir.glob("ppt/slideMasters/*.xml"))
if not slide_masters:
if self.verbose:
print("PASSED - No slide masters found")
return True
for slide_master in slide_masters:
try:
# Parse the slide master file
root = lxml.etree.parse(str(slide_master)).getroot()
# Find the corresponding _rels file for this slide master
rels_file = slide_master.parent / "_rels" / f"{slide_master.name}.rels"
if not rels_file.exists():
errors.append(
f" {slide_master.relative_to(self.unpacked_dir)}: "
f"Missing relationships file: {rels_file.relative_to(self.unpacked_dir)}"
)
continue
# Parse the relationships file
rels_root = lxml.etree.parse(str(rels_file)).getroot()
# Build a set of valid relationship IDs that point to slide layouts
valid_layout_rids = set()
for rel in rels_root.findall(
f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship"
):
rel_type = rel.get("Type", "")
if "slideLayout" in rel_type:
valid_layout_rids.add(rel.get("Id"))
# Find all sldLayoutId elements in the slide master
for sld_layout_id in root.findall(
f".//{{{self.PRESENTATIONML_NAMESPACE}}}sldLayoutId"
):
r_id = sld_layout_id.get(
f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id"
)
layout_id = sld_layout_id.get("id")
if r_id and r_id not in valid_layout_rids:
errors.append(
f" {slide_master.relative_to(self.unpacked_dir)}: "
f"Line {sld_layout_id.sourceline}: sldLayoutId with id='{layout_id}' "
f"references r:id='{r_id}' which is not found in slide layout relationships"
)
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {slide_master.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print(f"FAILED - Found {len(errors)} slide layout ID validation errors:")
for error in errors:
print(error)
print(
"Remove invalid references or add missing slide layouts to the relationships file."
)
return False
else:
if self.verbose:
print("PASSED - All slide layout IDs reference valid slide layouts")
return True
def validate_no_duplicate_slide_layouts(self):
"""Validate that each slide has exactly one slideLayout reference."""
errors = []
slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels"))
for rels_file in slide_rels_files:
try:
root = lxml.etree.parse(str(rels_file)).getroot()
# Find all slideLayout relationships
layout_rels = [
rel
for rel in root.findall(
f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship"
)
if "slideLayout" in rel.get("Type", "")
]
if len(layout_rels) > 1:
errors.append(
f" {rels_file.relative_to(self.unpacked_dir)}: has {len(layout_rels)} slideLayout references"
)
except Exception as e:
errors.append(
f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
if errors:
print("FAILED - Found slides with duplicate slideLayout references:")
for error in errors:
print(error)
return False
else:
if self.verbose:
print("PASSED - All slides have exactly one slideLayout reference")
return True
def validate_notes_slide_references(self):
"""Validate that each notesSlide file is referenced by only one slide."""
errors = []
notes_slide_references = {} # Track which slides reference each notesSlide
# Find all slide relationship files
slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels"))
if not slide_rels_files:
if self.verbose:
print("PASSED - No slide relationship files found")
return True
for rels_file in slide_rels_files:
try:
# Parse the relationships file
root = lxml.etree.parse(str(rels_file)).getroot()
# Find all notesSlide relationships
for rel in root.findall(
f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship"
):
rel_type = rel.get("Type", "")
if "notesSlide" in rel_type:
target = rel.get("Target", "")
if target:
# Normalize the target path to handle relative paths
normalized_target = target.replace("../", "")
# Track which slide references this notesSlide
slide_name = rels_file.stem.replace(
".xml", ""
) # e.g., "slide1"
if normalized_target not in notes_slide_references:
notes_slide_references[normalized_target] = []
notes_slide_references[normalized_target].append(
(slide_name, rels_file)
)
except (lxml.etree.XMLSyntaxError, Exception) as e:
errors.append(
f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}"
)
# Check for duplicate references
for target, references in notes_slide_references.items():
if len(references) > 1:
slide_names = [ref[0] for ref in references]
errors.append(
f" Notes slide '{target}' is referenced by multiple slides: {', '.join(slide_names)}"
)
for slide_name, rels_file in references:
errors.append(f" - {rels_file.relative_to(self.unpacked_dir)}")
if errors:
print(
f"FAILED - Found {len([e for e in errors if not e.startswith(' ')])} notes slide reference validation errors:"
)
for error in errors:
print(error)
print("Each slide may optionally have its own slide file.")
return False
else:
if self.verbose:
print("PASSED - All notes slide references are unique")
return True
if __name__ == "__main__":
raise RuntimeError("This module should not be run directly.")
+279
View File
@@ -0,0 +1,279 @@
"""
Validator for tracked changes in Word documents.
"""
import subprocess
import tempfile
import zipfile
from pathlib import Path
class RedliningValidator:
"""Validator for tracked changes in Word documents."""
def __init__(self, unpacked_dir, original_docx, verbose=False):
self.unpacked_dir = Path(unpacked_dir)
self.original_docx = Path(original_docx)
self.verbose = verbose
self.namespaces = {
"w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
}
def validate(self):
"""Main validation method that returns True if valid, False otherwise."""
# Verify unpacked directory exists and has correct structure
modified_file = self.unpacked_dir / "word" / "document.xml"
if not modified_file.exists():
print(f"FAILED - Modified document.xml not found at {modified_file}")
return False
# First, check if there are any tracked changes by Z.AI to validate
try:
import xml.etree.ElementTree as ET
tree = ET.parse(modified_file)
root = tree.getroot()
# Check for w:del or w:ins tags authored by Z.AI
del_elements = root.findall(".//w:del", self.namespaces)
ins_elements = root.findall(".//w:ins", self.namespaces)
# Filter to only include changes by Z.AI
zai_del_elements = [
elem
for elem in del_elements
if elem.get(f"{{{self.namespaces['w']}}}author") == "Z.AI"
]
zai_ins_elements = [
elem
for elem in ins_elements
if elem.get(f"{{{self.namespaces['w']}}}author") == "Z.AI"
]
# Redlining validation is only needed if tracked changes by Z.AI have been used.
if not zai_del_elements and not zai_ins_elements:
if self.verbose:
print("PASSED - No tracked changes by Z.AI found.")
return True
except Exception:
# If we can't parse the XML, continue with full validation
pass
# Create temporary directory for unpacking original docx
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Unpack original docx
try:
with zipfile.ZipFile(self.original_docx, "r") as zip_ref:
zip_ref.extractall(temp_path)
except Exception as e:
print(f"FAILED - Error unpacking original docx: {e}")
return False
original_file = temp_path / "word" / "document.xml"
if not original_file.exists():
print(
f"FAILED - Original document.xml not found in {self.original_docx}"
)
return False
# Parse both XML files using xml.etree.ElementTree for redlining validation
try:
import xml.etree.ElementTree as ET
modified_tree = ET.parse(modified_file)
modified_root = modified_tree.getroot()
original_tree = ET.parse(original_file)
original_root = original_tree.getroot()
except ET.ParseError as e:
print(f"FAILED - Error parsing XML files: {e}")
return False
# Remove Z.AI's tracked changes from both documents
self._remove_zai_tracked_changes(original_root)
self._remove_zai_tracked_changes(modified_root)
# Extract and compare text content
modified_text = self._extract_text_content(modified_root)
original_text = self._extract_text_content(original_root)
if modified_text != original_text:
# Show detailed character-level differences for each paragraph
error_message = self._generate_detailed_diff(
original_text, modified_text
)
print(error_message)
return False
if self.verbose:
print("PASSED - All changes by Z.AI are properly tracked")
return True
def _generate_detailed_diff(self, original_text, modified_text):
"""Generate detailed word-level differences using git word diff."""
error_parts = [
"FAILED - Document text doesn't match after removing Z.AI's tracked changes",
"",
"Likely causes:",
" 1. Modified text inside another author's <w:ins> or <w:del> tags",
" 2. Made edits without proper tracked changes",
" 3. Didn't nest <w:del> inside <w:ins> when deleting another's insertion",
"",
"For pre-redlined documents, use correct patterns:",
" - To reject another's INSERTION: Nest <w:del> inside their <w:ins>",
" - To restore another's DELETION: Add new <w:ins> AFTER their <w:del>",
"",
]
# Show git word diff
git_diff = self._get_git_word_diff(original_text, modified_text)
if git_diff:
error_parts.extend(["Differences:", "============", git_diff])
else:
error_parts.append("Unable to generate word diff (git not available)")
return "\n".join(error_parts)
def _get_git_word_diff(self, original_text, modified_text):
"""Generate word diff using git with character-level precision."""
try:
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create two files
original_file = temp_path / "original.txt"
modified_file = temp_path / "modified.txt"
original_file.write_text(original_text, encoding="utf-8")
modified_file.write_text(modified_text, encoding="utf-8")
# Try character-level diff first for precise differences
result = subprocess.run(
[
"git",
"diff",
"--word-diff=plain",
"--word-diff-regex=.", # Character-by-character diff
"-U0", # Zero lines of context - show only changed lines
"--no-index",
str(original_file),
str(modified_file),
],
capture_output=True,
text=True,
)
if result.stdout.strip():
# Clean up the output - remove git diff header lines
lines = result.stdout.split("\n")
# Skip the header lines (diff --git, index, +++, ---, @@)
content_lines = []
in_content = False
for line in lines:
if line.startswith("@@"):
in_content = True
continue
if in_content and line.strip():
content_lines.append(line)
if content_lines:
return "\n".join(content_lines)
# Fallback to word-level diff if character-level is too verbose
result = subprocess.run(
[
"git",
"diff",
"--word-diff=plain",
"-U0", # Zero lines of context
"--no-index",
str(original_file),
str(modified_file),
],
capture_output=True,
text=True,
)
if result.stdout.strip():
lines = result.stdout.split("\n")
content_lines = []
in_content = False
for line in lines:
if line.startswith("@@"):
in_content = True
continue
if in_content and line.strip():
content_lines.append(line)
return "\n".join(content_lines)
except (subprocess.CalledProcessError, FileNotFoundError, Exception):
# Git not available or other error, return None to use fallback
pass
return None
def _remove_zai_tracked_changes(self, root):
"""Remove tracked changes authored by Z.AI from the XML root."""
ins_tag = f"{{{self.namespaces['w']}}}ins"
del_tag = f"{{{self.namespaces['w']}}}del"
author_attr = f"{{{self.namespaces['w']}}}author"
# Remove w:ins elements
for parent in root.iter():
to_remove = []
for child in parent:
if child.tag == ins_tag and child.get(author_attr) == "Z.AI":
to_remove.append(child)
for elem in to_remove:
parent.remove(elem)
# Unwrap content in w:del elements where author is "Z.AI"
deltext_tag = f"{{{self.namespaces['w']}}}delText"
t_tag = f"{{{self.namespaces['w']}}}t"
for parent in root.iter():
to_process = []
for child in parent:
if child.tag == del_tag and child.get(author_attr) == "Z.AI":
to_process.append((child, list(parent).index(child)))
# Process in reverse order to maintain indices
for del_elem, del_index in reversed(to_process):
# Convert w:delText to w:t before moving
for elem in del_elem.iter():
if elem.tag == deltext_tag:
elem.tag = t_tag
# Move all children of w:del to its parent before removing w:del
for child in reversed(list(del_elem)):
parent.insert(del_index, child)
parent.remove(del_elem)
def _extract_text_content(self, root):
"""Extract text content from Word XML, preserving paragraph structure.
Empty paragraphs are skipped to avoid false positives when tracked
insertions add only structural elements without text content.
"""
p_tag = f"{{{self.namespaces['w']}}}p"
t_tag = f"{{{self.namespaces['w']}}}t"
paragraphs = []
for p_elem in root.findall(f".//{p_tag}"):
# Get all text elements within this paragraph
text_parts = []
for t_elem in p_elem.findall(f".//{t_tag}"):
if t_elem.text:
text_parts.append(t_elem.text)
paragraph_text = "".join(text_parts)
# Skip empty paragraphs - they don't affect content validation
if paragraph_text:
paragraphs.append(paragraph_text)
return "\n".join(paragraphs)
if __name__ == "__main__":
raise RuntimeError("This module should not be run directly.")
+712
View File
@@ -0,0 +1,712 @@
#!/usr/bin/env python3
"""CLI for slide dimension / overflow / blank-region / overlap checks.
Renders each HTML slide in headless Chromium (via Playwright) at 1280x720 and
reports vertical overflow, large blank regions, and text-vs-text overlaps.
Standalone — does not import from any sibling agent package.
Usage:
# Check specific files (preferred — matches whatever naming the brief uses):
python check_slides.py /work/slides/slide_01.html /work/slides/slide_02.html
# Check every *.html in a directory (sorted by filename):
python check_slides.py --dir /work/slides
# Restrict to a few positions when --dir is used (matches files whose stem
# ends in those numbers, e.g. `1.html`, `slide_01.html`):
python check_slides.py --dir /work/slides --positions 1 3 5
# JSON output for downstream tooling:
python check_slides.py --dir /work/slides --json --output /work/check.json
Stdout is a compact text report by default; pass --json for the raw measurement
dict per file. Exit code is 0 when every checked slide fits the viewport
vertically with no large blank / text overlap, otherwise 1.
"""
from __future__ import annotations
import argparse
import asyncio
import json
import re
import sys
import traceback
from pathlib import Path
_VIEWPORT_W = 1280
_VIEWPORT_H = 720
_MAX_CONCURRENCY = 81
_PROXY = {"server": "http://httpproxy.glm.ai:8888"}
_TEXT_OVERLAP_JS = r"""
(opts) => {
const minOverlapPx = opts.minOverlapPx || 3;
const maxPairs = opts.maxPairs || 50;
// Relative overlap threshold: the intersection must consume at least
// `relMin` of the smaller element's corresponding side on BOTH axes.
// Suppresses line-height bleed where vertically-stacked siblings share
// 3-5px of line-box overhang without true visual collision.
const relMin = opts.relMin != null ? opts.relMin : 0.10;
// Decorative-text thresholds (mirrors genspark html2pptx checkTextOverlaps):
// - opacity ≤ 0.2 → watermark / ghost layer
// - font-size ≥ 150px → giant ghost numeral / chapter mark
const decorOpacity = opts.decorOpacity != null ? opts.decorOpacity : 0.2;
const decorFontPx = opts.decorFontPx || 150;
function ownText(el) {
let t = '';
for (const n of el.childNodes) {
if (n.nodeType === Node.TEXT_NODE) t += n.textContent;
}
return t.trim();
}
// Effective opacity = product of opacity along ancestor chain (CSS-correct).
// Returns 0 if any ancestor is display:none / visibility:hidden.
function effectiveOpacity(el) {
let op = 1;
let cur = el;
while (cur && cur !== document.body) {
const s = getComputedStyle(cur);
if (s.display === 'none' || s.visibility === 'hidden') return 0;
const o = parseFloat(s.opacity);
if (!isNaN(o)) op *= o;
cur = cur.parentElement;
}
return op;
}
// First ancestor (incl. self) that establishes a stacking context with a
// numeric z-index. Returns 0 if none — matches "auto" → 0 for sibling
// comparison purposes. We don't try to fully simulate the stacking algorithm;
// this is a heuristic to skip *intentional* layered decorations.
function effectiveZ(el) {
let cur = el;
while (cur && cur !== document.body) {
const s = getComputedStyle(cur);
if (s.position && s.position !== 'static') {
const z = parseInt(s.zIndex, 10);
if (!isNaN(z)) return z;
}
cur = cur.parentElement;
}
return 0;
}
// Restrict reporting to text inside the slide canvas — body padding / preview
// gaps / off-canvas overflow shouldn't surface as overlap noise.
const canvas = document.querySelector('.slide, .slide-canvas');
const canvasRect = canvas ? canvas.getBoundingClientRect() : null;
function insideCanvas(r) {
if (!canvasRect) return true;
// Require ≥50% of the rect's area to lie inside the canvas.
const x = Math.max(r.x, canvasRect.left);
const y = Math.max(r.y, canvasRect.top);
const w = Math.min(r.x + r.w, canvasRect.right) - x;
const h = Math.min(r.y + r.h, canvasRect.bottom) - y;
if (w <= 0 || h <= 0) return false;
return (w * h) >= 0.5 * (r.w * r.h);
}
const items = [];
const walker = document.querySelectorAll('body *');
for (const el of walker) {
const text = ownText(el);
if (!text) continue;
const op = effectiveOpacity(el);
if (op === 0) continue;
const cs = getComputedStyle(el);
const fontPx = parseFloat(cs.fontSize) || 0;
const decorative = op <= decorOpacity || fontPx >= decorFontPx;
const z = effectiveZ(el);
const rects = el.getClientRects();
if (!rects.length) continue;
for (const r of rects) {
if (r.width < 2 || r.height < 2) continue;
const rect = {x: r.x, y: r.y, w: r.width, h: r.height};
if (!insideCanvas(rect)) continue;
items.push({
el, text, decorative, z, fontPx,
tag: el.tagName.toLowerCase(),
rect,
});
}
}
function rel(a, b) { return a.contains(b) || b.contains(a); }
function inter(r1, r2) {
const x = Math.max(r1.x, r2.x);
const y = Math.max(r1.y, r2.y);
const w = Math.min(r1.x + r1.w, r2.x + r2.w) - x;
const h = Math.min(r1.y + r1.h, r2.y + r2.h) - y;
if (w < minOverlapPx || h < minOverlapPx) return null;
// Relative gate: overlap must occupy ≥ relMin of the smaller side.
if (w < relMin * Math.min(r1.w, r2.w)) return null;
if (h < relMin * Math.min(r1.h, r2.h)) return null;
return {x, y, w, h};
}
function trunc(s, n) { return s.length > n ? s.slice(0, n) + '…' : s; }
function fmtRect(r) {
return {
x: Math.round(r.x), y: Math.round(r.y),
w: Math.round(r.w), h: Math.round(r.h),
};
}
// Dedupe by element pair: a 4-line <p> overlapping another 4-line <p> via
// getClientRects() would otherwise produce up to 16 entries. Keep the
// largest overlap rect per (elA, elB) pair.
const seen = new Map(); // key → {a, b, overlap}
for (let i = 0; i < items.length; i++) {
for (let j = i + 1; j < items.length; j++) {
const A = items[i], B = items[j];
if (A.el === B.el) continue;
if (rel(A.el, B.el)) continue;
// Skip intentional layered decorations (genspark parity).
if (A.decorative || B.decorative) continue;
if (A.z !== B.z) continue;
const ov = inter(A.rect, B.rect);
if (!ov) continue;
// Stable pair key: use DOM position to order.
const order = A.el.compareDocumentPosition(B.el) & Node.DOCUMENT_POSITION_FOLLOWING;
const first = order ? A : B;
const second = order ? B : A;
const key = first.tag + ' ' + first.text + ' '
+ second.tag + ' ' + second.text;
const area = ov.w * ov.h;
const prev = seen.get(key);
if (!prev || area > prev._area) {
seen.set(key, {
a: {tag: first.tag, text: trunc(first.text, 80), rect: fmtRect(first.rect)},
b: {tag: second.tag, text: trunc(second.text, 80), rect: fmtRect(second.rect)},
overlap: fmtRect(ov),
_area: area,
});
}
}
}
const overlaps = [];
for (const v of seen.values()) {
delete v._area;
overlaps.push(v);
if (overlaps.length >= maxPairs) break;
}
return overlaps;
}
"""
def _detect_largest_blank(screenshot_bytes: bytes,
color_tolerance: int = 15,
downsample: int = 4,
decor_rects: list[dict] | None = None) -> dict | None:
"""Find the largest contiguous background-colored rectangle in a screenshot.
Background color is sampled from the four corners (median). A pixel is
"non-background" if its max channel diff from the sample exceeds
`color_tolerance`. The image is downsampled by `downsample`× for the
rectangle search (1280x720 → 320x180 by default), then results are scaled
back to original pixels. Returns None on failure (e.g. PIL/numpy missing).
`decor_rects` (list of {x, y, w, h} in screenshot pixels) marks regions
occupied by decorative elements — those rects are forced to "non-
background" in the mask so faint / ghost decoration that blends with the
bg color does not register as a false blank patch.
"""
try:
import io
import numpy as np
from PIL import Image
except Exception:
return None
try:
img = Image.open(io.BytesIO(screenshot_bytes)).convert("RGB")
arr = np.array(img, dtype=np.uint8)
H, W, _ = arr.shape
if H < 8 or W < 8:
return None
corner = max(4, min(W, H) // 64)
samples = np.concatenate([
arr[:corner, :corner].reshape(-1, 3),
arr[:corner, -corner:].reshape(-1, 3),
arr[-corner:, :corner].reshape(-1, 3),
arr[-corner:, -corner:].reshape(-1, 3),
], axis=0).astype(np.int32)
bg = np.median(samples, axis=0)
diff = np.abs(arr.astype(np.int32) - bg).max(axis=2)
nonbg = (diff > color_tolerance).astype(np.uint8)
# Mask out decorative element rects: their visual area is intentional
# design, not blank. Without this, a low-opacity ghost numeral or
# gradient blob whose pixels blend into bg would show up as one big
# contiguous blank region.
if decor_rects:
for r in decor_rects:
x0 = max(0, int(round(r.get("x", 0))))
y0 = max(0, int(round(r.get("y", 0))))
x1 = min(W, int(round(r.get("x", 0) + r.get("w", 0))))
y1 = min(H, int(round(r.get("y", 0) + r.get("h", 0))))
if x1 > x0 and y1 > y0:
nonbg[y0:y1, x0:x1] = 1
s = max(1, downsample)
Hd, Wd = H // s, W // s
if Hd < 2 or Wd < 2:
return None
# Pool: cell is "blocked" if any underlying pixel is non-background.
nonbg_d = nonbg[:Hd * s, :Wd * s].reshape(Hd, s, Wd, s).max(axis=(1, 3))
heights = [0] * Wd
best_area = 0
best = (0, 0, 0, 0) # x, y, w, h in downsampled units
for r in range(Hd):
row = nonbg_d[r]
for c in range(Wd):
heights[c] = 0 if row[c] else heights[c] + 1
stack = [] # (start_index, height)
for c in range(Wd + 1):
cur = heights[c] if c < Wd else 0
start = c
while stack and stack[-1][1] > cur:
idx, h = stack.pop()
w = c - idx
area = w * h
if area > best_area:
best_area = area
best = (idx, r - h + 1, w, h)
start = idx
stack.append((start, cur))
x_d, y_d, w_d, h_d = best
x, y, w, h = x_d * s, y_d * s, w_d * s, h_d * s
return {
"x": int(x), "y": int(y), "w": int(w), "h": int(h),
"area_ratio": round((w * h) / float(W * H), 3),
"bg_rgb": [int(c) for c in bg],
}
except Exception:
return None
async def measure_slide_dimensions(html_content, viewport_width=1280, viewport_height=720, proxy=None,
detect_overlaps=True, detect_blank=True,
blank_min_area_ratio=0.25, blank_min_side=180,
overflow_tolerance_ratio=0.005, overflow_tolerance_px=3,
file_path: str | None = None):
"""Render HTML at the given viewport and report ACTUAL content width + height.
Returns a dict with success bool, plus on success: width, height,
viewport_width, viewport_height, overflow_x (px past viewport, 0 if fits),
overflow_y (px past viewport, 0 if fits), `details` subdict with raw
browser measurements, `overlaps` (when detect_overlaps): a list of
text-vs-text overlapping pairs each shaped {a: {tag, text, rect}, b:
{tag, text, rect}, overlap: {x,y,w,h}}, and `blank` (when detect_blank):
{x, y, w, h, area_ratio, bg_rgb, flagged: bool} for the largest contiguous
background-colored region. `flagged` is True only when area_ratio ≥
blank_min_area_ratio AND min(w, h) ≥ blank_min_side.
Notes:
- Width and height are both measured (not assumed = viewport_*) and
reported. Horizontal AND vertical overflow are flagged when content
extends past the viewport beyond the tolerance buffer.
- If the slide root uses `overflow:hidden` the inner content is clipped
for the user — we measure content-bearing children to surface the
underlying content size when possible. Elements clipped by any
ancestor with `overflow:hidden|clip` have their rect intersected
with the clipping box before contributing to overflow, so bleed
decoration that's invisible to the user is not flagged.
- Decorative elements are excluded from overflow measurement when
ANY of these hold: (a) the element (or an ancestor) carries
`data-decor` or `aria-hidden="true"`, (b) effective opacity along
the ancestor chain is ≤ 0.2 (ghost / watermark layer). Authors
should tag intentional background blobs, oversized ghost numerals,
and bleed shapes with `data-decor` to silence false-positive
overflow flags.
- Overlap detection only flags pairs of visible text-bearing elements
that are NOT in an ancestor/descendant relationship and whose
bounding rects intersect by ≥3px on both axes (suppresses border
touches) AND by ≥20% of the smaller element's corresponding side
(suppresses line-height bleed in vertically-stacked siblings).
Capped at 50 pairs per slide. Filters that mirror the
genspark html2pptx checker:
* effective opacity ≤ 0.2 → decorative watermark, skipped
* font-size ≥ 150px → ghost numeral / chapter mark, skipped
* different effective z-index → intentional layering, skipped
* rect mostly outside `.slide` canvas (preview chrome / cropped
overflow) → skipped
Multi-line text pairs are deduped to one entry per element pair
(largest overlap rect wins).
- Blank detection samples background color from the four corners and
finds the largest rectangle whose pixels are all within
`color_tolerance` (15) of that color. Operates on a 4×-downsampled
screenshot for speed. Decorative element rects (same opt-out as
the overflow check: `data-decor` / `aria-hidden="true"`, plus
elements whose effective opacity is ≤ 0.2) are forced to "non-
background" in the mask, so faint ghost decoration that blends
with the bg color is not flagged as a large blank region.
"""
from playwright.async_api import async_playwright
try:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True, proxy=proxy)
page = await browser.new_page()
await page.set_viewport_size({"width": viewport_width, "height": viewport_height})
# When a real file path is supplied, navigate to it via file://
# so relative resources (global.css, images/, fonts) resolve.
# set_content has no base URL and would render the slide
# unstyled — which collapses layouts and hides real overflow.
if file_path:
await page.goto(f"file://{file_path}")
else:
await page.set_content(html_content)
await page.wait_for_load_state("networkidle")
await page.wait_for_timeout(2000)
dims = await page.evaluate("""() => {
// Canvas = the .slide element. With overflow:hidden|clip, decorative
// absolute children (background blobs, bleed shapes) extend past the
// box but are visually clipped — using scrollWidth/Height would count
// them as overflow. Instead: use the declared box as canvas, then
// measure overflow from CONTENT-BEARING elements (text + media) only.
const slides = document.querySelectorAll('.slide, .slide-canvas');
const ContentTags = new Set(['IMG','SVG','VIDEO','CANVAS','PICTURE','IFRAME']);
function ownText(el){
let t='';
for (const n of el.childNodes){
if (n.nodeType === Node.TEXT_NODE) t += n.textContent;
}
return t.trim();
}
function isContent(el){
if (ContentTags.has(el.tagName)) return true;
return ownText(el).length > 0;
}
// Effective opacity along the ancestor chain. Returns 0 if any
// ancestor is display:none / visibility:hidden — collapses the
// old isVisible check into the same signal.
function effectiveOpacity(el){
let op = 1;
let cur = el;
while (cur && cur !== document.body){
const s = getComputedStyle(cur);
if (s.display === 'none' || s.visibility === 'hidden') return 0;
const o = parseFloat(s.opacity);
if (!isNaN(o)) op *= o;
cur = cur.parentElement;
}
return op;
}
// Three signals that an element is decorative and should not
// contribute to overflow:
// 1. Author opt-out via data-decor / aria-hidden="true".
// 2. Effective opacity ≤ 0.2 (mirrors the text-overlap
// checker's ghost / watermark threshold).
function isDecorative(el){
if (el.closest('[data-decor], [aria-hidden="true"]')) return true;
if (effectiveOpacity(el) <= 0.2) return true;
return false;
}
// Walk ancestors UP TO (but not including) the slide root;
// if any inner ancestor clips overflow, intersect the rect
// with the clipping box. Decoration that bleeds past an
// inner card / panel and is hidden by overflow:hidden|clip
// is invisible to the user and must not be flagged. We
// intentionally do NOT clip by the .slide canvas itself —
// even when .slide has overflow:hidden (it almost always
// does), content that wants to extend past 1280x720 is
// still bad layout and must surface as overflow.
function visibleRect(el, slide){
let r = el.getBoundingClientRect();
let left = r.left, top = r.top, right = r.right, bottom = r.bottom;
let cur = el.parentElement;
while (cur && cur !== slide && cur !== document.body){
const s = getComputedStyle(cur);
const clipped = (
s.overflow === 'hidden' || s.overflow === 'clip' ||
s.overflowX === 'hidden' || s.overflowX === 'clip' ||
s.overflowY === 'hidden' || s.overflowY === 'clip'
);
if (clipped){
const cr = cur.getBoundingClientRect();
left = Math.max(left, cr.left);
top = Math.max(top, cr.top);
right = Math.min(right, cr.right);
bottom = Math.min(bottom, cr.bottom);
if (right <= left || bottom <= top) return null;
}
cur = cur.parentElement;
}
return {left, top, right, bottom,
width: right - left, height: bottom - top};
}
function maxOverflow(slide){
const cr = slide.getBoundingClientRect();
let ox = 0, oy = 0;
for (const el of slide.querySelectorAll('*')){
if (!isContent(el)) continue;
if (isDecorative(el)) continue;
const r = visibleRect(el, slide);
if (!r || r.width < 1 || r.height < 1) continue;
ox = Math.max(ox, r.right - cr.right, cr.left - r.left);
oy = Math.max(oy, r.bottom - cr.bottom, cr.top - r.top);
}
return {ox: Math.max(0, ox), oy: Math.max(0, oy),
cw: slide.offsetWidth, ch: slide.offsetHeight};
}
if (slides.length){
let cw=0, ch=0, ox=0, oy=0;
for (const s of slides){
const m = maxOverflow(s);
cw = Math.max(cw, m.cw);
ch = Math.max(ch, m.ch);
ox = Math.max(ox, m.ox);
oy = Math.max(oy, m.oy);
}
return {scrollWidth: cw + ox, scrollHeight: ch + oy,
clientWidth: cw, clientHeight: ch};
}
const body = document.body;
const html = document.documentElement;
return {
scrollWidth: Math.max(body.scrollWidth, html.scrollWidth, body.offsetWidth, html.offsetWidth),
scrollHeight: Math.max(body.scrollHeight, html.scrollHeight, body.offsetHeight, html.offsetHeight),
clientWidth: html.clientWidth,
clientHeight: html.clientHeight,
};
}""")
overlaps = []
if detect_overlaps:
overlaps = await page.evaluate(_TEXT_OVERLAP_JS, {"minOverlapPx": 3, "maxPairs": 50})
screenshot_bytes = None
decor_rects = []
if detect_blank:
# Bounding rects of decorative elements, in viewport pixel
# coordinates that match the screenshot clip. Used by the
# blank detector to avoid flagging intentional decoration as
# a blank patch.
decor_rects = await page.evaluate(r"""() => {
const out = [];
function pushRects(el){
const rs = el.getClientRects();
for (const r of rs){
if (r.width >= 1 && r.height >= 1){
out.push({x: r.x, y: r.y, w: r.width, h: r.height});
}
}
}
// Explicit author opt-out.
for (const el of document.querySelectorAll('[data-decor], [aria-hidden="true"]')){
pushRects(el);
}
// Effective-opacity heuristic: ghost / watermark layer.
function effOpacity(el){
let op = 1;
let cur = el;
while (cur && cur !== document.body){
const s = getComputedStyle(cur);
if (s.display === 'none' || s.visibility === 'hidden') return 0;
const o = parseFloat(s.opacity);
if (!isNaN(o)) op *= o;
cur = cur.parentElement;
}
return op;
}
for (const el of document.querySelectorAll('body *')){
if (el.closest('[data-decor], [aria-hidden="true"]')) continue;
const op = effOpacity(el);
if (op > 0 && op <= 0.2) pushRects(el);
}
return out;
}""")
screenshot_bytes = await page.screenshot(
clip={"x": 0, "y": 0, "width": viewport_width, "height": viewport_height},
type="png",
)
await browser.close()
width = int(dims["scrollWidth"])
height = int(dims["scrollHeight"])
# Tolerance is ratio + fixed buffer (default 2% + 5px → ~19px at 720 /
# ~30px at 1280) to absorb sub-pixel rendering noise and line-box
# bleed. Both axes use the same ratio + buffer formula, scaled to the
# axis's viewport length.
tol_y = int(viewport_height * overflow_tolerance_ratio) + overflow_tolerance_px
tol_x = int(viewport_width * overflow_tolerance_ratio) + overflow_tolerance_px
result = {
"success": True,
"width": width,
"height": height,
"viewport_width": viewport_width,
"viewport_height": viewport_height,
"overflow_x": max(0, width - viewport_width - tol_x),
"overflow_y": max(0, height - viewport_height - tol_y),
"details": dims,
"overlaps": overlaps,
}
if detect_blank and screenshot_bytes is not None:
blank = _detect_largest_blank(screenshot_bytes, decor_rects=decor_rects)
if blank is not None:
blank["flagged"] = (
blank["area_ratio"] >= blank_min_area_ratio
and min(blank["w"], blank["h"]) >= blank_min_side
)
result["blank"] = blank
return result
except Exception:
return {"success": False, "error": traceback.format_exc()}
def _trailing_int(stem: str) -> int | None:
m = re.search(r"(\d+)$", stem)
return int(m.group(1)) if m else None
def _collect_targets(args) -> list[Path]:
if args.files:
return [Path(f).resolve() for f in args.files]
base = Path(args.dir).resolve()
if not base.is_dir():
raise SystemExit(f"check_slides: --dir {base} is not a directory")
files = sorted(base.glob("*.html"), key=lambda p: (_trailing_int(p.stem) or 0, p.name))
if args.positions:
wanted = set(args.positions)
files = [p for p in files if (_trailing_int(p.stem) in wanted)]
return files
async def _measure_one(path: Path, sem: asyncio.Semaphore) -> tuple[Path, dict]:
try:
html = path.read_text(encoding="utf-8")
except Exception as exc:
return path, {"success": False, "error": f"read failed: {exc}"}
async with sem:
result = await measure_slide_dimensions(
html,
viewport_width=_VIEWPORT_W,
viewport_height=_VIEWPORT_H,
proxy=_PROXY,
detect_overlaps=True,
detect_blank=True,
file_path=str(path),
)
return path, result
async def _measure_all(targets: list[Path]) -> list[tuple[Path, dict]]:
# One event loop drives every slide concurrently; the semaphore caps how
# many headless browsers run at once so a large deck can't exhaust memory.
sem = asyncio.Semaphore(min(_MAX_CONCURRENCY, len(targets)))
return await asyncio.gather(*(_measure_one(p, sem) for p in targets))
def _format_report(results: list[tuple[Path, dict]]) -> tuple[str, bool]:
lines = [
f"Checked {len(results)} slide(s) at viewport {_VIEWPORT_W}x{_VIEWPORT_H} "
f"(actual rendered width × height; '✗' marks overflow, overlap, or large blank):"
]
overflow_count = blank_count = overlap_count = fail_count = ok_count = 0
for path, r in results:
label = path.name
if not r.get("success"):
err = (r.get("error") or "unknown")
err = err.strip().splitlines()[-1][:140] if err else "unknown"
lines.append(f"- {label}: render FAILED — {err}")
fail_count += 1
continue
w, h = r["width"], r["height"]
ox = r.get("overflow_x", 0)
oy = r.get("overflow_y", 0)
blank = r.get("blank") or {}
overlaps = r.get("overlaps") or []
flags = []
if ox > 0:
flags.append(f"horizontal overflow +{ox}px")
if oy > 0:
flags.append(f"vertical overflow +{oy}px")
if blank.get("flagged"):
flags.append(
f"large blank region {blank['w']}x{blank['h']}px "
f"@ ({blank['x']},{blank['y']}) ≈ {int(blank['area_ratio'] * 100)}% of canvas"
)
if overlaps:
flags.append(f"{len(overlaps)} text overlap(s)")
if flags:
if ox > 0 or oy > 0:
overflow_count += 1
if blank.get("flagged"):
blank_count += 1
if overlaps:
overlap_count += 1
lines.append(f"- {label}: {w}x{h} ✗ {', '.join(flags)}")
for o in overlaps[:5]:
a, b, ov = o["a"], o["b"], o["overlap"]
ta = (a["text"][:36] + "…") if len(a["text"]) > 37 else a["text"]
tb = (b["text"][:36] + "…") if len(b["text"]) > 37 else b["text"]
lines.append(
f" · <{a['tag']}>'{ta}' ⨯ <{b['tag']}>'{tb}' "
f"(overlap {ov['w']}x{ov['h']}px @ ({ov['x']},{ov['y']}))"
)
if len(overlaps) > 5:
lines.append(f" · … +{len(overlaps) - 5} more overlap pair(s)")
else:
ok_count += 1
lines.append(f"- {label}: {w}x{h} ✓")
all_clean = (overflow_count == blank_count == overlap_count == fail_count == 0)
if all_clean:
lines.append(f"Summary: all {len(results)} slides fit the {_VIEWPORT_W}x{_VIEWPORT_H} canvas with no overlaps or large blank regions.")
else:
lines.append(
f"Summary: {overflow_count} overflowing, "
f"{overlap_count} with text overlaps, "
f"{blank_count} with large blank, {fail_count} failed, {ok_count} OK. "
"Use the Read tool to inspect each flagged file, then Edit for surgical fixes."
)
return "\n".join(lines), all_clean
def main() -> int:
p = argparse.ArgumentParser(description="Check rendered slide dimensions / overflow / blank / overlaps.")
p.add_argument("files", nargs="*", help="Slide HTML files to check (paths).")
p.add_argument("--dir", help="Directory of slide HTMLs (used when no positional files are given).")
p.add_argument("--positions", type=int, nargs="*",
help="When --dir is used, restrict to files whose name ends with these integers.")
p.add_argument("--json", action="store_true", help="Emit raw per-file measurements as JSON.")
p.add_argument("--output", type=Path, default=None, help="Write output here instead of stdout.")
args = p.parse_args()
if not args.files and not args.dir:
p.error("provide one or more file paths, or --dir <slides_dir>")
targets = _collect_targets(args)
if not targets:
msg = "check_slides: no slide files found."
if args.output:
args.output.write_text(msg + "\n", encoding="utf-8")
else:
sys.stdout.write(msg + "\n")
return 1
results = asyncio.run(_measure_all(targets))
if args.json:
payload = json.dumps(
{str(path): r for path, r in results},
ensure_ascii=False,
indent=2,
)
all_clean = all(r.get("success") and r.get("overflow_x", 0) == 0
and r.get("overflow_y", 0) == 0
and not (r.get("blank") or {}).get("flagged") and not (r.get("overlaps") or [])
for _, r in results)
else:
payload, all_clean = _format_report(results)
if args.output:
args.output.write_text(payload + "\n", encoding="utf-8")
else:
sys.stdout.write(payload + "\n")
return 0 if all_clean else 1
if __name__ == "__main__":
sys.exit(main())
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""Extract every <style>...</style> block from a reference HTML and write the
concatenated CSS to <output_dir>/global.css.
Usage:
python extract_html_style.py <html_path> <output_dir> [--name FILE]
The output file name defaults to global.css so it slots straight into the
ppt skill's `<work_dir>/slides/global.css` convention. Pass --name to override.
"""
from __future__ import annotations
import argparse
import re
import sys
from pathlib import Path
STYLE_RE = re.compile(r"<style\b[^>]*>(.*?)</style>", re.DOTALL | re.IGNORECASE)
def extract_style_blocks(html_text: str) -> list[str]:
return [m.group(1) for m in STYLE_RE.finditer(html_text)]
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
ap.add_argument("html_path", type=Path, help="path to source HTML file")
ap.add_argument("output_dir", type=Path, help="directory to write the CSS into")
ap.add_argument("--name", default="global.css",
help="output CSS file name (default: global.css)")
args = ap.parse_args()
if not args.html_path.is_file():
print(f"error: {args.html_path} is not a file", file=sys.stderr)
return 2
text = args.html_path.read_text(encoding="utf-8", errors="replace")
blocks = extract_style_blocks(text)
if not blocks:
print(f"error: no <style> blocks found in {args.html_path}", file=sys.stderr)
return 1
args.output_dir.mkdir(parents=True, exist_ok=True)
css = "\n\n".join(b.strip() for b in blocks)
out = args.output_dir / args.name
out.write_text(css, encoding="utf-8")
print(f"wrote {out} ({len(css)} bytes from {len(blocks)} <style> block(s))")
return 0
if __name__ == "__main__":
sys.exit(main())
+270
View File
@@ -0,0 +1,270 @@
#!/usr/bin/env python3
"""Analyse a reference .pptx and synthesise a deck-wide design footprint.
Reads `ppt/theme/theme1.xml` (color scheme + font scheme) and the first
`ppt/slides/slide1.xml` (actual rPr usage) from the pptx zip. Produces:
<output_dir>/global.css — slide-canvas CSS using extracted variables
<output_dir>/design.json — palette + typography in slides_brief.json
shape, plus the raw theme dump for audit
Usage:
python extract_pptx_style.py <pptx_path> <output_dir>
The script imitates the pptx's palette / typography so downstream slide
generation can stay faithful to the reference deck. No external deps —
uses zipfile + xml.etree only.
"""
from __future__ import annotations
import argparse
import collections
import json
import re
import sys
import zipfile
from pathlib import Path
from xml.etree import ElementTree as ET
NS = {
"a": "http://schemas.openxmlformats.org/drawingml/2006/main",
"p": "http://schemas.openxmlformats.org/presentationml/2006/main",
"r": "http://schemas.openxmlformats.org/officeDocument/2006/relationships",
}
# Standard PowerPoint preset colors → hex. Used when a clrScheme slot points at
# a named preset rather than an explicit srgbClr.
PRESET_COLORS = {
"windowText": "#000000", "windowBackground": "#FFFFFF",
"menuText": "#000000", "menuBar": "#FFFFFF",
}
def _hex_from_node(node: ET.Element | None) -> str | None:
if node is None:
return None
srgb = node.find("a:srgbClr", NS)
if srgb is not None and srgb.get("val"):
return "#" + srgb.get("val").upper()
sys_ = node.find("a:sysClr", NS)
if sys_ is not None:
# sysClr always carries a lastClr fallback for the rendered hex.
last = sys_.get("lastClr")
if last:
return "#" + last.upper()
val = sys_.get("val")
if val and val in PRESET_COLORS:
return PRESET_COLORS[val]
preset = node.find("a:prstClr", NS)
if preset is not None and preset.get("val") in PRESET_COLORS:
return PRESET_COLORS[preset.get("val")]
return None
def parse_theme(theme_xml: bytes) -> dict:
root = ET.fromstring(theme_xml)
out: dict = {"palette": {}, "fonts": {"heading": None, "body": None}}
clr = root.find(".//a:clrScheme", NS)
if clr is not None:
for slot in (
"dk1", "lt1", "dk2", "lt2",
"accent1", "accent2", "accent3", "accent4", "accent5", "accent6",
"hlink", "folHlink",
):
node = clr.find(f"a:{slot}", NS)
hex_val = _hex_from_node(node)
if hex_val:
out["palette"][slot] = hex_val
fonts = root.find(".//a:fontScheme", NS)
if fonts is not None:
major = fonts.find("a:majorFont/a:latin", NS)
if major is not None and major.get("typeface"):
out["fonts"]["heading"] = major.get("typeface")
minor = fonts.find("a:minorFont/a:latin", NS)
if minor is not None and minor.get("typeface"):
out["fonts"]["body"] = minor.get("typeface")
# East-asian fallback — useful when the deck is Chinese.
major_ea = fonts.find("a:majorFont/a:ea", NS)
if major_ea is not None and major_ea.get("typeface"):
out["fonts"]["heading_ea"] = major_ea.get("typeface")
minor_ea = fonts.find("a:minorFont/a:ea", NS)
if minor_ea is not None and minor_ea.get("typeface"):
out["fonts"]["body_ea"] = minor_ea.get("typeface")
return out
def sample_slide_usage(slide_xml: bytes) -> dict:
"""Inspect actual srgbClr fills + rPr typefaces used on slide 1."""
root = ET.fromstring(slide_xml)
colors = collections.Counter()
typefaces = collections.Counter()
sizes = collections.Counter()
for srgb in root.iter(f"{{{NS['a']}}}srgbClr"):
if srgb.get("val"):
colors["#" + srgb.get("val").upper()] += 1
for rpr in root.iter(f"{{{NS['a']}}}rPr"):
sz = rpr.get("sz")
if sz:
sizes[int(sz)] += 1
for tag in ("latin", "ea", "cs"):
face = rpr.find(f"a:{tag}", NS)
if face is not None and face.get("typeface"):
typefaces[face.get("typeface")] += 1
return {
"top_colors": colors.most_common(8),
"top_typefaces": typefaces.most_common(6),
"top_font_sizes_hpt": sizes.most_common(6), # OOXML stores sz in hundredths of pt
}
def synthesize_design(theme: dict, slide: dict | None) -> dict:
p = theme["palette"]
bg = p.get("lt1") or p.get("lt2") or "#FFFFFF"
primary = p.get("dk1") or p.get("dk2") or "#1C1917"
accent = p.get("accent1") or "#2563EB"
# If slide1 uses a markedly more common non-bg, non-primary color, prefer
# it as the accent — themes sometimes carry a placeholder accent that the
# actual deck never uses.
if slide:
for hex_val, _count in slide["top_colors"]:
if hex_val.upper() not in {bg.upper(), primary.upper()}:
accent = hex_val
break
heading = theme["fonts"].get("heading") or "Inter"
body = theme["fonts"].get("body") or heading
return {
"background": bg,
"primary": primary,
"accent": accent,
"heading_font": heading,
"body_font": body,
}
CSS_TEMPLATE = """\
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=Crimson+Text:wght@400;600;700&family=Noto+Sans+SC:wght@400;500;700&family=Noto+Serif+SC:wght@400;700&display=swap');
:root {{
--bg: {bg};
--primary: {primary};
--accent: {accent};
}}
* {{ box-sizing: border-box; margin: 0; padding: 0; }}
body {{
font-family: '{body_font}', 'Inter', 'Helvetica Neue', 'Segoe UI', Arial,
'Noto Sans SC', 'PingFang SC', 'Microsoft YaHei', sans-serif;
background-color: var(--bg);
color: var(--primary);
}}
.slide {{
width: 1280px;
height: 720px;
background-color: var(--bg);
padding: 64px;
position: relative;
overflow: hidden;
}}
h1, h2, h3, .heading {{
font-family: '{heading_font}', 'Crimson Text', 'Times New Roman', Georgia,
'Noto Serif SC', 'Songti SC', 'SimSun', serif;
color: var(--primary);
line-height: 1.1;
}}
h1 {{ font-size: 88px; }}
h2 {{ font-size: 52px; }}
h3 {{ font-size: 32px; }}
p, li, td, .body-text {{
font-family: '{body_font}', 'Inter', 'Helvetica Neue', Arial,
'Noto Sans SC', 'PingFang SC', 'Microsoft YaHei', sans-serif;
font-size: 22px;
line-height: 1.55;
color: var(--primary);
}}
.accent-fg {{ color: var(--accent); }}
.accent-bg {{ background-color: var(--accent); color: var(--bg); }}
.accent-border {{ border: 4px solid var(--accent); }}
"""
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
ap.add_argument("pptx_path", type=Path)
ap.add_argument("output_dir", type=Path)
args = ap.parse_args()
if not args.pptx_path.is_file() or args.pptx_path.suffix.lower() != ".pptx":
print(f"error: {args.pptx_path} is not a .pptx file", file=sys.stderr)
return 2
args.output_dir.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(args.pptx_path) as z:
names = z.namelist()
theme_files = sorted(n for n in names if re.match(r"ppt/theme/theme\d+\.xml$", n))
if not theme_files:
print("error: no ppt/theme/theme*.xml inside pptx", file=sys.stderr)
return 1
theme_xml = z.read(theme_files[0])
slide_xml = None
if "ppt/slides/slide1.xml" in names:
slide_xml = z.read("ppt/slides/slide1.xml")
theme = parse_theme(theme_xml)
slide = sample_slide_usage(slide_xml) if slide_xml else None
design = synthesize_design(theme, slide)
css = CSS_TEMPLATE.format(
bg=design["background"],
primary=design["primary"],
accent=design["accent"],
heading_font=design["heading_font"],
body_font=design["body_font"],
)
css_path = args.output_dir / "global.css"
css_path.write_text(css, encoding="utf-8")
design_payload = {
"design": {
"palette": {
"background": design["background"],
"primary": design["primary"],
"accent": design["accent"],
},
"typography": {
"heading": design["heading_font"],
"body": design["body_font"],
},
"reference": f"imitating {args.pptx_path.name}",
},
"source_pptx": str(args.pptx_path.resolve()),
"raw_theme": theme,
"slide1_usage": slide,
}
design_path = args.output_dir / "design.json"
design_path.write_text(json.dumps(design_payload, ensure_ascii=False, indent=2),
encoding="utf-8")
print(f"wrote {css_path}")
print(f"wrote {design_path}")
print(f"palette: bg={design['background']} primary={design['primary']} accent={design['accent']}")
print(f"fonts: heading={design['heading_font']} body={design['body_font']}")
if slide:
print(f"slide1 top colors: {slide['top_colors'][:3]}")
print(f"slide1 top typefaces: {slide['top_typefaces'][:3]}")
return 0
if __name__ == "__main__":
sys.exit(main())