Commit Graph
106 Commits
Author SHA1 Message Date
Alex-GilbertandClaude Opus 4.6 bde5fce636 feat: update frontend JS for new modes, known start/end, and filter visibility
Renamed solveFilterPrefix/Suffix to solveKnownStart/End. Word filter
label hidden by default, shown only when enumerate mode is selected.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 13:12:54 -06:00
Alex-GilbertandClaude Opus 4.6 a9117e3876 feat: update HTML with new modes, charsets, and known start/end fields
Modes: Name (default), Full Name, Email, Enumerate
Charsets: lowercase, UPPERCASE, Capitalized
Renamed filter prefix/suffix to known start/end
Word filter hidden by default (shown only for enumerate mode)

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 13:12:17 -06:00
Alex-GilbertandClaude Opus 4.6 d0bd7967e4 feat: update solve_full_name_dictionary with casing and known_start/known_end
Replaces uppercase_only bool with casing string parameter. Adds
known_start/known_end support for filtering and gap-width measurement.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 13:11:53 -06:00
Alex-GilbertandClaude Opus 4.6 d32f27cb8d feat: add solve_name_dictionary for single-word associate name matching
Supports casing (lowercase/uppercase/capitalized), known_start/known_end
filtering with gap-width measurement of the unknown portion only.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 13:11:13 -06:00
Alex-GilbertandClaude Opus 4.6 57650af1fe docs: add mode/charset restructuring design and implementation plan
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 13:07:54 -06:00
Alex-GilbertandClaude Opus 4.6 c04dc589f4 feat: wire dictionary full-name solver into solve endpoint
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 12:01:53 -06:00
Alex-GilbertandClaude Opus 4.6 7b0802d005 feat: add solve_full_name_dictionary with Cartesian product matching
Generate first x last name combinations from associate lists, apply
Title Case or UPPER casing, measure rendered widths, and return matches
within tolerance. Also checks associate name variants (initials,
nicknames) with deduplication.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 12:00:45 -06:00
Alex-GilbertandClaude Opus 4.6 72df8d3579 feat: extract associate first/last name lists from registry
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 11:55:33 -06:00
Alex-GilbertandClaude Opus 4.6 921d702c07 feat: add lazy loaders for associate name lists
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 11:55:09 -06:00
Alex-GilbertandClaude Opus 4.6 291eadaba8 docs: add dictionary full-name solver implementation plan
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 11:52:39 -06:00
Alex-GilbertandClaude Opus 4.6 04aa089c7f docs: add dictionary-based full name solver design
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 11:50:50 -06:00
Alex-GilbertandClaude Opus 4.6 b50a32a038 fix(solver): add max_results cap to prevent OOM in full-name solver
The full-name solver collected all results into a Vec with no limit,
causing OOM crashes for realistic targets where millions of valid
first×last combinations exist. Thread max_results through
solve_subtree, dfs_first, and the Cartesian product loop so the
search stops early once enough results are found.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 11:20:16 -06:00
Alex-GilbertandClaude Opus 4.6 11339d0ecc test(solver): add 3-char equivalence expansion integration test
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 10:52:54 -06:00
Alex-GilbertandClaude Opus 4.6 25a8d47d53 feat(solver): wire equiv classes into DFS pipeline
Thread EquivClasses and deduped_allowed through solve_subtree, dfs,
generate_prefixes, handle_solve, and dfs_first (full_name). The DFS
hot loop now iterates deduped representatives instead of all allowed
chars, and expand_match restores full class-member variants at match
time. This reduces branching factor for charsets with metrically
identical glyphs while preserving complete result sets.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 10:50:31 -06:00
Alex-GilbertandClaude Opus 4.6 700a76eeee feat(solver): add dedup_state_allowed, build_byte_to_idx, expand_match
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 10:43:57 -06:00
Alex-GilbertandClaude Opus 4.6 be9dd07a66 feat(solver): add EquivClasses struct and compute_equiv_classes
Foundation for glyph equivalence class optimization. Groups charset
characters with identical metric signatures (width table columns/rows,
left/right edges, optional extra columns) into equivalence classes
using exact f64::to_bits() comparison. Includes 3 unit tests covering
duplicate detection, all-unique case, and extra column splitting.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 10:39:23 -06:00
Alex-Gilbert a5505c1fbb attempting to make redaction guessing better 2026-02-18 10:35:42 -06:00
Alex-GilbertandClaude Opus 4.6 cddbf74492 docs: add glyph equivalence classes implementation plan
7 bite-sized tasks with TDD steps: EquivClasses struct, dedup/expand
functions, DFS wiring, prefix generation, full_name integration, and
end-to-end testing.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 10:34:58 -06:00
Alex-GilbertandClaude Opus 4.6 23f535a614 docs: add glyph equivalence classes design
Group metrically identical characters into equivalence classes to reduce
the DFS branching factor from N to K per node, with Cartesian product
expansion on matches to recover all solutions.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-18 10:28:41 -06:00
Alex-GilbertandClaude Opus 4.6 b577f2b52e feat: update frontend to use SSE analysis and pre-computed redaction data
Replace the click-to-analyze and spot-detection flows with an automatic
SSE-driven analysis pipeline. After upload, an EventSource connects to
/api/doc/{id}/analyze and reloads page data as each page completes,
consuming pre-computed analysis from the page data endpoint.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 15:38:13 -06:00
Alex-GilbertandClaude Opus 4.6 1a65cb9a87 refactor: replace per-redaction endpoints with SSE upload analysis pipeline
Replace the old /api/redaction/analyze and /api/redaction/spot endpoints
with a new /api/doc/{id}/analyze SSE endpoint that pre-computes all
analysis during upload. The page data endpoint now returns pre-computed
analysis results instead of requiring per-redaction API calls.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 15:35:09 -06:00
Alex-GilbertandClaude Opus 4.6 0f9c7ff858 feat: add page analysis pipeline orchestrating OCR, LLM, OpenCV, and font detection
Wire together the per-page pipeline: OCR -> LLM redaction detection ->
guided OpenCV search -> font detection with masking -> pixel alignment.
Includes font caching per line and progress callbacks.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 15:31:17 -06:00
Alex-GilbertandClaude Opus 4.6 8b5511e7fd feat: add detect_font_masked for redaction-aware font detection
Adds a new detect_font_masked() function that masks redaction boxes to
white (255) before running pixel-based font detection on the full line
crop. This replaces the fragile approach of cropping just the left/right
portion of a line, which lost signal from narrow crops and coordinate
mismatches. The Dice-based scoring naturally ignores masked (white)
regions since they contain no ink pixels.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 15:26:14 -06:00
Alex-GilbertandClaude Opus 4.6 c39d74e46d feat: add guided OpenCV search for LLM-directed redaction regions
Add find_redaction_in_region() that searches for a black rectangle
within a specific page region identified by the LLM. Uses a lower
minimum area threshold (100px vs 500px) since the LLM provides
spatial guidance, preventing merged redactions and false positives.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 15:22:28 -06:00
Alex-GilbertandClaude Opus 4.6 51ef030322 feat: add LLM-based redaction detection module
Introduce llm_detect.py that sends OCR lines to Claude Haiku to identify
redaction locations via broken text patterns (garbled chars, brackets,
pipes). The LLM returns boundary words which are mapped back to pixel
positions using OCR character data. Includes comprehensive tests covering
single/multiple redactions, line start/end edge cases, and error handling.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 13:59:49 -06:00
Alex-GilbertandClaude Opus 4.6 72eae5a468 docs: add pipeline refactor implementation plan
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 13:53:10 -06:00
Alex-GilbertandClaude Opus 4.6 23376ebe00 docs: add pipeline refactor design (LLM-guided redaction detection)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 13:48:29 -06:00
Alex-GilbertandClaude Opus 4.6 dbc12f6064 fix: use same-line non-redacted text for font detection
Instead of using a neighboring line (which may have a different font/size,
e.g. email header vs body text), crop the non-redacted left or right
portion of the actual redacted line for font detection. Falls back to
clean neighbor lines only when neither side has enough text (>=50px).

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 13:16:23 -06:00
Alex-GilbertandClaude Opus 4.6 a2ef97b349 feat: pixel-aligned text positioning for redaction overlay
Replace OCR-based offset calculation with pixel alignment that slides
rendered text across the page image to find the exact position with
maximum Dice overlap. Also fix char filtering to use center-point
(c.x + c.w/2) instead of right-edge, recovering characters like "nt"
in "Sent" that partially overlap the redaction box.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 13:02:36 -06:00
Alex-GilbertandClaude Opus 4.6 d00b189684 docs: add pixel-aligned text positioning design
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 12:47:07 -06:00
Alex-GilbertandClaude Opus 4.6 230d0ed0a1 fix: use clean neighboring lines for font detection near redactions
Redaction boxes corrupt OCR text and add spurious dark pixels that
break pixel-based font scoring. Now uses wider crop padding (3x)
and prefers the nearest non-overlapping line for font detection.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 12:33:47 -06:00
Alex-GilbertandClaude Opus 4.6 1ad17a72a9 refactor: remove old word-width scoring function
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 12:17:29 -06:00
Alex-GilbertandClaude Opus 4.6 5590c3c0d3 fix: pass page image to detect_font_for_line in analysis
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 12:14:43 -06:00
Alex-GilbertandClaude Opus 4.6 c0b4cc2d61 feat: update font detection API to pass page image for pixel scoring
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 11:43:11 -06:00
Alex-GilbertandClaude Opus 4.6 93f4502bfa fix: update score docstring, remove dead SIZE_RANGE, fix unused import
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 11:39:13 -06:00
Alex-GilbertandClaude Opus 4.6 1b863fdf21 feat: wire pixel scoring into _full_search and _fine_search
Replace word-width MAE scoring with pixel-overlap Dice scoring in both
_full_search() and _fine_search(). Both functions now accept a line_crop
parameter and use _score_font_line_pixel() instead of _score_font_line().
Comparisons are flipped (higher is better). The size range is now derived
from the OCR line height instead of the fixed SIZE_RANGE constant, and
uses step=1 since pixel scoring has sharp peaks at the exact font size.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 11:33:10 -06:00
Alex-GilbertandClaude Opus 4.6 043aae5abb feat: add pixel-based font scoring function
Add _score_font_line_pixel() which renders candidate text and compares
pixels against the actual page image using Dice coefficient overlap.
This captures glyph shape differences that the old word-width MAE
approach cannot distinguish (e.g. serif vs sans-serif).

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 11:16:20 -06:00
Alex-GilbertandClaude Opus 4.6 cb15bb068b docs: add pixel-based font detection implementation plan
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 11:11:53 -06:00
Alex-GilbertandClaude Opus 4.6 f4703285ed docs: add pixel-based font detection design
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 11:08:41 -06:00
Alex-GilbertandClaude Opus 4.6 8f432cc220 refactor: extract CSS custom properties for theme colors
Centralize 21 repeated color values into :root variables.
Makes theme changes trivial and improves readability.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 10:52:53 -06:00
Alex-GilbertandClaude Opus 4.6 bc65b82b26 feat: split app.js into ES modules with JSDoc types
Reorganize the monolithic 1,436-line frontend into 9 focused modules:
types, state, dom, canvas, viewport, associates, popover, solver, main.

No build step — browsers load ES modules natively. JSDoc @typedef
annotations provide type safety via @ts-check + jsconfig.json.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 10:42:11 -06:00
Alex-GilbertandClaude Opus 4.6 fd3617f98d docs: add frontend modularization implementation plan
Complete code for all 9 ES modules with JSDoc types.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 08:30:46 -06:00
Alex-GilbertandClaude Opus 4.6 7a8cda542c docs: add frontend modularization design
Split app.js into ES modules with JSDoc types, no build step.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-17 08:18:35 -06:00
Alex-GilbertandClaude Opus 4.6 6d4a91f7d0 fix: add viewer-section flex layout to fill viewport
Without flex properties, #viewer-section collapsed to content
height (just the controls bar), giving the workspace 0 height
and causing zoom to compute as 0%.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 18:17:07 -06:00
Alex-GilbertandClaude Opus 4.6 35b898af50 fix: remove unused import, add duplicate redaction detection
- Remove unused detect_redactions import from app.py
- Add de-duplication check in dblclick handler to prevent
  creating duplicate redaction entries for the same dark region

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 18:02:54 -06:00
Alex-GilbertandClaude Opus 4.6 6ca949b233 feat: remap shift+drag to gap width, remove manual marking
Shift+drag now adjusts gap width (was ctrl+shift+drag).
Manual redaction marking via shift+drag is removed;
users now double-click to spot-detect redactions.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 18:00:45 -06:00
Alex-GilbertandClaude Opus 4.6 dc40c1db00 feat: replace double-click zoom with spot redaction detection
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 18:00:08 -06:00
Alex-GilbertandClaude Opus 4.6 3d739b13f5 feat: add toast notification system
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 17:59:40 -06:00
Alex-GilbertandClaude Opus 4.6 fe9eb289ac feat: remove auto-detection from upload pipeline
Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 17:59:00 -06:00
Alex-GilbertandClaude Opus 4.6 7fb5a85634 feat: extract _run_analysis helper and add /api/redaction/spot endpoint
Extract inline analysis logic from the analyze endpoint into a reusable
_run_analysis() helper function, then add a new /api/redaction/spot
endpoint that chains spot_redaction detection with analysis in one call.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-16 17:56:48 -06:00