Add an ocr_lines slot (initially None) to each page's data dictionary
in the upload endpoint. This prepares for OCR caching so results can
be reused between the auto-detect and spot-redaction pipelines.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Captures the plan to decouple auto-detection from upload, make manual
(double-click) redactions first-class with full analysis, and move to
inline-on-canvas editing.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The rename from solveFilterPrefix/Suffix to solveKnownStart/End missed
associates.js, breaking the entire JS module tree and disabling upload.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replaced test_solve_endpoint_dictionary and test_dictionary_crud with
test_solve_endpoint_name_mode testing the new name mode.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Modes: name (default), full_name, email, enumerate
Renamed filter_prefix/suffix to known_start/known_end
Removed DictionaryStore and dictionary API endpoints
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Renamed solveFilterPrefix/Suffix to solveKnownStart/End. Word filter
label hidden by default, shown only when enumerate mode is selected.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Modes: Name (default), Full Name, Email, Enumerate
Charsets: lowercase, UPPERCASE, Capitalized
Renamed filter prefix/suffix to known start/end
Word filter hidden by default (shown only for enumerate mode)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replaces uppercase_only bool with casing string parameter. Adds
known_start/known_end support for filtering and gap-width measurement.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Supports casing (lowercase/uppercase/capitalized), known_start/known_end
filtering with gap-width measurement of the unknown portion only.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Generate first x last name combinations from associate lists, apply
Title Case or UPPER casing, measure rendered widths, and return matches
within tolerance. Also checks associate name variants (initials,
nicknames) with deduplication.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
The full-name solver collected all results into a Vec with no limit,
causing OOM crashes for realistic targets where millions of valid
first×last combinations exist. Thread max_results through
solve_subtree, dfs_first, and the Cartesian product loop so the
search stops early once enough results are found.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Thread EquivClasses and deduped_allowed through solve_subtree, dfs,
generate_prefixes, handle_solve, and dfs_first (full_name). The DFS
hot loop now iterates deduped representatives instead of all allowed
chars, and expand_match restores full class-member variants at match
time. This reduces branching factor for charsets with metrically
identical glyphs while preserving complete result sets.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Foundation for glyph equivalence class optimization. Groups charset
characters with identical metric signatures (width table columns/rows,
left/right edges, optional extra columns) into equivalence classes
using exact f64::to_bits() comparison. Includes 3 unit tests covering
duplicate detection, all-unique case, and extra column splitting.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Group metrically identical characters into equivalence classes to reduce
the DFS branching factor from N to K per node, with Cartesian product
expansion on matches to recover all solutions.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace the click-to-analyze and spot-detection flows with an automatic
SSE-driven analysis pipeline. After upload, an EventSource connects to
/api/doc/{id}/analyze and reloads page data as each page completes,
consuming pre-computed analysis from the page data endpoint.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace the old /api/redaction/analyze and /api/redaction/spot endpoints
with a new /api/doc/{id}/analyze SSE endpoint that pre-computes all
analysis during upload. The page data endpoint now returns pre-computed
analysis results instead of requiring per-redaction API calls.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Wire together the per-page pipeline: OCR -> LLM redaction detection ->
guided OpenCV search -> font detection with masking -> pixel alignment.
Includes font caching per line and progress callbacks.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Adds a new detect_font_masked() function that masks redaction boxes to
white (255) before running pixel-based font detection on the full line
crop. This replaces the fragile approach of cropping just the left/right
portion of a line, which lost signal from narrow crops and coordinate
mismatches. The Dice-based scoring naturally ignores masked (white)
regions since they contain no ink pixels.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add find_redaction_in_region() that searches for a black rectangle
within a specific page region identified by the LLM. Uses a lower
minimum area threshold (100px vs 500px) since the LLM provides
spatial guidance, preventing merged redactions and false positives.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Introduce llm_detect.py that sends OCR lines to Claude Haiku to identify
redaction locations via broken text patterns (garbled chars, brackets,
pipes). The LLM returns boundary words which are mapped back to pixel
positions using OCR character data. Includes comprehensive tests covering
single/multiple redactions, line start/end edge cases, and error handling.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Instead of using a neighboring line (which may have a different font/size,
e.g. email header vs body text), crop the non-redacted left or right
portion of the actual redacted line for font detection. Falls back to
clean neighbor lines only when neither side has enough text (>=50px).
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace OCR-based offset calculation with pixel alignment that slides
rendered text across the page image to find the exact position with
maximum Dice overlap. Also fix char filtering to use center-point
(c.x + c.w/2) instead of right-edge, recovering characters like "nt"
in "Sent" that partially overlap the redaction box.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Redaction boxes corrupt OCR text and add spurious dark pixels that
break pixel-based font scoring. Now uses wider crop padding (3x)
and prefers the nearest non-overlapping line for font detection.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace word-width MAE scoring with pixel-overlap Dice scoring in both
_full_search() and _fine_search(). Both functions now accept a line_crop
parameter and use _score_font_line_pixel() instead of _score_font_line().
Comparisons are flipped (higher is better). The size range is now derived
from the OCR line height instead of the fixed SIZE_RANGE constant, and
uses step=1 since pixel scoring has sharp peaks at the exact font size.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add _score_font_line_pixel() which renders candidate text and compares
pixels against the actual page image using Dice coefficient overlap.
This captures glyph shape differences that the old word-width MAE
approach cannot distinguish (e.g. serif vs sans-serif).
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Centralize 21 repeated color values into :root variables.
Makes theme changes trivial and improves readability.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Reorganize the monolithic 1,436-line frontend into 9 focused modules:
types, state, dom, canvas, viewport, associates, popover, solver, main.
No build step — browsers load ES modules natively. JSDoc @typedef
annotations provide type safety via @ts-check + jsconfig.json.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Without flex properties, #viewer-section collapsed to content
height (just the controls bar), giving the workspace 0 height
and causing zoom to compute as 0%.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Remove unused detect_redactions import from app.py
- Add de-duplication check in dblclick handler to prevent
creating duplicate redaction entries for the same dark region
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Shift+drag now adjusts gap width (was ctrl+shift+drag).
Manual redaction marking via shift+drag is removed;
users now double-click to spot-detect redactions.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Extract inline analysis logic from the analyze endpoint into a reusable
_run_analysis() helper function, then add a new /api/redaction/spot
endpoint that chains spot_redaction detection with analysis in one call.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Uses OpenCV connected components to find a dark region at a specific
click point, complementing the existing detect_redactions() which
auto-detects all redaction bars via morphological operations.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Adds green segment overlay, floating font toolbar, bottom text edit
bar, and backend offset guess to the redaction-centric UI.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Hold Shift and drag on the canvas to draw a custom redaction
rectangle. Creates an unanalyzed redaction entry that triggers
on-demand OCR when clicked.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace line list with redaction list, add canvas-native redaction
overlays with hit-testing, popover with on-demand analysis and
solver integration. Remove old font/segment/overlay controls.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
When a user clicks a redaction, this endpoint crops the surrounding line,
runs Tesseract OCR, detects the font, and returns text segments (before/after
the redaction), gap width, font info, and character-level bounding boxes.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Replace upfront OCR + font detection + overlay rendering with lightweight
OpenCV-based redaction detection. Upload now just rasterizes and detects
black rectangles, making it significantly faster. Remove the overlay
endpoint and update the data endpoint to return redaction bboxes.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Shift from upfront OCR to redaction-centric workflow with OpenCV
detection, canvas-native clickable overlays, on-demand analysis,
and per-redaction popover solver UI.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Fetches from 3 data sources (HuggingFace parquet, rhowardstone entities,
Epstein Exposed API) plus 8 court-verified addresses. Merges, deduplicates,
validates, and writes 644 unique emails to unredact/data/emails.txt.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Victim names are stored as a flat set of lowercased name variants
with no person details — the V badge appears but is not clickable,
protecting victim identities. A result can show both T1/T2/T3 and V
badges simultaneously.
- 15 known victims from public court records → 43 name variants
- Enabler/perpetrator categories promoted to T2
- Purple V badge with pointer-events disabled
- Victim matching uses prefix/suffix combination like associate matching
Co-Authored-By: Claude Opus 4.6 <[email protected]>
When matching solver results against associates, now tries:
- gap text alone ("oe")
- prefix + gap ("joe" if prefix="j")
- gap + suffix
- prefix + gap + suffix
Composite matches (prefix+gap) get a +3 score boost since the
surrounding context confirms the match. Deduplicates by person_id,
keeping the highest-scoring match.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Badge is now a button — clicking it opens a popup listing all
matching associates with their tier description (flight logs /
inner circle / named in files), category, and match type.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Sort results strictly by tier (T1 > T2 > T3 > none), then by score
within the same tier. Previously a T3 full-name match could outrank
a T1 first-name match.
- Boost associate score by +2 when the person's name matches the
user's filter prefix/suffix, surfacing likely candidates higher.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Lazy-loads associates.json from data dir and serves it at /api/associates,
with fallback to empty dicts if file is missing.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
7 tasks covering build script, API endpoint, frontend matching/sorting/badges,
and end-to-end testing. TDD approach with exact file paths and code.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Defines tiered prioritization of known Epstein associates in solver
results using rhowardstone/Epstein-research-data as the data source.
Associates are matched by full name, first/last, nicknames, and aliases
with frontend-side sorting and badge display.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Measures a known word's width in font context, then verifies the solver
recovers that word from its width. Proves the width table and solver
work correctly end-to-end.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add a solve panel UI that lets users attempt to recover redacted text
by finding strings whose rendered width matches the redaction gap.
Includes charset selection, length bounds, tolerance slider, and
dictionary/enumerate/both mode controls. Results stream via SSE and
can be clicked to insert into the gap.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add POST /api/solve endpoint that streams results via Server-Sent Events,
supporting both enumerate (brute-force) and dictionary solve modes. Add
dictionary CRUD endpoints (POST/GET/DELETE /api/dictionary) and solve
cancellation (DELETE /api/solve/{id}). Move static files mount to end of
app.py so API routes take priority.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add solve_dictionary() for linear-scan matching of dictionary entries
against a target pixel width, and DictionaryStore for managing named
word/name lists. Reuses SolveResult from the solver module.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Distribute branch-and-bound search across CPU cores by generating
depth-2 prefixes and farming each subtree to a worker process.
Includes on_progress callback and automatic prefix depth selection.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
