docs: add design for email unredaction via known email dictionary

Co-Authored-By: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Alex-GilbertandClaude Opus 4.6 committed 2026-02-16 13:16:37 -06:00
1 parent 2268e8549e
commit d0a39c6dce
1 file changed
+62
@@ -0,0 +1,62 @@
# Email Unredaction via Known Email Dictionary
## Problem
Many redactions in the Epstein case files are email addresses, typically enclosed in `< >` angle brackets. Some have a visible `@` splitting the redaction into username/domain sub-gaps. The current solver supports name and word matching but has no email-specific mode.
## Solution
Add a known-email dictionary mode to the solver. Given a pixel-width gap and font/size, scan a list of known Epstein-network email addresses and return any that fit the gap within tolerance, using kerning-aware width computation with the surrounding context characters (typically `<` and `>`).
## Data Sources
Build `unredact/data/emails.txt` (one email per line, lowercased, deduplicated, sorted) from:
1. **Hugging Face `notesbymuneeb/epstein-emails`** — 5,082 email threads (16,447 messages) with sender/recipient fields. Extract unique email addresses from `sender` and `recipients` fields. Parquet format, free download.
2. **Epstein Exposed Black Book API** (`epsteinexposed.com/api/v1/persons`) — ~395 email addresses from the seized address book. Free REST API, no auth, 60 req/min rate limit.
3. **rhowardstone `extracted_entities_filtered.json`** — 357 emails from DOJ document entity extraction. Already used by the project for associates data.
4. **Court-verified (hardcoded)** — 8 addresses from primary court documents:
- `[email protected]` (Epstein, Giuffre v. Maxwell Exhibit J)
- `[email protected]` (State Dept FOIA)
- `[email protected]` (State Dept FOIA)
- `[email protected]` (State Dept FOIA)
- `[email protected]` (FOIA + Bryant v. Indyke)
- `[email protected]` (Bryant v. Indyke)
- `[email protected]` (NM State Land Office)
- `[email protected]` (Maxwell, court testimony)
### Build Script
`scripts/build_emails.py` — fetches from sources, merges, deduplicates, writes `emails.txt`. Added as `make build-emails` Makefile target.
## Integration
### Backend (`app.py` / `solver.py`)
When solve mode is `"emails"`:
- Load `emails.txt` into memory (lazy, cached like other wordlists)
- For each email, compute kerning-aware width: `font.getlength(left_ctx + email + right_ctx) - left_width - right_width`
- Return matches within tolerance, sorted by width error
This is the same linear-scan approach `dictionary.py` already uses. The left/right context (typically `<` / `>`) is already sent in `SolveRequest`.
### Frontend (`index.html`)
Add `"emails"` option to the solve mode dropdown, alongside `"enumerate"`, `"dictionary"`, and `"both"`.
### No changes needed to:
- Rust solver (not involved in dictionary-style scans)
- Width table construction
- Charset constraints / state machines
- Font detection pipeline
- OCR pipeline
## Key Sources
- smw.ai/epstein-files/epstein-email-accounts — verified Epstein email addresses
- epsteinexposed.com — Black Book API with contact emails
- github.com/notesbymuneeb/epstein-emails — Hugging Face email threads dataset
- github.com/rhowardstone/Epstein-research-data — extracted entities