diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..50e8d7b --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Alex + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..dafe5a8 --- /dev/null +++ b/README.md @@ -0,0 +1,190 @@ +# Unredact + +**Reveal what's hidden in redacted PDFs.** + +Unredact uses computer vision, font-aware constraint solving, and LLM reasoning to figure out what text is hiding under those black bars. Upload a PDF, and it will detect redactions, calculate exactly which strings could fit based on pixel-width constraints, and let you visually verify guesses with a live overlay. + + +![Unredact analyzing a redacted PDF](docs/images/hero.png) + +![Green overlay text aligning with surrounding visible text](docs/images/overlay-verification.png) + +## What it does + +- **Detects redactions** automatically using computer vision, or manually by clicking +- **Solves for hidden text** by finding every string that fits the exact pixel width of a redaction, accounting for font metrics and kerning +- **Ranks results with AI** using Claude to score candidates by contextual fit with surrounding text +- **Lets you verify visually** by overlaying guessed text in green on the original document — if the characters align, the guess fits + +## How it works + +Unredact combines three techniques: + +1. **Computer vision** — OCR extracts visible text with character-level bounding boxes. OpenCV detects black rectangles. A pixel-level font matcher identifies the document's typeface and size. + +2. **Constraint solving** — Using the detected font's exact character widths (including kerning pairs), a parallel branch-and-bound solver written in Rust enumerates every string that fits the redaction's pixel width within a configurable tolerance. + +3. **LLM validation** — Claude reads the surrounding text context and scores each candidate for plausibility, then results are ranked by a composite of width fit and contextual score. + +``` +PDF ──→ Rasterize ──→ OCR ──→ Font Detection ──→ Redaction Detection + │ │ + ▼ ▼ + Width Tables ──→ Constraint Solver (Rust) + │ + ▼ + Candidates ──→ LLM Validation ──→ Ranked Results + │ + ▼ + Visual Overlay +``` + +## Quick start + +### Prerequisites + +- Python 3.12+ +- Rust toolchain ([rustup.rs](https://rustup.rs)) +- System packages: + - **poppler** — PDF rendering (`libpoppler-cpp-dev` on Debian/Ubuntu, `poppler` on Arch/macOS) + - **Tesseract** — OCR engine (`tesseract-ocr` on Debian/Ubuntu, `tesseract` on Arch/macOS) + - **fontconfig** — Font lookup (`fontconfig` — usually preinstalled on Linux) +- An [Anthropic API key](https://console.anthropic.com/) for LLM features + +### Install + +```bash +git clone https://github.com/Alex-Gilbert/unredact.git +cd unredact + +# Create Python virtualenv and install dependencies +python -m venv .venv +source .venv/bin/activate +pip install -e . + +# Build the Rust constraint solver +make build-solver +``` + +### Configure + +On first run, you'll be prompted for your Anthropic API key, which gets saved to `.env`: + +```bash +make run +# Enter your Anthropic API key (from console.anthropic.com): sk-ant-... +``` + +Or create `.env` manually: + +```bash +echo "ANTHROPIC_API_KEY=sk-ant-your-key-here" > .env +``` + +### Run + +```bash +make run +``` + +Open [http://localhost:8000](http://localhost:8000) in your browser. + +This starts both the Python web server (port 8000) and the Rust constraint solver (port 3100). Press `Ctrl+C` to stop everything. + +## Usage guide + +### 1. Upload a PDF + +Drag and drop a redacted PDF onto the page, or use the file picker. The tool will automatically run OCR on every page and detect redactions and fonts. + +### 2. Select a redaction + +Click on a detected redaction (highlighted on the page). A panel opens with analysis details and solve options. + +### 3. Choose a solve mode + +| Mode | What it searches | Use case | +|------|-----------------|----------| +| **Name** | First names or last names from name dictionaries | Redacted names | +| **Full Name** | Two-word combinations (first + last) | Full name redactions | +| **Email** | Common email addresses | Redacted email addresses | +| **Word** | English nouns and adjectives from dictionary | General text redactions | +| **Enumerate** | All possible character combinations | Short redactions, anything goes | + +### 4. Configure and solve + +- Set the **character set** (lowercase, uppercase, capitalized) +- Adjust **tolerance** if results are too narrow or too broad +- Add **known characters** if you can tell what the first or last letter is +- For word mode, toggle **plural only** or adjust **vocabulary size** +- Click **Solve** — results stream in as they're found + +### 5. Validate with AI + +Click **Validate** to have Claude score each candidate based on the surrounding text context. Results are re-ranked by a composite score combining width fit and contextual plausibility. + +### 6. Verify visually + +Select a result to see it overlaid in green on the original document. If the green text aligns character-for-character with the surrounding visible text, the guess is a strong match. You can fine-tune font, size, position, and character spacing manually. + +## Architecture + +Unredact runs as two local services: + +- **Python server** (FastAPI, port 8000) — Handles PDF processing, OCR, font detection, redaction detection, LLM calls, and serves the web UI +- **Rust solver** (Axum, port 3100) — Runs the parallel constraint solver, streams results back via SSE + +The frontend is vanilla JavaScript with no build step. It uses canvas for rendering and SSE for real-time result streaming. + +``` +Browser ◄──── SSE ────► FastAPI (Python) + │ + ├── OCR (Tesseract) + ├── Font detection (pixel matching) + ├── Redaction detection (OpenCV) + ├── LLM validation (Claude API) + │ + └──── HTTP ────► Constraint Solver (Rust) +``` + +## Development + +### Project structure + +``` +unredact/ +├── unredact/ # Python package +│ ├── app.py # FastAPI server +│ ├── pipeline/ # Processing modules (OCR, fonts, solver, LLM, etc.) +│ ├── static/ # Frontend (HTML, CSS, JS) +│ └── data/ # Bundled dictionaries and word lists +├── solver_rs/ # Rust constraint solver +│ └── src/ # Axum server + DFS solver +├── tests/ # Test suite +├── scripts/ # Data build scripts +└── Makefile # Build, run, test automation +``` + +### Useful commands + +```bash +make run # Start everything (foreground, Ctrl+C to stop) +make app # Start in background +make stop # Stop background services +make status # Check what's running +make logs # Tail logs +make test # Run test suite +make debug # Run with font debug images saved to debug/ +make build-word-lists # Rebuild noun/adjective dictionaries from WordNet +``` + +### Running tests + +```bash +# Tests require the Rust solver to be running +make test +``` + +## License + +[MIT](LICENSE) diff --git a/docs/images/.gitkeep b/docs/images/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/images/hero.png b/docs/images/hero.png new file mode 100644 index 0000000..bbcc036 Binary files /dev/null and b/docs/images/hero.png differ diff --git a/docs/images/overlay-verification.png b/docs/images/overlay-verification.png new file mode 100644 index 0000000..5422bfa Binary files /dev/null and b/docs/images/overlay-verification.png differ diff --git a/pyproject.toml b/pyproject.toml index b105c3f..3b65805 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "unredact" version = "0.1.0" -description = "AI-powered redaction analysis for Epstein case files" +description = "AI-powered redacted PDF analysis using computer vision, constraint solving, and LLM reasoning" requires-python = ">=3.12" dependencies = [ "fastapi>=0.115",