mirror of
https://github.com/PR0M3TH3AN/unredact.git
synced 2026-10-27 08:40:29 +00:00
readme + pyproject
This commit is contained in:
1 parent
3dfa8b3ce8
commit
1d9e877c2f
6 files changed
+212
-1
No files matched your search
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Alex
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -0,0 +1,190 @@
|
|||||||
|
# Unredact
|
||||||
|
|
||||||
|
**Reveal what's hidden in redacted PDFs.**
|
||||||
|
|
||||||
|
Unredact uses computer vision, font-aware constraint solving, and LLM reasoning to figure out what text is hiding under those black bars. Upload a PDF, and it will detect redactions, calculate exactly which strings could fit based on pixel-width constraints, and let you visually verify guesses with a live overlay.
|
||||||
|
|
||||||
|
<!-- TODO: Replace with actual screenshots -->
|
||||||
|

|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
- **Detects redactions** automatically using computer vision, or manually by clicking
|
||||||
|
- **Solves for hidden text** by finding every string that fits the exact pixel width of a redaction, accounting for font metrics and kerning
|
||||||
|
- **Ranks results with AI** using Claude to score candidates by contextual fit with surrounding text
|
||||||
|
- **Lets you verify visually** by overlaying guessed text in green on the original document — if the characters align, the guess fits
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
Unredact combines three techniques:
|
||||||
|
|
||||||
|
1. **Computer vision** — OCR extracts visible text with character-level bounding boxes. OpenCV detects black rectangles. A pixel-level font matcher identifies the document's typeface and size.
|
||||||
|
|
||||||
|
2. **Constraint solving** — Using the detected font's exact character widths (including kerning pairs), a parallel branch-and-bound solver written in Rust enumerates every string that fits the redaction's pixel width within a configurable tolerance.
|
||||||
|
|
||||||
|
3. **LLM validation** — Claude reads the surrounding text context and scores each candidate for plausibility, then results are ranked by a composite of width fit and contextual score.
|
||||||
|
|
||||||
|
```
|
||||||
|
PDF ──→ Rasterize ──→ OCR ──→ Font Detection ──→ Redaction Detection
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
Width Tables ──→ Constraint Solver (Rust)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Candidates ──→ LLM Validation ──→ Ranked Results
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Visual Overlay
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- Python 3.12+
|
||||||
|
- Rust toolchain ([rustup.rs](https://rustup.rs))
|
||||||
|
- System packages:
|
||||||
|
- **poppler** — PDF rendering (`libpoppler-cpp-dev` on Debian/Ubuntu, `poppler` on Arch/macOS)
|
||||||
|
- **Tesseract** — OCR engine (`tesseract-ocr` on Debian/Ubuntu, `tesseract` on Arch/macOS)
|
||||||
|
- **fontconfig** — Font lookup (`fontconfig` — usually preinstalled on Linux)
|
||||||
|
- An [Anthropic API key](https://console.anthropic.com/) for LLM features
|
||||||
|
|
||||||
|
### Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/Alex-Gilbert/unredact.git
|
||||||
|
cd unredact
|
||||||
|
|
||||||
|
# Create Python virtualenv and install dependencies
|
||||||
|
python -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -e .
|
||||||
|
|
||||||
|
# Build the Rust constraint solver
|
||||||
|
make build-solver
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configure
|
||||||
|
|
||||||
|
On first run, you'll be prompted for your Anthropic API key, which gets saved to `.env`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make run
|
||||||
|
# Enter your Anthropic API key (from console.anthropic.com): sk-ant-...
|
||||||
|
```
|
||||||
|
|
||||||
|
Or create `.env` manually:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo "ANTHROPIC_API_KEY=sk-ant-your-key-here" > .env
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make run
|
||||||
|
```
|
||||||
|
|
||||||
|
Open [http://localhost:8000](http://localhost:8000) in your browser.
|
||||||
|
|
||||||
|
This starts both the Python web server (port 8000) and the Rust constraint solver (port 3100). Press `Ctrl+C` to stop everything.
|
||||||
|
|
||||||
|
## Usage guide
|
||||||
|
|
||||||
|
### 1. Upload a PDF
|
||||||
|
|
||||||
|
Drag and drop a redacted PDF onto the page, or use the file picker. The tool will automatically run OCR on every page and detect redactions and fonts.
|
||||||
|
|
||||||
|
### 2. Select a redaction
|
||||||
|
|
||||||
|
Click on a detected redaction (highlighted on the page). A panel opens with analysis details and solve options.
|
||||||
|
|
||||||
|
### 3. Choose a solve mode
|
||||||
|
|
||||||
|
| Mode | What it searches | Use case |
|
||||||
|
|------|-----------------|----------|
|
||||||
|
| **Name** | First names or last names from name dictionaries | Redacted names |
|
||||||
|
| **Full Name** | Two-word combinations (first + last) | Full name redactions |
|
||||||
|
| **Email** | Common email addresses | Redacted email addresses |
|
||||||
|
| **Word** | English nouns and adjectives from dictionary | General text redactions |
|
||||||
|
| **Enumerate** | All possible character combinations | Short redactions, anything goes |
|
||||||
|
|
||||||
|
### 4. Configure and solve
|
||||||
|
|
||||||
|
- Set the **character set** (lowercase, uppercase, capitalized)
|
||||||
|
- Adjust **tolerance** if results are too narrow or too broad
|
||||||
|
- Add **known characters** if you can tell what the first or last letter is
|
||||||
|
- For word mode, toggle **plural only** or adjust **vocabulary size**
|
||||||
|
- Click **Solve** — results stream in as they're found
|
||||||
|
|
||||||
|
### 5. Validate with AI
|
||||||
|
|
||||||
|
Click **Validate** to have Claude score each candidate based on the surrounding text context. Results are re-ranked by a composite score combining width fit and contextual plausibility.
|
||||||
|
|
||||||
|
### 6. Verify visually
|
||||||
|
|
||||||
|
Select a result to see it overlaid in green on the original document. If the green text aligns character-for-character with the surrounding visible text, the guess is a strong match. You can fine-tune font, size, position, and character spacing manually.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
Unredact runs as two local services:
|
||||||
|
|
||||||
|
- **Python server** (FastAPI, port 8000) — Handles PDF processing, OCR, font detection, redaction detection, LLM calls, and serves the web UI
|
||||||
|
- **Rust solver** (Axum, port 3100) — Runs the parallel constraint solver, streams results back via SSE
|
||||||
|
|
||||||
|
The frontend is vanilla JavaScript with no build step. It uses canvas for rendering and SSE for real-time result streaming.
|
||||||
|
|
||||||
|
```
|
||||||
|
Browser ◄──── SSE ────► FastAPI (Python)
|
||||||
|
│
|
||||||
|
├── OCR (Tesseract)
|
||||||
|
├── Font detection (pixel matching)
|
||||||
|
├── Redaction detection (OpenCV)
|
||||||
|
├── LLM validation (Claude API)
|
||||||
|
│
|
||||||
|
└──── HTTP ────► Constraint Solver (Rust)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Project structure
|
||||||
|
|
||||||
|
```
|
||||||
|
unredact/
|
||||||
|
├── unredact/ # Python package
|
||||||
|
│ ├── app.py # FastAPI server
|
||||||
|
│ ├── pipeline/ # Processing modules (OCR, fonts, solver, LLM, etc.)
|
||||||
|
│ ├── static/ # Frontend (HTML, CSS, JS)
|
||||||
|
│ └── data/ # Bundled dictionaries and word lists
|
||||||
|
├── solver_rs/ # Rust constraint solver
|
||||||
|
│ └── src/ # Axum server + DFS solver
|
||||||
|
├── tests/ # Test suite
|
||||||
|
├── scripts/ # Data build scripts
|
||||||
|
└── Makefile # Build, run, test automation
|
||||||
|
```
|
||||||
|
|
||||||
|
### Useful commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make run # Start everything (foreground, Ctrl+C to stop)
|
||||||
|
make app # Start in background
|
||||||
|
make stop # Stop background services
|
||||||
|
make status # Check what's running
|
||||||
|
make logs # Tail logs
|
||||||
|
make test # Run test suite
|
||||||
|
make debug # Run with font debug images saved to debug/
|
||||||
|
make build-word-lists # Rebuild noun/adjective dictionaries from WordNet
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Tests require the Rust solver to be running
|
||||||
|
make test
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
[MIT](LICENSE)
|
||||||
Whitespace-only changes.
Binary file not shown.
|
After Width: | Height: | Size: 672 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1006 KiB |
+1
-1
@@ -1,7 +1,7 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "unredact"
|
name = "unredact"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
description = "AI-powered redaction analysis for Epstein case files"
|
description = "AI-powered redacted PDF analysis using computer vision, constraint solving, and LLM reasoning"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"fastapi>=0.115",
|
"fastapi>=0.115",
|
||||||
|
|||||||
Reference in new issue
Block a user