diff --git a/docs/plans/2026-02-18-mode-charset-restructuring-design.md b/docs/plans/2026-02-18-mode-charset-restructuring-design.md new file mode 100644 index 0000000..95b94bf --- /dev/null +++ b/docs/plans/2026-02-18-mode-charset-restructuring-design.md @@ -0,0 +1,94 @@ +# Mode/Charset Restructuring + +## Problem + +The current UI exposes implementation details as user-facing concepts. The mode dropdown (enumerate/dictionary/both/emails) maps to backend strategies, not user intent. The charset dropdown has 7 options including "Full Name" and "FULL NAME" which are really modes, not charsets. The "filter prefix/suffix" fields have a confusing name. The overall UX doesn't match how users think about redaction solving. + +## Solution + +Restructure into orthogonal **mode** (what to search) and **charset** (how text is cased) dimensions. Rename filter prefix/suffix to "known start"/"known end". + +## Modes + +| Mode | Value | Source | Description | +|------|-------|--------|-------------| +| Name (default) | `name` | `associate_first_names.txt` + `associate_last_names.txt` | Single-word associate names: first names, last names, nicknames | +| Full Name | `full_name` | Cartesian product first x last + `associates.json` variants | Two-word names: "John Doe", "B. Clinton" | +| Email | `email` | `emails.txt` | Associate email addresses | +| Enumerate | `enumerate` | Rust DFS backend | Brute-force character enumeration | + +"Dictionary" mode is removed entirely. The user-uploaded dictionary store (`DictionaryStore`) and its API endpoints are removed. + +## Charsets + +| Value | Display | Effect | +|-------|---------|--------| +| `lowercase` | lowercase | All lowercase | +| `uppercase` | UPPERCASE | All uppercase | +| `capitalized` | Capitalized | First letter of each word uppercase | + +For **name** and **full name** modes: charset determines casing applied to stored lowercase names. + +For **enumerate** mode: charset determines the character set sent to the Rust backend. `lowercase` = `a-z`, `uppercase` = `A-Z`, `capitalized` = `a-zA-Z` + space with the capitalize constraint. + +For **email** mode: charset is ignored (emails matched as-is in their stored casing). + +## Known Start / Known End (renamed from filter prefix/suffix) + +These represent known characters at the start/end of the redacted text. The gap width measures only the unknown portion between them. + +**Dictionary modes (name, full_name, email):** +- Filter candidates to those starting with known_start and/or ending with known_end (case-insensitive) +- Measure only the unknown portion (after stripping known_start/known_end) against the gap width +- Use the last char of known_start as left kerning context; first char of known_end as right kerning context +- Display the full candidate text (e.g., "joe" not "oe") + +**Enumerate mode:** Unchanged Rust behavior — known_start/known_end prepended/appended to result text before word/name filter check. + +## Backend Changes + +### New function: `solve_name_dictionary()` + +In `dictionary.py`: +- Loads associate first + last names, combines and deduplicates +- Applies casing based on charset +- Filters by known_start/known_end +- Measures the unknown portion against gap width with kerning +- Returns sorted `SolveResult` list + +### Updated: `solve_full_name_dictionary()` + +- Adds `casing` parameter (replaces `uppercase_only` boolean) +- Adds `known_start`/`known_end` support + +### Updated: `app.py` solve handler + +Dispatch by mode: +- `name` -> `solve_name_dictionary()` +- `full_name` -> `solve_full_name_dictionary()` +- `email` -> `solve_dictionary()` with email list +- `enumerate` -> existing Rust backend + +Remove: `DictionaryStore`, dictionary API endpoints, `dictionary`/`both` mode branches. + +### Updated: `SolveRequest` + +- `mode`: "name" | "full_name" | "email" | "enumerate" +- `hints.charset`: "lowercase" | "uppercase" | "capitalized" +- Rename `filter_prefix` -> `known_start`, `filter_suffix` -> `known_end` +- Keep `word_filter` (used by enumerate mode only) + +## Frontend Changes + +- **Mode dropdown**: Name (default), Full Name, Email, Enumerate +- **Charset dropdown**: lowercase, UPPERCASE, Capitalized +- **Filter dropdown**: hidden unless enumerate mode is selected +- **Rename**: "Filter prefix" -> "Known start", "Filter suffix" -> "Known end" +- Remove unused charset options: full_name_capitalized, full_name_caps, alpha, alphanumeric + +## What Gets Removed + +- `DictionaryStore` class and its API endpoints (`/api/dictionary` POST/GET/DELETE) +- `dictionary` and `both` mode values +- Charset values: `full_name_capitalized`, `full_name_caps`, `alpha`, `alphanumeric` +- The "Filter" label in the HTML dropdown (renamed to "Word filter" for clarity when visible) diff --git a/docs/plans/2026-02-18-mode-charset-restructuring-impl.md b/docs/plans/2026-02-18-mode-charset-restructuring-impl.md new file mode 100644 index 0000000..0edab65 --- /dev/null +++ b/docs/plans/2026-02-18-mode-charset-restructuring-impl.md @@ -0,0 +1,679 @@ +# Mode/Charset Restructuring — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Restructure UI modes (name/full_name/email/enumerate) and charsets (lowercase/uppercase/capitalized), rename filter prefix/suffix to known start/end, fix dictionary mode width measurement with known start/end. + +**Architecture:** Frontend dropdowns are simplified. Backend dispatches by mode to the appropriate solver function. New `solve_name_dictionary()` for single-word name matching. Existing functions updated for casing and known_start/known_end. DictionaryStore removed. + +**Tech Stack:** Python, FastAPI, HTML/JS, PIL/Pillow + +--- + +### Task 1: Add `solve_name_dictionary()` with TDD + +**Files:** +- Modify: `unredact/pipeline/dictionary.py` +- Create: `tests/test_name_dictionary.py` + +**Step 1: Write the failing test** + +Create `tests/test_name_dictionary.py`: + +```python +"""Tests for dictionary-based single-name solving.""" + +from unittest.mock import MagicMock, patch + +from unredact.pipeline.dictionary import solve_name_dictionary +from unredact.pipeline.solver import SolveResult + + +def _mock_font(width_map: dict[str, float]) -> MagicMock: + """Create a mock font that returns widths from a map.""" + font = MagicMock() + font.getlength.side_effect = lambda text: width_map.get(text, len(text) * 7.0) + return font + + +class TestSolveNameDictionary: + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_basic_lowercase(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["john", "jane"] + mock_lasts.return_value = ["doe", "smith"] + + font = _mock_font({"john": 30.0, "jane": 28.0, "doe": 20.0, "smith": 32.0}) + + results = solve_name_dictionary(font, 30.0, 1.0) + texts = [r.text for r in results] + assert "john" in texts + assert "doe" not in texts # 20.0 is outside tolerance + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_uppercase_casing(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["john"] + mock_lasts.return_value = [] + + font = _mock_font({"JOHN": 35.0}) + + results = solve_name_dictionary(font, 35.0, 1.0, casing="uppercase") + assert len(results) == 1 + assert results[0].text == "JOHN" + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_capitalized_casing(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["john"] + mock_lasts.return_value = [] + + font = _mock_font({"John": 32.0}) + + results = solve_name_dictionary(font, 32.0, 1.0, casing="capitalized") + assert len(results) == 1 + assert results[0].text == "John" + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_known_start_filters_and_strips(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["joe", "john", "bob"] + mock_lasts.return_value = [] + + # Gap width is for "oe" (the unknown part after "j") + font = _mock_font({"oe": 14.0, "ohn": 21.0}) + + results = solve_name_dictionary( + font, 14.0, 1.0, known_start="j", + ) + texts = [r.text for r in results] + assert "joe" in texts # "j" + "oe", "oe" width matches + assert "bob" not in texts # doesn't start with "j" + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_known_end_filters_and_strips(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["johnson", "jackson"] + mock_lasts.return_value = [] + + # Gap width is for "john" (the unknown part before "son") + font = _mock_font({"john": 30.0, "jack": 28.0}) + + results = solve_name_dictionary( + font, 30.0, 1.0, known_end="son", + ) + texts = [r.text for r in results] + assert "johnson" in texts + assert "jackson" not in texts # doesn't end with "son" + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_dedup_first_and_last(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["lee"] + mock_lasts.return_value = ["lee"] # same name in both lists + + font = _mock_font({"lee": 20.0}) + + results = solve_name_dictionary(font, 20.0, 1.0) + assert len(results) == 1 # no duplicates + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_sorted_by_error(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["john", "jane"] + mock_lasts.return_value = [] + + font = _mock_font({"john": 30.5, "jane": 30.0}) + + results = solve_name_dictionary(font, 30.0, 1.0) + assert len(results) == 2 + assert results[0].error <= results[1].error + + @patch("unredact.pipeline.dictionary._get_associate_firsts") + @patch("unredact.pipeline.dictionary._get_associate_lasts") + def test_context_chars(self, mock_lasts, mock_firsts): + mock_firsts.return_value = ["john"] + mock_lasts.return_value = [] + + font = _mock_font({ + "": 45.0, + "<": 5.0, + ">": 5.0, + }) + + results = solve_name_dictionary( + font, 30.0, 5.0, left_context="<", right_context=">", + ) + # Width = 45 - 5 - 5 = 35, error = |35-30| = 5, within tolerance + assert len(results) == 1 + assert results[0].text == "john" +``` + +**Step 2: Run test to verify it fails** + +Run: `python -m pytest tests/test_name_dictionary.py -v` +Expected: FAIL — `ImportError: cannot import name 'solve_name_dictionary'` + +**Step 3: Implement `solve_name_dictionary`** + +Add to `unredact/pipeline/dictionary.py` after `solve_full_name_dictionary`: + +```python +def solve_name_dictionary( + font: ImageFont.FreeTypeFont, + target_width: float, + tolerance: float = 0.0, + left_context: str = "", + right_context: str = "", + casing: str = "lowercase", + known_start: str = "", + known_end: str = "", +) -> list[SolveResult]: + """Match single associate names against target width. + + Loads first and last name lists, applies casing, filters by + known_start/known_end, and measures the unknown portion against + the target width. + """ + from unredact.pipeline.word_filter import ( + _get_associate_firsts, + _get_associate_lasts, + ) + + firsts = _get_associate_firsts() + lasts = _get_associate_lasts() + + # Combine and dedup + seen: set[str] = set() + names: list[str] = [] + for name in firsts + lasts: + if name not in seen: + seen.add(name) + names.append(name) + + results: list[SolveResult] = [] + seen_results: set[str] = set() + + ks_lower = known_start.lower() + ke_lower = known_end.lower() + + for name in names: + # Filter by known start/end (case-insensitive on raw lowercase name) + if ks_lower and not name.startswith(ks_lower): + continue + if ke_lower and not name.endswith(ke_lower): + continue + + # Apply casing to the full name (for display) + if casing == "uppercase": + display = name.upper() + elif casing == "capitalized": + display = name.title() + else: + display = name + + if display in seen_results: + continue + seen_results.add(display) + + # The unknown portion is the name minus known_start and known_end + unknown = name[len(known_start):len(name) - len(known_end) if known_end else len(name)] + # Apply same casing to unknown portion + if casing == "uppercase": + unknown_display = unknown.upper() + elif casing == "capitalized": + # For single word, only first char is upper if at start of word + # But since known_start is stripped, unknown is mid-word: lowercase + unknown_display = unknown.lower() if known_start else unknown.title() + else: + unknown_display = unknown + + if not unknown_display: + continue # entire name is known, nothing to measure + + # Determine kerning context + # If known_start is set, its last char is left context for the unknown part + effective_left = known_start[-1] if known_start else left_context + effective_right = known_end[0] if known_end else right_context + + # Measure width of unknown portion with kerning context + if effective_left or effective_right: + full = effective_left + unknown_display + effective_right + full_len = font.getlength(full) + left_len = font.getlength(effective_left) if effective_left else 0.0 + right_len = font.getlength(effective_right) if effective_right else 0.0 + width = full_len - left_len - right_len + else: + width = font.getlength(unknown_display) + + error = abs(width - target_width) + if error <= tolerance: + results.append(SolveResult(text=display, width=float(width), error=float(error))) + + results.sort(key=lambda r: (r.error, r.text)) + return results +``` + +**Step 4: Run tests to verify they pass** + +Run: `python -m pytest tests/test_name_dictionary.py -v` +Expected: all 8 tests PASS + +**Step 5: Commit** + +```bash +git add unredact/pipeline/dictionary.py tests/test_name_dictionary.py +git commit -m "feat: add solve_name_dictionary for single-word associate name matching" +``` + +--- + +### Task 2: Update `solve_full_name_dictionary()` — add casing and known_start/known_end + +**Files:** +- Modify: `unredact/pipeline/dictionary.py` (update `solve_full_name_dictionary`) +- Modify: `tests/test_full_name_dictionary.py` (update tests) + +**Step 1: Update the function signature and implementation** + +Change `solve_full_name_dictionary` in `unredact/pipeline/dictionary.py`: + +Replace parameter `uppercase_only: bool = False` with: +```python + casing: str = "lowercase", + known_start: str = "", + known_end: str = "", +``` + +Update the casing logic inside the function: +- Replace `if uppercase_only:` checks with casing-based logic: + - `"uppercase"`: `(first + " " + last).upper()` + - `"capitalized"`: `first.title() + " " + last.title()` + - `"lowercase"`: `first + " " + last` + +For variants: + - `"uppercase"`: `variant.upper()` + - `"capitalized"`: `variant.title()` + - `"lowercase"`: `variant` + +Add known_start/known_end filtering: +- Before checking width, filter candidates by starts-with/ends-with +- Measure only the unknown portion (strip known_start from front, known_end from back) +- Display the full name + +**Step 2: Update tests** + +In `tests/test_full_name_dictionary.py`: +- Replace `uppercase_only=True` with `casing="uppercase"` in `test_uppercase_mode` +- Add a test for `casing="capitalized"` (existing default behavior) +- Add a test for `known_start` with full names + +**Step 3: Run tests** + +Run: `python -m pytest tests/test_full_name_dictionary.py tests/test_name_dictionary.py -v` +Expected: all tests PASS + +**Step 4: Commit** + +```bash +git add unredact/pipeline/dictionary.py tests/test_full_name_dictionary.py +git commit -m "feat: update solve_full_name_dictionary with casing and known_start/known_end" +``` + +--- + +### Task 3: Update frontend HTML — modes, charsets, and field names + +**Files:** +- Modify: `unredact/static/index.html` + +**Step 1: Replace the charset dropdown (lines 84-92)** + +Replace the current `` contents with: +```html + + + + +``` + +**Step 3: Rename filter prefix/suffix labels (lines 117-124)** + +Change "Filter prefix" to "Known start" and `id="solve-filter-prefix"` to `id="solve-known-start"`. +Change "Filter suffix" to "Known end" and `id="solve-filter-suffix"` to `id="solve-known-end"`. +Update placeholders: `placeholder="e.g. j"` and `placeholder="e.g. son"`. + +**Step 4: Add id to the filter label/container for show/hide (line 108-116)** + +Wrap the Filter `