[ci] Add a workflow to regenerate visual baselines on the CI runner

The visual-regression job pixel-compares tests/visual/baselines.json on
ubuntu-latest, but the committed baselines were captured on developer machines.
Chromium builds and font fallbacks differ between environments — the token
stack's "Inter"/"Fira Code" are not shipped or installed anywhere, so every
machine renders its own system fallback — and the kitchen-sink default-theme
diff drifted past MAX_DIFF_RATIO on CI while passing 13/13 locally. ci.yml
documents exactly this mismatch as the reason the job is quarantined from
blocking deploys.

The CI diff artifact confirms it: baseline and actual render identically except
for ghosted/offset text in the high-contrast lower half — font-metric drift,
not a design change. (The dark upper half diffs too little to trip the 0.2
pixelmatch threshold, which is why only the default theme fails.)

Fix the class of problem instead of the instance: capture baselines in the SAME
runner image, browser install and font stack that the comparison job uses.
workflow_dispatch only — regenerating baselines is an explicit decision made
when a visual change is intentional, never a side effect of a push. The
workflow re-runs the comparison against its own output before committing, so a
bad capture cannot land.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LHH3wPBsoruwCaJo8Rpf6a
This commit is contained in:
thePR0M3TH3ANandClaude Fable 5 committed 2026-07-30 11:11:14 -04:00
1 parent f48150ed6b
commit 31be84ba82
1 file changed
+86
@@ -0,0 +1,86 @@
# Regenerate the visual-regression baselines IN the environment that compares
# them.
#
# The baselines in tests/visual/baselines.json are pixel-compared on
# ubuntu-latest CI, but they were historically captured on a developer machine.
# Chromium builds and font fallbacks differ between the two ("Inter"/"Fira
# Code" are not shipped or installed anywhere, so every environment renders its
# own system fallback), so the kitchen-sink diff drifted past MAX_DIFF_RATIO on
# CI while passing locally — ci.yml even documents the mismatch as the reason
# visual-regression must not block deploys. Capturing the baselines on the same
# runner image, browser install and font stack as the comparison removes the
# environment variable from the equation entirely.
#
# Deliberately workflow_dispatch ONLY: baselines are a review artifact, and
# regenerating them must be an explicit human decision made when a visual
# change is intentional — never a side effect of a push.
name: Update visual baselines
on:
workflow_dispatch:
permissions:
contents: write
jobs:
update-baselines:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Check out repository
uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Set up Node.js 22
uses: actions/setup-node@v5
with:
node-version: 22
# Mirror the visual-regression job's caches so the regenerated baselines
# come from the same dependency tree the comparison job will use.
- name: Cache npm directory
uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-node-22-npm-${{ hashFiles('package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-22-npm-
- name: Cache node_modules
id: node-modules-cache
uses: actions/cache@v4
with:
path: node_modules
key: ${{ runner.os }}-node-22-modules-${{ hashFiles('package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-22-modules-
- name: Install dependencies
if: steps.node-modules-cache.outputs.cache-hit != 'true'
run: npm ci
- name: Install Playwright browsers
run: npx playwright install --with-deps chromium
- name: Regenerate baselines
run: npm run test:visual:update
- name: Verify the fresh baselines pass in this environment
run: npx playwright test --project=visual
- name: Commit updated baselines
run: |
if git diff --quiet -- tests/visual/baselines.json; then
echo "Baselines unchanged; nothing to commit."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add tests/visual/baselines.json
git commit -m "test(visual): regenerate baselines on ubuntu-latest
Captured by the update-visual-baselines workflow so the baseline environment
matches the visual-regression job that compares against them."
git push