Migration tracker

Registry, dashboard and API for migrating afl.com.au to Next.js and Optimizely. It tracks items of different kinds, each moving through its own pipeline of stages:

migrated_unverified marks widgets backfilled from an earlier port that has not been verified. Each item has a track: parity copies the live widget, redesign rebuilds it from the design system. Stages are keyed by kind and track. merged (outside the pipeline, not shown in charts) marks a widget that was combined into a redesign; merged_into points at it.

Stages are rows in the stages table, not enums, so a new kind of tracked work is a new set of stage rows. Fixtures, HARs and baselines stay in git; the tracker only indexes their paths.

SetupCopy Anchor Link

1. Create a Supabase project.

2. Link it and apply the schema and the page seed:

   Copy Text

   cd apps/migration-tracker
   bunx supabase@2.120.0 link --project-ref <project-ref>
   bunx supabase@2.120.0 db push --include-seed

3. Copy .env.example to .env.local and fill it in:

   Variable	Purpose
   SUPABASE_URL	Project URL
   SUPABASE_SERVICE_ROLE_KEY	Server-only key; the app and scripts write with it
   TRACKER_API_TOKEN	Bearer token required by every write endpoint
   TRACKER_URL	Base URL used by scripts/ticket.ts and parity:publish
   JIRA_BASE_URL	e.g. https://<site>.atlassian.net
   JIRA_EMAIL	Account that owns JIRA_API_TOKEN
   JIRA_API_TOKEN	Jira Cloud API token
   JIRA_PROJECT_KEY	Defaults to AFLSF
   JIRA_ISSUE_TYPE	Defaults to Task
   JIRA_FACTORY_ACCOUNT_ID	Software Factory agent account, set as the ticket assignee
   LINEAR_API_KEY	Linear API key; when set, tickets go to Linear, not Jira
   LINEAR_TEAM_KEY	Team the issues are created in, e.g. AFL
   LINEAR_WORKSPACE	Workspace URL slug, for issue links
   LINEAR_PROJECT_ID	Optional project for the issues
   LINEAR_FACTORY_DELEGATE_ID	Software Factory agent user the issue is delegated to
   LEGACY_CMS_BASE_URL, LEGACY_CMS_API_TOKEN	Legacy CMS the content pipeline fetches from
   OPTIMIZELY_CLIENT_ID, OPTIMIZELY_CLIENT_SECRET	Content Management API credentials (also OPTIMIZELY_CMA_URL, OPTIMIZELY_TOKEN_URL)

4. Load the backlog and the seed, then run the app:

   Copy Text

   bun run import-backlog   # Pulse widgets, legacy folders, RFP features from data/backlog
   bun run seed             # pages and items from data/seed.json
   bun run import-backlog   # again, to link features to the newly seeded items
   bun run dev              # https://migration-tracker.afl.localhost via portless

   Both scripts are idempotent. Re-seeding never moves an existing item's status; status only changes through PATCH /api/items/:id.

Scraping site pages (Migration tab)Copy Anchor Link

The Migration tab lists every URL in the sitemaps of the existing sites, as page items. The sites (afl.com.au and the club sites) are configured in data/sites.json; add or fix a site there.

bun run scrape-sitemaps            # all sites, or name some: bun run scrape-sitemaps afl carlton
bun run import-pages               # needs SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY

Per site, sitemapPath (for example "/news-sitemap") names the sitemap path and skips robots.txt; sitemaps takes full URLs. With neither, the scraper reads robots.txt and, if that lists no sitemap, tries /sitemap first, then other common paths. The scraper reads robots.txt, follows sitemap indexes (including gzipped sitemaps), waits 500 ms between requests and identifies itself with an AFL-MigrationTracker user agent. A failing site or sitemap is reported and skipped; the other sites still run, and a failed site keeps its previous data/sitemap/<site>.json. Scrapes are written to data/sitemap/ and are not committed by default.

Each URL is classified by the ordered path rules in src/lib/page-types.ts (news article, match centre, player profile, team page, ...); anything that matches no rule is unclassified and shown on the tab for review. import-pages only inserts pages it has not seen, keyed by site and URL, so re-running it never moves an existing page's status. Pages that later disappear from a sitemap are kept. Apply supabase/migrations/0005_page_kind.sql first (supabase db push). The "Last refreshed" date under the Refresh button needs 0007_sitemap_refresh.sql.

Running on VercelCopy Anchor Link

vercel.json runs GET /api/sync-pages daily (17:00 UTC). It scrapes every site in data/sites.json and inserts new pages straight into the database, the same as the two scripts above but without the JSON files. Set CRON_SECRET in the Vercel project; Vercel Cron sends it as a bearer token. To run it by hand, call it with CRON_SECRET or TRACKER_API_TOKEN, optionally limited to some sites:

curl -H "Authorization: Bearer $TRACKER_API_TOKEN" "https://<app>/api/sync-pages?site=afl,carlton"

The run is capped at 300 s (maxDuration). If the full list outgrows that, call it per site.

Content migrationCopy Anchor Link

The Migration tab also runs a five-step pipeline (fetch, map, normalise, validate, migrate) that moves content from the legacy CMS into Optimizely on Vercel Queues. Apply supabase/migrations/0006_content_migration.sql. See content/docs/content-migration-pipeline.md and content/docs/run-content-migration.md.

DeployingCopy Anchor Link

Vercel project with root directory apps/migration-tracker, framework Next.js, and the env vars above. Every page and route is dynamic, so the build needs no Supabase credentials.

Data modelCopy Anchor Link

• stages(kind, key, position): the pipelines. items(kind, status) references it, so a status can only be a stage of its own kind.
• items: one row per tracked thing; id is a kebab-case slug. meta holds anything kind-specific.
• pages and item_pages: where an item appears (region, position).
• dependencies: e.g. a widget depends on a content type.
• artifacts: fixture | har | baseline | spec | requirements | stories | data_model | mapping | config_instructions | pr | other, with a git path or a uri. The fixture index is the artifacts of kind fixture.
• field_mappings: source to target field mappings, with an optional transform.
• test_runs: parity results per commit, matrix cell (must-watch/1280/light/carl) and check: screenshot, styles, tokens or behaviour:<name> (for example behaviour:hover headline). warn marks a passed result whose screenshot differed within the harness threshold (passed stays true, so it counts as a pass).
• rebuilds: one row per rebuild (trigger, actor from x-tracker-actor, commit, optional Software Factory job, PR and rule id, change-log note, and a ticket reference of type original, child or changelog). Sequence numbers are allocated per item by record_rebuild.
• events: written by a trigger on every status change. The actor is the x-tracker-actor request header (the API sets it from the actor field on PATCH).
• pulse_widgets, item_pulse_widgets: the legacy Pulse CMS widgets with AFL and club usage; one item can replace several Pulse widgets.
• legacy_folders: aflm-web widget source folders, matched to Pulse widgets by display name.
• features, item_features: RFP epics and features, linked to items whose id or legacy_ref matches the feature's mapped widget folder.
• Views: item_summary (items plus page ids and the latest commit's pass/fail counts, counting the newest run per cell and check) and pulse_summary (Pulse widgets plus in_use and the covering item_ids).

RLS is on for every table: anon and authenticated can only read. All writes go through the API, which uses the service role key.

Item writes are atomic. POST /api/items and PATCH /api/items/:id call one database function, save_item, which updates the item and replaces any supplied pages, dependencies and Pulse links in a single transaction. If any part fails (an unknown page, a missing dependency, a stage that is not in the item's pipeline), nothing changes.

Content-model pipelineCopy Anchor Link

Content types reuse the same tables. An item of kind content_type moves through understood (the Pulselive data model is understood), designed (the Optimizely model is designed), implemented and migrated. Its outputs are artifacts: data_model (Excel model), mapping (Excel mappings), requirements (Confluence functional requirements), stories (Jira stories) and config_instructions. Field-level mappings go in field_mappings, and widgets that render the type list it in dependsOn.

To add another kind, insert its stages and start creating items of that kind; no code changes.

APICopy Anchor Link

Responses are JSON with snake_case columns. Request bodies are camelCase and validated with Zod. Writes need Authorization: Bearer $TRACKER_API_TOKEN. Errors are { "error": "..." } with 400 (invalid body or status), 401, 404, 409 or 502 (Jira).

Item shape (also the data/seed.json item shape):

{
  "id": "article-list-sidebar-must-watch",
  "kind": "widget",
  "name": "Article list sidebar ('Must Watch')",
  "type": "element",
  "status": "verified",
  "priority": "high",
  "action": "rebuild",
  "legacyRef": "src/widgets/sites/aflm/article-list/article-list-must-watch.ftl",
  "optimizelyType": "ArticleListElement",
  "themeable": false,
  "jiraKey": null,
  "sfJob": null,
  "owner": null,
  "notes": null,
  "meta": {},
  "pages": [{ "page": "article", "region": "sidebar", "position": 19 }],
  "dependsOn": ["article"],
  "pulseIds": [105]
}

type is one of element | shared-block | page-section | chrome | behaviour | content-type. priority is one of highest | high | medium | low | lowest (Jira's scale) or null; set it on the dashboard or through the API. action is the migration decision from the master widget list: rebuild | create_from_scratch | dont_migrate | combine, or null until decided (apply supabase/migrations/0009_item_action.sql). Set it on the item page or through the API; the explorer filters on it. sfJob is the Software Factory job number; both skills set it when a run starts. Re-seeding keeps both. track switches between parity and redesign; switching moves the status to the first stage of the new track unless the current one exists there. designComponent and figmaUrl are the AFLDS-Web-Library component a redesign follows. pages, dependsOn and pulseIds, when present, replace the existing links.

TicketsCopy Anchor Link

Tickets go to Linear when LINEAR_API_KEY is set and to Jira otherwise. Either way the key (AFLSF-12 or AFL-12) is stored in the item's jira_key.

src/lib/ticket.ts builds the content: title Migrate widget: <name> (<id>), and a description that names the migrate-afl-widget skill, the widget id, legacy ref, Optimizely type, the Pulse widgets it replaces with their usage, its pages and live sample URL, and the definition of done. src/lib/jira.ts renders it as ADF into JIRA_PROJECT_KEY with labels software-factory, widget-migration and tracker-<item-id>, assigned to JIRA_FACTORY_ACCOUNT_ID. src/lib/linear.ts renders it as markdown into the LINEAR_TEAM_KEY team with labels software-factory and widget-migration (created on the team if missing), and delegates it to LINEAR_FACTORY_DELEGATE_ID, which hands it to the agent.

Creation cannot duplicate issues. The route first reserves the item's jira_key with a conditional update, so a second request at the same time gets 409. It then looks for an issue for the item (labelled tracker-<item-id> in Jira, titled … (<item-id>) in Linear) and reuses it, which makes a retry after a failure safe, and only creates one if none exists. If Jira or Linear fails, the reservation is released.

RebuildsCopy Anchor Link

jira_key stays the item's original build ticket: a rebuild never creates a new top-level ticket. POST /api/items/:id/rebuilds always writes the tracker-side change log (a rebuilds row). It references the original ticket (original), or nothing (changelog) when the item has none yet. With track: true it also files a child ticket under the original one (child): a Jira Sub-task labelled software-factory, widget-rebuild, tracker-<item-id> and tracker-<item-id>-rebuild-<n>, or a Linear child issue titled … (<item-id> rebuild <n>). Both are looked up by that label or suffix before creating, so a retry reuses the ticket. track without an original ticket is a 409. If the ticket system fails the change log is kept as original/changelog and the response carries ticket_error.

bun run ticket article-list-sidebar-must-watch --dry-run   # print the ticket
bun run ticket article-list-sidebar-must-watch             # create it via the API

DashboardCopy Anchor Link

Built with shadcn/ui (Base UI, the same base-nova style as packages/ui) and Recharts, themed with the afl.com.au tokens from @afl/config and Inter (src/app/globals.css maps them onto the shadcn variables). Add components with bunx shadcn@4.21.4 add <name> from this folder.

• /: headline numbers, overall progress and charts (items by stage, progress over time from status history, items by type, priority).
• /widgets: the item list with instant search, stage, type, page and priority filters, an editable priority per item and the SF job. Filters live in the URL.
• /items/:id: a stage stepper, priority, SF job, Jira, the parity results grid with screenshot comparison, Pulse widgets, pages, features, dependencies, artifacts, mappings and history.
• /map: a tree of every page, its regions and the widgets placed there, with the Pulse widgets, features and merged widgets linked to each. Hovering or clicking a Pulse widget or feature highlights every widget it links to.
• /pulse: the most used Pulse widgets as a chart, then a searchable, sortable list.
• /docs: the framework docs, built with Fumadocs inside this app's next build. The widget migration overview and the legacy audit live here as markdown in content/docs. The other pages pull in a file from elsewhere in the repo with <include> (both skills, the parity harness and this README), so every doc has one source. src/lib/repo-docs.ts drops an included file's own title and rewrites relative links: to the docs page when the target is a docs page or included in one, otherwise to GitHub. The diagrams are static pages in public/diagrams, generated by python3 scripts/build-diagrams.py public/diagrams. Search is /api/search. To add a page, add a .md or .mdx file with a title and a description (and an <include> if the text lives elsewhere), and list it in that folder's meta.json.

On the dashboard, select widgets and choose "Combine into redesign" to create one redesign item from them. An item page has "Rebuild from designs" (or "Back to parity"), shows the design system component and Figma link for redesigns, and an "Approve design" button once a redesign is reviewed; the approver's name is remembered in the browser. Redesign tickets name the redesign-afl-widget skill and ask for no pixel parity.

Priority changes, combining, approving and track switches from the dashboard go through server actions (src/app/actions.ts) with the service key, so anyone who can open the dashboard can make these changes. Keep the deployment behind Vercel authentication.

Parity resultsCopy Anchor Link

bun run --cwd tools/parity parity:publish uploads the run's screenshots to /api/items/:id/screenshots, posts the results to /api/items/:id/test-runs with the stored screenshot paths in detail.screenshots, and exits with an error if the tracker rejects either.

The item page shows only the latest commit's results, newest per cell and check. Select a result to compare afl.com.au with the build side by side, with a slider, or as a difference image (identical pixels are black). Warnings (yellow !) pass but are shown apart from clean passes, with the yellow-highlighted diff image under the comparison. Screenshots are served through signed URLs that last an hour. Migration 0002_parity_screenshots.sql creates the bucket; each run stores about 4 MB per widget. Migration 0003_priority_and_sf_job.sql adds priority and sf_job. Migration 0004_redesign_track.sql adds tracks, the redesign stages, merged, the design and approval fields, the figma and a11y checks and the combine_items function.

Seed dataCopy Anchor Link

data/seed.json holds the Article page and its 35 widgets from data/legacy-audit.json: earlier ports as migrated_unverified, the rest as discovered. Their themeable flags are provisional (meta.themeableProvisional) until the factory checks each widget's club version. data/backlog/ is the backlog spreadsheet exported to JSON for import-backlog.

ChecksCopy Anchor Link

bun run typecheck
bun run build