Scan the live sites for widgets

Use this to see which legacy widgets each real page uses. How matches are made is in How the page scan works, and the tables and scripts are in Page scan data.

Run every command from apps/migration-tracker.

Before you startCopy Anchor Link

• Apply migrations 0010_page_widget_scan.sql and 0011_widget_mapping.sql (supabase db push).
• Have the page list: run bun run scrape-sitemaps and bun run import-pages, or wait for the daily refresh.
• Load the legacy widget names with bun run import-backlog. The scan reads them to label widgets.
• Set SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY in .env.local. fetch-pages reads the page list from the database and import-scans writes to it.

Fetch the pagesCopy Anchor Link

bun run fetch-pages afl carlton

Name one or more site ids from data/sites.json. Options:

• --type news-article fetches only that page type.
• --limit 50 stops after 50 pages. Pages come in id order, so a limit gives a mixed sample.
• --refetch fetches pages that were already saved.

Each site is fetched one page at a time with a 500 ms pause, and the sites run in parallel. The HTML is saved gzipped to data/html/<site>/<id>.html.gz, and every attempt is added to that folder's manifest.jsonl with its status, size and hash. These folders are gitignored.

The command is safe to stop and run again. Pages already saved, or that answered 404 or 410, are skipped. Failures such as a 503 or a timeout are tried again. A 429 or 5xx is retried twice within a run, and some sites (Gold Coast, Brisbane) throttle, so a later pass picks up what was missed.

Parse the pagesCopy Anchor Link

bun run parse-pages afl carlton

This reads the saved HTML and data/crosswalk.json, and writes data/parsed/<site>.jsonl. It needs no database and takes seconds. It prints, for each site, the instances by kind and confidence, any markers it could not resolve, the widgets whose template is still ambiguous and the template guesses.

Import the resultsCopy Anchor Link

bun run import-scans afl carlton

This loads the legacy widgets, fetch results, scan run, pages, widget instances and anchors. When a site's new run is complete it replaces that site's previous run (--keep keeps the older runs). A run that stops part way is never shown in the tracker.

Load the widget mappingCopy Anchor Link

bun run import-widget-mapping
bun run import-widget-mapping --create-items

This reads migration-plan/output/widget-mapping.csv, or the path you give as the last argument, and replaces all mapping rows. With --create-items it also creates a discovered widget item for each target widget, except targets still marked for separate review. A widget item that already has the same id is linked and left unchanged. Running it again only updates the name and mapping details of items it created and never changes a status.

Read the resultsCopy Anchor Link

• Migration > Items: each page's best-guess template and widget count. Open a page for its widgets, template evidence and anchors.
• Widgets > Legacy widgets: every legacy widget with its decision and usage. Open one to see where it is used across the network.
• An item page: "Seen on scanned pages" lists the pages its legacy widgets appear on.

Refresh a siteCopy Anchor Link

Fetch again with --refetch, parse, then import:

bun run fetch-pages afl --refetch
bun run parse-pages afl
bun run import-scans afl

Rebuild the crosswalkCopy Anchor Link

Do this when the legacy code changes:

bun run build-crosswalk

It reads checkouts named telstra-legacy-aflm-web and telstra-legacy-aflc-web next to this repo. To use another parent folder, pass its path with a trailing slash. It rewrites data/crosswalk.json, which is committed and records the legacy commits it was built from. Run parse-pages and import-scans again afterwards.

Run against another databaseCopy Anchor Link

Every script that touches the database reads SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY. To point one at a different database, keep those in their own file (the .env.* pattern is gitignored) and pass it in:

bun --env-file=.env.prod run import-scans afl

Check the target first:

bun --env-file=.env.prod -e 'console.log(new URL(process.env.SUPABASE_URL).host)'

The downloaded HTML and parsed results stay on the machine that fetched them, so import from that machine.

If something goes wrongCopy Anchor Link

• A foreign key error on page_fetches or page_scans: a scanned page is missing from the database. Run bun run import-pages, which only adds missing pages.
• Many 503 responses: run fetch-pages again later. Throttled sites can take a few passes.
• Widgets show as unmapped: run import-widget-mapping. Only the templates in the mapping file have a decision.