Page scan data
Reference: the scan commands, the tables and views they fill, and the values each column can take.
How to run the scan is in Scan the live sites for widgets. How matches are made is in How the page scan works.
Commands
Run from apps/migration-tracker. Sites are the ids in data/sites.json; afl is matched against aflm-web and every other site against aflc-web.
| Command | Reads | Writes | Database |
|---|---|---|---|
fetch-pages <site...> [--type t] [--limit n] [--refetch] | page items | data/html/<site>/ | read |
build-crosswalk [parent-folder/] | the two legacy repos | data/crosswalk.json | none |
parse-pages <site...> [--limit n] | data/html, data/crosswalk.json | data/parsed/<site>.jsonl | none |
import-scans <site...> [--keep] | data/parsed, data/html manifests, data/crosswalk.json | the scan tables | write |
import-widget-mapping [--create-items] [csv] | the mapping CSV | legacy_template_mappings, items | write |
Tables
legacy_widgets: one row per legacy widget folder.
| Column | Meaning |
|---|---|
id | aflm:<folder> or aflc:<folder> |
site | aflm or aflc |
folder | the widget folder, or a React path for shared components |
display_name, tech_era, api_dependency, data_pattern | from legacy_folders; aflm only |
templates | the folder's template files |
has_script | whether the folder has JavaScript |
page_fetches: the latest fetch per page item (item_id, fetched_at, status, final_url, bytes, sha1, content_type, error).
scan_runs: one per site import (id, site, crosswalk with the legacy commits, pages, started_at, finished_at). Views read only runs with finished_at set.
page_scans: one per page per run.
| Column | Meaning |
|---|---|
item_id, run_id, scanned_at, html_sha1 | which page, which run |
signals | JSON: title, canonical, ogType, bodyClass, mainClass, mainChildClass, articleVariants, templateEvidence, templateAlternatives |
template_guess, template_confidence, template_method | the template guess; confidence high, medium, low; method markup, url-path, shell-default |
instance_count, widget_count, anchor_count, hash_links | totals and the page's own # links |
page_widgets: one per marker or React mount, in document order (scan_id, ord).
| Column | Values |
|---|---|
source | data-widget, react |
kind | widget, react, generic, shared, unresolved |
confidence | high, medium, low, or null |
method | declared, react-class, js-handler, folder-name |
legacy_widget_id, file | the matched widget and template (file is null when ambiguous) |
region | header, main, sidebar, footer, unknown |
parent_ord, element_id, anchor | the enclosing widget, the element id, the anchor id |
data_script, component_name, data_title, data_template | attributes on the element |
evidence, candidates | JSON: why it matched, and the other templates still possible |
page_anchors: scan_id, anchor_id, widget_ord (null when attached to no widget).
legacy_template_mappings: one per legacy template, keyed by legacy_path and legacy_name.
| Column | Values |
|---|---|
legacy_widget_id, template | the widget and the template path inside its folder |
target_widget | the widget it becomes, or null |
disposition | merge, own, defer, drop, out-of-family |
wave, notes | free text |
status | proposed, accepted, provisional, migrated |
item_id | the item for the target widget |
template_item_id | an existing item already tracking exactly this template |
Views
| View | Gives |
|---|---|
latest_page_scans | the newest finished scan per page |
page_widget_usage | real widgets on pages from the latest scans |
legacy_widget_usage | pages and instances per legacy widget |
legacy_widget_summary | each legacy widget with its mapping, Pulse usage and scanned usage |
item_page_usage | the pages an item's legacy widgets appear on, exact or possible |
legacy_widget_page_usage, legacy_widget_site_usage, legacy_widget_type_usage, legacy_widget_file_usage | one widget's usage by page, site, page type and template |
scan_site_coverage | pages and scanned pages per site |
An item_page_usage match is exact when the page's template file equals the mapped template, and possible when the template is unknown or sits in a partial shared by the folder's templates.