Content migration pipeline
Reference: the five steps that move content from the legacy AFL CMS into Optimizely, the queue topics, run state, limits and environment variables.
The Migration tab runs each page through five steps on Vercel Queues. Run state lives in Supabase, so the tab's progress widget reads the same rows the queue consumers write.
Steps
An item is picked up by step N+1 only once step N is done.
| # | Step | Topic | What it does |
|---|---|---|---|
| 1 | Fetch | migrate.fetch | Pulls the article from the legacy CMS (/content/afl/text/en/<id>, the id taken from the page URL /news/<id>/<slug>) and archives the raw JSON in migration_payloads. Also refreshes the reference data (players, teams, competitions, grounds) once per run. |
| 2 | Map | migrate.map | Resolves the legacy IDs the payload refers to (media, players, ...) to Optimizely keys: manual overrides in migration_overrides first, then the catalogue in migration_reference_entities. An unresolved ID fails the step with the IDs listed; nothing is guessed. |
| 3 | Normalise | migrate.normalise | Converts the payload into the content type's fields and a Visual Builder composition (section, row, column, then RichTextElement and ImageElement components). Body HTML is kept as is, so links survive. A legacy type with no content type in apps/cms fails the step: raise a content-model gap instead of mapping it by hand. |
| 4 | Validate (gate) | migrate.validate | Checks the normalised payload against the Optimizely content type and every composition component type, read-only. Rich text may only use paragraphs, headings, links, basic emphasis, lists and blockquotes; anything else (images, scripts, event handlers) fails the step rather than being dropped. Needs an approval before Migrate. |
| 5 | Migrate | migrate.migrate | Creates or updates the content through the Content Management API and stores the target key and URL. |
A consumer that finishes an item publishes it to the next topic. Only ids travel in queue messages (runId, itemId, step, generation, chain); payloads stay in Supabase.
Only the article legacy type has an Optimizely content type today. Other types stop at Normalise.
Step status
Each run has one row per item and step in migration_steps.
| Status | Means |
|---|---|
pending | Not started, or waiting to retry after a failed attempt (attempts > 0) |
running | A consumer holds it |
done | Succeeded |
error | Out of attempts, or failed with an error retrying cannot fix. last_error has the reason. This is the dead-letter state |
skipped | Not going to run: Migrate in a dry run. Skipped rows are left out of the counts |
Retries, limits and rate limiting
Failed attempts are counted in migration_steps.attempts, not by the queue's delivery count, and retried with exponential backoff (10 s, 20 s, 40 s, up to 5 minutes). Waiting for a concurrency or rate-limit slot, or for reference data, is retried after a short delay and does not count as an attempt.
| Step | Max in flight | Starts per minute | Attempts |
|---|---|---|---|
| Fetch | 5 | 120 | 4 |
| Map | 10 | 600 | 3 |
| Normalise | 10 | 600 | 3 |
| Validate | 3 | 90 | 4 |
| Migrate | 2 | 60 | 5 |
The limits are enforced across all runs by claim_migration_step. They are in stepConfig in src/lib/migration/steps.ts. They are starting values: Optimizely's Content Management API rate limit for our tenant is not confirmed.
A step that crashed mid-run is claimed again after 10 minutes.
Idempotency
- Queue publishes carry the key
<run>:<step>:<generation>:<hash of the item>. Publishing the same attempt twice is deduplicated by the queue; a manual retry bumpsgenerationand is not. - Delivery is at-least-once, so every consumer first claims its row. A message for a
donestep does no work but still publishes the next step, in case the earlier publish was lost. - Migrate addresses Optimizely content by a key derived from the item id (
optimizelyKey). It reads that key first and updates the content if it exists, so a re-run, or a run whose stored target was lost, never creates a duplicate. The key is also kept inmigration_targets.
The approval gate
Validate finishing does not publish to Migrate. One approval per run releases every item that passed Validate; items that pass later go straight on. The approver and time are stored on migration_runs. A dry run stops after Validate: its Migrate rows are skipped and Migrate refuses to run.
Tables
Created by supabase/migrations/0006_content_migration.sql.
migration_runs: one row per run (dry_run,scope,approved_at,approved_by).migration_steps: item x step state:status,attempts,generation,last_error,target_id,target_url, timestamps.migration_payloads: theraw,mappedandnormalisedpayload per run and item, so overlapping runs (a dry run beside a real one) never share payloads. Not readable with the anon key.migration_targets: item to Optimizely key and URL, kept across runs.migration_reference_entities,migration_reference_refreshes: the reference catalogue and, per run, the state of each entity's refresh.migration_overrides: manuallegacy_idto Optimizely key mappings that win over the catalogue.
Environment variables
Set these in the Vercel project, next to the variables in the README.
| Variable | Purpose |
|---|---|
LEGACY_CMS_BASE_URL | Base URL of the legacy AFL CMS API |
LEGACY_CMS_API_TOKEN | Bearer token for the legacy CMS |
OPTIMIZELY_CMA_URL | CMS 13 REST API base on the CMS instance, https://<cms host>/_cms/v1. No default |
OPTIMIZELY_TOKEN_URL | OAuth token endpoint of the same CMS instance. No default |
OPTIMIZELY_CLIENT_ID | Client credentials for the Content Management API |
OPTIMIZELY_CLIENT_SECRET | Client credentials for the Content Management API |
Vercel Queues authenticates with the project's OIDC token on Vercel. Locally, run vercel link and vercel env pull first.
Why Vercel Queues
Vercel Queues is in public beta on all plans. It gives at-least-once delivery, a retry callback for backoff, a 100 MB message limit (ours are a few hundred bytes), 7-day retention and idempotent publishes. It has no built-in dead-letter queue, so the error state is ours (migration_steps.status = 'error'), and its per-topic concurrency control is not part of the trigger configuration, so the limits above are enforced in the database. If Queues' beta limits get in the way, Vercel Workflows (built on Queues) is the fallback: the step functions in src/lib/migration/handlers.ts do not depend on the queue.