Migrate tenant data with a resumable backfill

Application background

A SaaS product stores several companies' records in an old database layout. The team needs a new layout, but customers must keep using the application during the move. Some still use older mobile apps that cannot change immediately.

You will move one company's data at a time. Copying existing records is called a backfill. While that copy runs, new edits still arrive, so the system must know which database is allowed to accept authoritative writes.

Example walkthrough

A cutover is that controlled switch. Copying data and changing write authority are different operations, and the old client contract still needs to work through both.

Your assignment

Deliver: Build a copy process that can resume for each company, compare the old and new records, and control when the new store becomes responsible for writes.

Required behavior: Each tenant has a recorded migration state and one write authority. Backfill, updates and deletions carry monotonically comparable source versions. Reads never resurrect an older row after a newer deletion.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/multi_tenant_migration.py

Supplied file: examples/architecture-starts/multi_tenant_migration.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for multi-tenant-migration. No AWS resources are created."""
target={}
def apply(key,version,value):
    old=target.get(key)
    if old and old['version']>=version: return 'ignored stale/duplicate'
    target[key]={'version':version,'value':value}; return 'applied'
for v,value in [(5,'live edit'),(6,None),(4,'old backfill')]:
    print(v,apply('acme:7',v,value))
print('Target:',target,'visible:',target['acme:7']['value'])

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

5 applied
6 applied
4 ignored stale/duplicate
Target: {'acme:7': {'version': 6, 'value': None}} visible: None

Set up your implementation workspace

Create work/multi-tenant-migration/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Make old and new contracts coexist

Write down the old client request/response shape and the target schema. Add an adapter that preserves old behavior while storing the new representation. Record the last client version requiring the adapter. Do not remove fields merely because the new UI stopped using them.

2. Backfill through the same versioned apply path

Take a consistent snapshot boundary and start change capture without a gap. Persist range checkpoints only after durable target writes. Use a conditional source-version comparison for snapshot rows, live changes and tombstones. Retry a range safely after process death.

3. Reconcile before changing reads

Compare canonical records or range hashes at an aligned source watermark. Account for deletes and null/default conversions. Shadow reads are useful only when you distinguish replication lag from transformation errors. Sample by tenant size and unusual schema values.

4. Transfer write authority once

Drain or fence old writers, apply the final change watermark, then advance the tenant’s writer epoch and routing state. A DNS change alone is not a write fence. Before new-format writes, document whether rollback is still possible or requires reverse transformation.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

DMS can move rows and changes, but application compatibility, semantic transformations and write fencing remain your responsibility. The design uses per-tenant cutover so one problematic customer does not force a global switch.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Cross the point where an old schema cannot represent new writes

The new tag model permits color and organization ownership, while the old model stores only strings. Sending traffic back to old code can silently discard meaning even when both databases are healthy.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Use explicit migration phases: copy, catch up, compatible serving, then target-only writes. Before that last phase, either implement a lossless reverse representation or declare forward repair as the recovery strategy. Fence old writers and retain versioned change records. Keep a per-tenant cutover record so one tenant's target-only write does not accidentally authorize all tenants to cross the boundary. AWS routing changes select servers, while the database migration controller owns data readiness.

Walk through the result. Create tag {name: urgent, color: red} after cutover. Attempt the documented rollback. It must either preserve color through the compatibility adapter or refuse and invoke forward repair. Supply a phase table with permitted writers and recovery action for each phase.

The target accepts data the old schema cannot represent. Mark that first write as an explicit rollback boundary. Design a forward repair path and explain why flipping traffic back would lose meaning even if every server is healthy.