Design and rehearse regional write failover

Application background

Customers place orders through an application running in one geographic region. A second region holds a copied database so it can take over if the first becomes unavailable. When copying happens after the original write, the second region can briefly be behind.

Suppose a customer sees Order saved just before the first region fails. If the second region has not received that order, sending traffic there restores access but may lose an order the customer was told was safe.

Example walkthrough

Failover means moving service to the replacement region. Recovery time and acknowledged-data loss are separate measurements. The exercise makes their trade-off explicit.

Sizing that affects this decision

At 3,000 writes/s, a replica that is 20 seconds behind may be missing roughly 60,000 recent writes. That is possible exposure, not an observed loss count. The requested 15-minute recovery time does not by itself satisfy a zero-data-loss objective.

These are exercise assumptions. The estimation reference explains the units and approximations. They do not establish the local demo's measured capacity.

Your assignment

Deliver: Write and rehearse the regional recovery procedure. Show when writes are acknowledged, how the old writer is stopped, which records the replacement has and what data may be lost.

Required behavior: Only one writer epoch may commit orders. A failover publishes a new epoch after the previous writer is fenced. Acknowledged means the durability policy has been met. Promotion must report the last recoverable commit position.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/regional_failover.py

Supplied file: examples/architecture-starts/regional_failover.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for regional-failover. No AWS resources are created."""
authority={'epoch':7,'region':'east'}; orders=[]
def commit(region,epoch,order):
    if (region,epoch)!=(authority['region'],authority['epoch']): return 'fenced'
    orders.append(order); return 'committed'
print(commit('east',7,'order-1'))
authority.update(epoch=8,region='west')
print(commit('east',7,'late-order'))
print(commit('west',8,'order-2'))
print(orders)

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

committed
fenced
committed
['order-1', 'order-2']

Set up your implementation workspace

Create work/regional-failover/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Name the failure and durability boundary

Distinguish one-AZ loss, whole-region loss and network partition. State where a successful response’s write is durable. If cross-region replication is asynchronous, publish the loss exposure and choose whether to wait for recovery rather than promote missing data.

2. Implement a real writer fence

The starting program illustrates the rule, not a distributed authority. In the cloud design, specify how the database rejects old-writer commits or how access is definitively revoked before promotion. A process that merely reads a cached region flag can continue writing during a partition.

3. Write the timed runbook

Record detection evidence, who makes promotion decisions, how to stop writes, checkpoint comparison, promotion command, read/write validation and traffic change. Keep each step restartable and record its completion. Do not route traffic until the new writer can enforce the chosen epoch.

4. Reconcile and return safely

After the old region returns, keep it fenced. Compare acknowledged operation identities with recovered state before rebuilding it as a replica. Plan failback as another authority transfer. An automatic DNS preference must not recreate two writers.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

AWS routing and database replication solve different parts of failover. The diagram deliberately separates them. If zero loss is mandatory, select and verify a synchronous cross-region durability design and accept its latency/availability tradeoff instead of relabeling an asynchronous replica.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Choose between regional durability and regional write availability

If region A acknowledges before B has a durable copy, A can fail during that gap. Zero acknowledged loss requires that the chosen acknowledgement boundary already spans the failure being promised.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Present two concrete choices. With asynchronous replication, local acknowledgement stays fast but failover can lose the unreplicated suffix. With synchronous cross-region durability, acknowledgement waits for the required remote durable copy and partitions can block writes. Specify the actual protocol and failure assumptions, not just two database icons. Promotion also needs storage-enforced writer fencing. Do not infer either contract from a product name or DNS failover.

Walk through the result. Use a constructed 70 ms cross-region round trip and a local 20 ms write target. Explain why a write that waits for that remote acknowledgement cannot retain the 20 ms target on this path. At 3,000 writes/s and 20 seconds of measured lag, up to 60,000 writes may be exposed in the asynchronous scenario. Record which trade the product accepts.

Product rejects both any data loss and the latency cost of synchronous regional durability. Write a decision record with the failure cases and measurable options. No infrastructure diagram can make contradictory guarantees simultaneously true.