Rehearse detection, rollback and service recovery

Application background

A reading-list service saves URLs and runs separate jobs to fetch their titles. An operator changes the title-provider address to an unavailable host. The configuration file is valid, but users stop receiving titles.

You will reproduce this situation in a disposable environment and practice the response. The point is to observe user impact, how the responder discovers it, how the change is reversed and whether useful work actually resumes.

Example walkthrough

Detection time, mitigation time and recovery time describe different moments. Record them separately instead of treating the rollback command as proof that users have recovered.

Your assignment

Deliver: Run one bounded staging incident and record user impact, detection, mitigation and demonstrated recovery. Complete one follow-up repair based on what the exercise revealed.

Required behavior: The drill has a named owner, bounded scope, observable stop condition and independently available rollback path. It produces a factual timeline and one completed repair. No recurring fault injection or production action is installed.

The primary deliverable is the report or operational procedure named above, backed by a reproducible local demonstration. Build the smallest supporting code needed to make that evidence visible.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/the_incident_you_caused_on_purpose.py

Supplied file: examples/architecture-starts/the_incident_you_caused_on_purpose.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for the-incident-you-caused-on-purpose. No AWS resources are created."""
times={'inject':0,'impact':5,'alert':60,'rollback':120,'recovered':210}
print({'detection_after_impact_s':times['alert']-times['impact'],'mitigation_after_alert_s':times['rollback']-times['alert'],'recovery_after_impact_s':times['recovered']-times['impact']})
print('Rollback is a milestone; backlog clear establishes recovery.')

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

{'detection_after_impact_s': 55, 'mitigation_after_alert_s': 60, 'recovery_after_impact_s': 205}
Rollback is a milestone; backlog clear establishes recovery.

Set up your implementation workspace

Create work/the-incident-you-caused-on-purpose/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Prepare a bounded staging drill

Name the exact configuration field, affected worker pool and synthetic workload. Save the previous configuration and verify emergency access before injecting anything. Define the stop condition in observable terms such as oldest-job age or an unexpected environment identifier.

2. Inject one fault and record evidence

Change only the selected staging route to an unavailable fixture host. Record the applied configuration version, first failed job, alert delivery and responder action. Do not add simultaneous failures that make cause and effect impossible to separate.

3. Mitigate and follow the backlog

Restore the prior valid configuration through the independent path. Confirm the version actually applied, then watch useful completions and oldest age until recovery. A successful rollback command is not proof that waiting users have recovered.

4. Complete one repair

Fix the discovered weakness: semantic configuration validation, independent rollback credentials, bounded retries or clearer evidence. Re-run only the relevant bounded scenario to see whether the repair changes the measured outcome. Keep the factual timeline and remaining limits.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

The exercise separates injection, detection, mitigation and recovery. A control plane that shares the failure can block rollback, so the recovery path needs its own availability assumptions.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

The drill exposes a repair that cannot be completed immediately. Name the temporary operating limit and owner, and preserve the evidence rather than declaring the exercise successful solely because the service eventually recovered.

Follow-up scenarios and worked designs

Follow-up 1 · The normal control plane fails

You cannot reach the deployment dashboard. How do you stop the drill?

Worked design and implementation

Use pre-authorized independent recovery access and a time-bounded failure injection. Test its credentials and dependencies beforehand. A stop button inside the failed system is not an independent stop mechanism.

Draw the stop path before injecting failure. The normal deployment UI and its credentials may depend on the system you are about to disrupt. Use a separately accessible recovery identity and an injection that expires automatically. Keep both paths scoped to the disposable exercise environment.

Make the normal dashboard unreachable, then stop the drill through the independent path. Record the dependency chain and elapsed recovery time. If both paths share the same failed identity service, describe that unresolved dependency instead of calling the second button independent.

Revised flow. These are proposed components to implement, not extra services started by the supplied demo.

Follow-up 2 · The fix changes only the alert

The page arrives sooner, but users still wait the same time for backlog drain. What improved?

Worked design and implementation

Detection improved, recovery did not. Report both. Choose a separate repair such as bounded retries or gradual admission, then measure net drain rather than claiming an earlier page solved capacity.

Separate the clocks. Record impact start, detection, operator action, dependency recovery and backlog cleared. An earlier alert changes the first response interval. It does not change the worker's net drain rate.

With 6,000 queued jobs, 120 completions/s and 100 new arrivals/s, net drain is 20/s and the ideal drain time is 300 seconds. State the simplifying constant-rate assumption. Deliver before/after timelines and choose a capacity or admission change if the recovery interval remains unacceptable.

Supplied mechanism practice

• Raw incident evidence exercise — includes its own run command, fixtures and validation limits.

These exercises verify specific boundaries. Completing their reference tests does not implement or assess the full project.