Bound retries across browser, API and SDK layers

Application background

A user presses Save once in a reading-list app. When a dependency is unavailable, the browser tries again. The API handler also retries, and its dependency library has retries of its own. One visible user action can therefore produce many calls to the failing service.

Retries can help after a brief interruption, but uncoordinated layers multiply work. They can also repeat a write whose success response was lost. First count the calls, then choose which layer should own the retry decision.

Count the calls caused by one operation

The supplied model prints:

3 total attempts per layer -> 27 dependency attempts
4 total attempts per layer -> 64 dependency attempts
Start another attempt: False

There are three retrying layers. The first line is 3 × 3 × 3, counting the original call among each layer's attempts. In the final line, only 500 ms remain but the suggested retry wait is 2,000 ms, so another attempt cannot fit the original deadline.

Retry amplification is this multiplication of calls. A retry budget limits the total extra work, and idempotency prevents repeated attempts from creating repeated effects.

Your assignment

Deliver: Count the dependency calls caused by one user operation. Revise the retry policy so one layer owns a bounded budget, preserves the original deadline and avoids repeated write effects.

Required behavior: One layer owns the retry budget for a logical operation. Count total attempts explicitly, preserve the original deadline and operation identity, and retry writes only under a verified idempotency/reconciliation contract.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/03_the_retry_storm_you_build_on_purpose.py

Supplied file: examples/architecture-starts/03_the_retry_storm_you_build_on_purpose.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for 03-the-retry-storm-you-build-on-purpose. No AWS resources are created."""
for attempts in (3,4): print(attempts,'total attempts per layer ->',attempts**3,'dependency attempts')
remaining_ms=500; retry_after_ms=2000
print('Start another attempt:',retry_after_ms<remaining_ms)
import random
rng=random.Random(7)
print('Bounded jitter delays ms:',[round(rng.uniform(0,400)) for _ in range(5)])

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

3 total attempts per layer -> 27 dependency attempts
4 total attempts per layer -> 64 dependency attempts
Start another attempt: False
Bounded jitter delays ms: [130, 60, 260, 29, 214]

Set up your implementation workspace

Create work/03-the-retry-storm-you-build-on-purpose/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Measure actual amplification

Attach one logical operation ID across browser, API and SDK. Count attempts at each layer and compare to user actions. Inspect SDK defaults rather than assuming the retry loop visible in application code is the only one.

2. Assign one retry owner

Disable or constrain nested retries so the total budget is explicit. Use a maximum total-attempt count, shared deadline and bounded exponential backoff with jitter. Stop when Retry-After exceeds remaining time. Do not start work that cannot contribute to the response.

3. Preserve write identity

Reuse one provider idempotency key for an exact write intent and reject changed payload under that key. A timeout after a successful provider commit is unknown, not definitely failed. Reconcile before switching providers or creating another operation.

4. Observe recovery under load

Use a finite local fault window and compare request count, attempt timing, useful throughput and queue age. A circuit breaker can reduce doomed calls, but half-open probes must also be bounded so recovery does not release every waiting client at once.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

The API and SDK are separate retry layers even when they run in one process. Explicit ownership makes the worst-case dependency load calculable.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

The provider’s idempotency retention expires before your retry horizon. Shorten the automatic retry window or add a provider lookup/reconciliation path. Do not assume an old key remains deduplicated forever.