Stage 3: Add durable jobs and bounded caching

Application background

Your reading list works for a small group, but more readers and slower title websites now put pressure on it. You want saves to return promptly while a separate worker fetches titles. You also want to reuse suitable read results so every page view does not query the database.

A queue holds title work waiting for workers. A cache holds reusable copies of read data. Both introduce new questions about repeated jobs, old workers and outdated or private results.

Example walkthrough

Adding a queue or cache is only part of the change. You must define who owns each result, when a copy may be reused and what happens when capacity is exhausted.

Sizing that affects this decision

200 readers making one request/s offer 200 reads/s to a database budgeted for 100. Reusing allowed results or refusing excess work must close that gap. Ten application instances can also initiate ten separate fills for one hot key even if each instance combines its own duplicate requests.

These are exercise assumptions. The estimation reference explains the units and approximations. They do not establish the local demo's measured capacity.

Your assignment

Deliver: Extend your application with saved title jobs and controlled caching. Prevent old workers from replacing current results and keep cache misses within the database's capacity budget.

Required behavior: Job acceptance is durable, result publication is fenced by current ownership epoch, and cache misses cannot exceed the database budget. A cache improves reads without becoming the authority for ownership, deletion or job completion.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/reading_list_under_load.py

Supplied file: examples/architecture-starts/reading_list_under_load.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for reading-list-under-load. No AWS resources are created."""
job={'epoch':2,'state':'succeeded','title':'New title'}
def late_publish(epoch,title):
    if job['epoch']!=epoch or job['state']!='running': return 'rejected stale attempt'
    job['title']=title; return 'published'
print(late_publish(1,'Old title'),job)
print('Offered 200/s, origin budget 100/s: at least 100/s need cache, bounded stale data, or rejection.')
print('Ten local coalescers can still produce ten origin fills for one hot miss.')

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

rejected stale attempt {'epoch': 2, 'state': 'succeeded', 'title': 'New title'}
Offered 200/s, origin budget 100/s: at least 100/s need cache, bounded stale data, or rejection.
Ten local coalescers can still produce ten origin fills for one hot miss.

Run the application you will extend

The reading-list API setup guide gives you a real local HTTP server, SQLite database, save/list/edit requests and controlled title success/timeout behavior. Start it in one terminal and send the documented curl requests from another. Read that setup before following the implementation steps below. The demo above isolates this lesson's mechanism. The server is where you integrate it.

Continue the application you built in the preceding stage. The supplied server is only a Stage 1 starting point. It does not contain the previous stages' completed UI, operations, jobs or AI feature.

For a first run, start this in terminal 1 from the repository root:

python3 examples/reading-list-starter/app.py --db /tmp/reading-list.sqlite3

In terminal 2, save one bookmark with a controlled title timeout:

curl -i http://127.0.0.1:8080/bookmarks \
  -H 'X-Demo-User: alice' -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/docs","title_mode":"timeout"}'

Expect 201 Created, a bookmark id and title_status: "timeout". The URL is persisted despite the title failure. This is the supplied baseline. The assignment adds the behavior described above. The lookup is a fixture, so no external website is contacted. For members Bob or Ben in a scenario, use the starter's second demo identity bob. Alice or Ana corresponds to alice.

Work in your own branch or copy examples/reading-list-starter/ to work/03-under-load/. app.py exists in that directory. Add the modules named below there as you separate HTTP, storage and background work. The server has demo membership, not production authentication.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Move enrichment behind durable jobs

Commit job/outbox identity with the saved link. Add worker claims with lease and epoch, bounded retries and a visible failed/repair state. A queue message is delivery, while the job record is truth.

2. Fence publication and source versions

Before storing a title, require both current worker epoch and current link URL version. A completed old fetch cannot overwrite a new URL’s title. Acknowledge after result commit and recognize already completed jobs on redelivery.

3. Add cache-aside with bounded origin work

Key by group and source version, enforce authorization before response and coalesce hot misses. Add an explicit database concurrency/rate budget shared across the relevant fleet scope. Choose safe stale serving or rejection when the budget is exhausted.

4. Observe useful load and recovery

Record offered/completed/rejected requests, cache hit rate, origin reads, queue age and stale publication rejections. Stop a worker and clear the cache in a controlled local run. Show bounded recovery instead of only a warm-cache latency number.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

The added services separate optional work and hot reads, but the original database/version authority remains. Explicit resource budgets make the ten-instance design understandable during cache failure.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Continue to stage 4 by adding optional AI tags without putting model latency or model guesses on the critical save/list path.

Follow-up scenarios and worked designs

Follow-up 1 · The owner stops after fetching

A crash occurs after the remote GET but before the durable record. How does retry recover?

Worked design and implementation

Refetch is allowed. Record outcome only with the current generation in an atomic completion transaction. Match payload hashes. A duplicate ID with different content is a conflict, not a replay.

Recover the logical job, not the old process. The remote GET may run again because its response was never durably recorded. Keep the original job ID and payload fingerprint, acquire a new generation and create a new attempt ID. Completion atomically stores the result only for the current owner and bookmark version.

Pause A after fetching, let B finish, then resume A. Show two fetch attempts but one published title result. Reuse the job ID with another URL and show conflict. If fetching later becomes billable or has side effects, this safe-repeat assumption must be revisited.

Follow-up 2 · The cache disappears

Traffic remains 1,000 reads/s but the database can handle only 100/s. What should users see?

Worked design and implementation

Bound origin/bypass work and choose authorized bounded-stale responses or quick 429/503. If read-your-writes is required, use primary/session watermark/confirmed progress. A finite primary pin cannot cover unbounded replica lag.

Make cache failure an admission problem. Keep a hard origin budget shared across instances or explicitly partitioned so its total stays within 100/s. Coalesce identical misses where possible. Serve stale content only within its freshness and authorization contract. Preserve read-your-writes using confirmed version progress or the primary path, rather than a timer that assumes replicas caught up.

At 1,000 offered reads/s, show how every request is completed, coalesced or rejected while origin traffic remains bounded. Hand over the recovery timeline and an immediate read after a save during replica lag.

Revised flow. These are proposed components to implement, not extra services started by the supplied demo.

Supplied mechanism practice

• Cache and replica boundary lab — includes its own run command, fixtures and validation limits.
• Lease and provider recovery lab — includes its own run command, fixtures and validation limits.
• Reliability arithmetic and incident lab — includes its own run command, fixtures and validation limits.

These exercises verify specific boundaries. Completing their reference tests does not implement or assess the full project.