Build restartable CSV export jobs

Application background

An accountant clicks Export invoices in a billing dashboard. Producing the CSV might take a minute, so making the browser wait for the entire file is inconvenient. The API should instead return a job ID that the accountant can use to check progress.

A separate process called a worker reads the job, creates the file and marks it ready. The job record must remain available if either process stops. A repeated message must not let two workers publish conflicting results.

Watch two workers attempt the same job

The numbers 1 and 2 identify successive owners of the same job, not two different exports. The diagram shows why checking ownership only when work starts is insufficient.

Durable means the accepted work survives a process restart. A lease gives one worker temporary ownership, and a fencing value lets storage reject a late result from an older owner.

Your assignment

Deliver: Build an export request, a background worker and a status/download interface. Keep accepted work across restarts and prevent a worker that no longer owns the job from publishing its result.

Required behavior: POST /exports with a tenant-scoped request ID returns 202 plus a stable job URL after durable acceptance. GET /exports/{id} returns accepted, running, succeeded, failed or cancelled. Expose a download URL only after the current worker has saved a completed result record.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/durable_jobs.py

Supplied file: examples/architecture-starts/durable_jobs.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for durable-jobs. No AWS resources are created."""
job={'epoch':4,'state':'running','result':None}; objects={}
def publish(epoch,key):
    if job['epoch']!=epoch or job['state']!='running': return 'stale publication rejected'
    job.update(state='succeeded',result=key); return 'published'
old_epoch=4
job['epoch']=5
objects['job7/5/result']='new worker bytes'
print(publish(5,'job7/5/result'))
objects['job7/4/result']='late old worker bytes'
print(publish(old_epoch,'job7/4/result'))
print('Visible result:',job['result'],objects[job['result']])

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

published
stale publication rejected
Visible result: job7/5/result new worker bytes

Set up your implementation workspace

Create work/durable-jobs/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Accept durably

Commit job identity and dispatch intent in one database transaction. A lost response followed by the same request ID returns the original job. Queue publication happens from the outbox. A queue message is a request to advance a job, not its authoritative status.

2. Claim and renew ownership

Store an incrementing epoch with lease expiry. Claim using a conditional update. Renew only the matching epoch. SQS visibility reduces concurrent work but cannot prevent every duplicate. A worker must stop publishing if renewal or the conditional ownership check fails.

3. Publish an immutable result

Upload to an attempt-specific object key. Conditionally write succeeded and the object pointer under the current epoch. Acknowledge the queue message after that commit. If redelivered, read completed state and acknowledge without generating another visible result.

4. Operate a bounded service

Reject new work with Retry-After once admitted backlog exceeds the product’s wait budget. Add cancellation and tenant fairness. Garbage-collect unreferenced attempt objects only after active leases and retry windows. Authorize current access when issuing a download.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

SQS can redeliver, so DynamoDB owns job state and the publication condition. ECS accommodates longer exports. Short bounded jobs could run in Lambda. S3 stores complete bytes, while the conditional result pointer decides which bytes users receive.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Erase data while an export worker is still running

Alice asks for deletion after the worker has read her records. Removing a queued message cannot stop bytes already held by a running process. The worker may finish and create a fresh downloadable copy after deletion.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Add a subject deletion generation and a publication pointer separate from the uploaded object. Upload into a private temporary prefix. Publish the pointer only through an atomic condition that checks job ownership, cancellation and the unchanged deletion generation. Deletion increments that generation and revokes the pointer before asynchronous object cleanup. Authorize every download if immediate revocation is required. On AWS, use a conditional database transaction for publication and S3 for private generation-specific objects.

Walk through the result. Pause worker A after upload. Delete the subject, then resume A. Publication must fail and the temporary object must appear in cleanup work. The status page reports cancelled rather than ready. Show how cleanup finds an orphan even if A crashes before reporting the failed publication.

Add data erasure while an export is running. Define how the worker discovers cancellation, how the current pointer is revoked, and how object cleanup is proven without assuming that queue cancellation stops an already running process.