Reject excess API work before queues grow without bound

Application background

A subscription API handles customer writes, interactive reads and large exports. Its machines can finish only a limited amount of work each second. If twice that amount keeps arriving, letting everything wait merely builds a longer queue.

The application must decide which work may start, which may wait briefly and which must be rejected. Rejecting excess work early can preserve useful service for accepted requests and give callers a clear response.

Example walkthrough

Load shedding means deliberately refusing work the service cannot complete within its limits. It needs a caller-visible response, not a hidden drop.

Your assignment

Deliver: Build request admission with finite waiting limits and explicit rejection responses. Apply the declared work priorities and demonstrate recovery when demand falls.

Required behavior: Admission is bounded by class and dependency capacity. Accepted work has a finite deadline. Rejected work returns a clear retryable or non-retryable outcome before expensive side effects begin.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/overload_shedding.py

Supplied file: examples/architecture-starts/overload_shedding.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for overload-shedding. No AWS resources are created."""
arrival=40000; service=20000; limit=100000; queued=0
for second in range(1,8):
    offered=queued+arrival; completed=min(service,offered)
    waiting=offered-completed; rejected=max(0,waiting-limit); queued=min(waiting,limit)
    print(second,'completed',completed,'queued',queued,'rejected',rejected)

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

1 completed 20000 queued 20000 rejected 0
2 completed 20000 queued 40000 rejected 0
3 completed 20000 queued 60000 rejected 0
4 completed 20000 queued 80000 rejected 0
… (more output follows)

Set up your implementation workspace

Create work/overload-shedding/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Locate the constrained dependency

Measure completed throughput, active work and queue age as arrivals rise. Identify whether CPU, database connections or a downstream quota saturates. Adding API instances cannot create more capacity in a fixed database dependency.

2. Classify and admit early

Derive class from trusted route/account context. Reserve explicit budgets for critical work and reject excess before allocating expensive connections or starting external effects. Keep fairness within a paid class so one tenant cannot spend the entire reserve.

3. Bound waiting and retries

Cap queue length and maximum age, expire work that cannot finish before its deadline, and propagate cancellation. Return bounded Retry-After and document client retry budgets with jitter. Retrying rejected traffic at every service layer multiplies the original overload.

4. Recover without a second surge

Reduce concurrency when the dependency degrades and ramp it back gradually after recovery. Keep hysteresis between shedding and reopening. Report completed useful work and class-specific rejection, not merely a lower response latency caused by rejecting everything.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Admission protects the scarce dependency. SQS is suitable for explicitly deferred work, but it cannot turn an unbounded interactive wait into a successful user experience.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Allocate scarce database work across competing products

Priority labels cannot create connections or CPU. A cheap read and an expensive export also should not consume the same unit of budget merely because each is one HTTP request.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Measure approximate database work per operation and reserve a small control/recovery share. Give each product a minimum allocation plus bounded borrowing from unused capacity. Enforce total active work at a shared admission boundary, then use separate queues or worker pools for exports and interactive requests. Expire queued work that can no longer meet its user deadline. Keep durable business obligations visible for later reconciliation.

Walk through the result. Use an illustrative budget of 100 work units/s. Saves cost one unit and exports cost 100. One export must not consume an unbounded slot alongside 100 supposedly admitted saves. Show the chosen allocation, denied work and recovery reserve during a 60-second overload. Deliver absolute rates as well as percentages.

Product labels every route critical. Use dependency consumption and user consequences to produce an allocation that fits actual capacity. A label cannot reserve resources that do not exist.