Deliver signed webhooks with retries and replay

Application background

A shop wants its own system to hear when an order changes in your commerce platform. It registers an HTTP address. Your service sends an event, such as Order 17 paid, to that address. This outgoing event request is a webhook.

The shop's server may be unavailable, or it may save the event but lose the response. Your service needs to retry without inventing a new order event, and a failing shop must not delay deliveries to healthy shops.

Example walkthrough

A delivery attempt is one network call. The event is the underlying business fact. Keeping their identities separate makes retry and investigation understandable.

Your assignment

Deliver: Build event delivery with signed requests, saved attempt history, bounded retries and an operator replay command that preserves the original event's identity.

Required behavior: Each committed source event creates one logical delivery per subscription. Attempts may repeat for 48 hours. Receivers use stable event IDs to deduplicate. Senders sign the exact body and expose delivery history.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/webhook_delivery.py

Supplied file: examples/architecture-starts/webhook_delivery.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for webhook-delivery. No AWS resources are created."""
import hmac,hashlib,json
body=json.dumps({'id':'evt-7','type':'order.paid'},separators=(',',':')).encode()
secret=b'local-fixture-secret'; timestamp='1700000000'
signature=hmac.new(secret,timestamp.encode()+b'.'+body,hashlib.sha256).hexdigest()
print('Signed bytes:',body.decode()); print('Signature:',signature)
seen=set()
for attempt in (1,2):
    event=json.loads(body)['id']; duplicate=event in seen; seen.add(event)
    print('attempt',attempt,'already handled' if duplicate else 'apply event')

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

Signed bytes: {"id":"evt-7","type":"order.paid"}
Signature: 5320fc164aa4700b4eca86de5a39d45cd8da9562e3daea0cf13bf0f92883e55c
attempt 1 apply event
attempt 2 already handled

Set up your implementation workspace

Create work/webhook-delivery/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Commit source and dispatch intent

Write an outbox event with the business transition. Expand eligible subscriptions using a recorded subscription snapshot or clearly defined selection time. Preserve event ID and immutable body through every retry and manual replay.

2. Implement signed HTTP delivery

Sign timestamp plus raw body bytes with the subscription secret. Document verification, timestamp tolerance and rotation overlap. Use HTTPS, strict deadlines and capped response bodies. Validate destination DNS and every redirect against an outbound policy. Block private and metadata addresses at connection time.

3. Isolate failing endpoints

Limit concurrency per endpoint and tenant. Apply bounded exponential backoff with jitter and a 48-hour terminal deadline. A healthy destination must keep progressing while another fails. Respect a capped Retry-After without allowing an endpoint to consume unbounded resources.

4. Expose replay and ambiguity

Return event and attempt history to authorized customers. A 2xx means the endpoint accepted the request, not that its business logic ran exactly once. Manual replay uses the same event identity and is recorded as another attempt.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

SQS transports work. The ledger owns retry eligibility and evidence. Per-endpoint limits are application behavior and do not appear automatically because the queue scales.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Rotate webhook secrets while deliveries are pending

A queued event may be delivered hours after it was created. If the receiver deletes the old secret immediately, otherwise valid delayed attempts fail authentication.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Choose signing at attempt time for this exercise. Keep event ID and payload bytes stable, but add a current timestamp, key ID and signature to each attempt. Announce the receiver overlap window and retain both verification keys during it. Never place signing secrets in queue payloads. Store key references in protected configuration and restrict access to the delivery runtime. Keep endpoint concurrency isolated so a receiver returning 429 cannot consume every worker.

Walk through the result. Queue event E while key K1 is active, activate K2, then deliver E signed with K2. Retry E later with a fresh timestamp and the same event ID. The receiver validates the signature and deduplicates the event. After overlap ends, an old K1 signature must be rejected. Supply the header contract and rotation timeline.

Allow customers to rotate a secret while attempts wait. Decide whether you sign with the current secret or an event-bound version, publish the overlap contract, and show how receivers verify a replay.