Build chat with durable messages and reconnect recoveryLESSON 9.07 · 7 OF 22 IN CHAPTER
PART C / System design under constraints
Step 137 of 252
LESSON 9.07 · 7 OF 22 IN CHAPTERTry it, then open the solution

Build chat with durable messages and reconnect recovery

Application background

Ana sends a message to her field-support team from her phone. The server saves it, forwards it to connected teammates and later records who has read it. Those are three different events, so the interface should not use one status to mean all three.

Ana's connection can drop after the server saves the message but before her phone receives confirmation. Her phone then resends it. Without a stable message identity, the conversation may show the same message twice.

Example walkthrough

01 · Try this input

Input / starting state
Ana sends message phone-42
Expected result
Store one conversation entry.

02 · Try this input

Input / starting state
Her phone reconnects and resends phone-42
Expected result
Return the existing entry rather than creating another.

03 · Try this input

Input / starting state
Ben reconnects after missing messages
Expected result
Resume from his last received position.

A reconnect cursor is a saved position in the conversation. It tells the server which later messages to send without assuming the phone received everything previously transmitted.

Your assignment

Deliver: Build message saving and reconnect recovery. Recognize repeated sends as the same message and report saved, delivered and read outcomes separately.

Required behavior: POST /conversations/{id}/messages uses a client_message_id. Acknowledged means the message is durably stored. Reconnect reads after a conversation cursor. Delivered and read receipts describe particular devices or users.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/realtime_chat.py

Supplied file: examples/architecture-starts/realtime_chat.py. You can also read or download the source here (download file, source below).

Read the supplied code · realtime_chat.py
read or download the source here · realtime_chat.py
"""Local mechanism demonstration for realtime-chat. No AWS resources are created."""
messages=[]; operations={}
def send(sender,key,text):
    identity=(sender,key)
    if identity in operations: return operations[identity]
    record={'sequence':len(messages)+1,'sender':sender,'text':text}
    messages.append(record); operations[identity]=record; return record
print(send('ana','phone-7','On my way'))
print(send('ana','phone-7','On my way'))
print('Reconnect after sequence 0:',[m for m in messages if m['sequence']>0])

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

{'sequence': 1, 'sender': 'ana', 'text': 'On my way'}
{'sequence': 1, 'sender': 'ana', 'text': 'On my way'}
Reconnect after sequence 0: [{'sequence': 1, 'sender': 'ana', 'text': 'On my way'}]

Set up your implementation workspace

Create work/realtime-chat/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Record / module Key or interface Responsibility
messages conversation_id,sequence,message_id Durable ordered conversation history.
client_operations sender,client_message_id,payload_hash Retry identity. Changed text under the same ID conflicts.
member_cursors conversation,member,delivered_seq,read_seq Monotonic receipt positions. Membership gates history.

Implement the assignment

1. Store before acknowledging

Authenticate membership, allocate a conversation sequence through a serialized partition owner or database transaction, then commit the message and retry result. Do not acknowledge based only on writing to a WebSocket buffer. Keep attachments outside message rows with authorized object references.

2. Implement reconnect replay

Persist each device’s last received sequence. Fetch bounded pages after that cursor, detect gaps, and deduplicate by message ID in the client. Push delivery is an acceleration. Durable history fills missed messages.

3. Separate receipt meanings

Show saved once the server commit is known, delivered after a device acknowledgement, and read after the user’s explicit view event. Advance cursors monotonically. A recipient going offline cannot retract the sender’s stored message.

4. Enforce membership across history

Check current membership before history reads and attachment downloads, and define access to messages from before joining. On removal, stop new push delivery and reject reads even if cached conversation metadata is old.

Demonstrate the completed local result

01 · Try this input

Input / starting state
Run the starting program
Expected result
Two sends with one client ID produce one stored message and the same sequence.

02 · Try this input

Input / starting state
Disconnect after server commit
Expected result
Reconnect replay includes the message. Retry does not duplicate it.

03 · Try this input

Input / starting state
Remove a member with a warm history cache
Expected result
New reads and attachment access are denied.

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Input or objective Calculation / consequence
Two million daily users. 30,000 messages/s peak At 1 KiB/message that is about 30 MiB/s before indexes, replicas and attachments.
Groups of 2–500 members A 500-member conversation can turn one write into 499 delivery attempts. Fan-out costs differ from stored message rate.
One-year history Estimate from average messages/day. Attachments need a separate size and retention model.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Build chat with durable messages and reconnect recovery: AWS services, their general roles, and the primary data flow

WebSockets carry live updates but are not the history authority. DynamoDB fits conversation-keyed access. PostgreSQL is a practical smaller baseline if sequence assignment and membership transactions dominate.

Local responsibility Cloud destination and role Implementation still required
Local HTTP boundary or the endpoint you will add Amazon API Gateway: WebSocket connections Create routes and an integration. Translate requests and responses and configure identity validation.
Application or worker process Amazon ECS: conversation application Build a container and task definition. Supply configuration, task roles and graceful shutdown behavior.
Local dictionary, SQLite records or state model Amazon DynamoDB: message and cursor store Design partition/sort keys and write a storage adapter with conditional updates or transactions. Python state and SQL are not uploaded as a database.
Local event sequence or input stream Amazon Kinesis: committed delivery events Implement producer/consumer adapters, partition keys, durable acceptance and checkpoint/replay behavior.
Application or worker process Amazon ECS: push delivery workers Build a container and task definition. Supply configuration, task roles and graceful shutdown behavior.
Local file, object fixture or exported payload Amazon S3: private attachments Implement upload/download and metadata adapters, scoped access, object naming, retention and incomplete-upload cleanup.

Provision resources, then connect the application

Resource or boundary Initial configuration and reason
Message partitions Measure hot conversations and avoid claiming an unlimited per-conversation write rate. Sequence allocation is coordination.
Connection registry Expiring device routes. Stale connection failures remove routes, never message history.
Storage and logs Encrypt private content. Bound attachment sizes. Avoid logging message text or signed URLs.

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Order a chat room across regions and scale its delivery

Connection proximity does not determine message order. If two regions independently allocate sequence 42, reconnecting clients cannot know which message belongs at that position.

Starting design Changed requirement
One room sequencer orders durable messages. Users connect in several regions and a hot room needs multiple fanout workers.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

Diagram: Worked follow-up: Order a chat room across regions and scale its delivery

What to implement. Choose a home sequencer for each room. Remote gateways forward sends to it and acknowledge only after the room log is durable under the stated recovery contract. Partition delivery by recipient, not by independent room writers. Every client keeps its last received sequence and deduplicates replayed messages. Recheck room membership before backlog delivery. On failover, fence the old sequencer and verify log coverage before the new owner allocates sequence numbers.

Walk through the result. Deliver message 42 to half the room, stop a fanout worker and reconnect a client at cursor 41. It receives 42 once visibly, then 43. Remove another member before replay and show denial. Your diagram and transcript must distinguish ordering, durable acknowledgement and delivery, which are three separate responsibilities.

Add cross-region conversations. State which region orders a conversation and what happens during its outage. Global delivery does not imply independent regions can allocate the same conversation sequence safely.

Additional design cases, alternatives and original source notes

Assume 2 million daily active users, 30,000 peak messages/s, groups of 2–500 members, and one-year history. These are exercise inputs. Distinguish server-accepted messages from messages displayed or read on a device.

01 · Try this input

Input / starting state
Send succeeds, reply disappears, same client message ID retried
Expected result
One stored message and one stable server message ID

02 · Try this input

Input / starting state
Ben reconnects after missing messages 18–21
Expected result
Resume after acknowledged cursor 17. Return 18–21, then live delivery

03 · Try this input

Input / starting state
Two devices send concurrently into one room
Expected result
Explain the chosen per-room ordering rule. No unearned global order

04 · Try this input

Input / starting state
Device is offline for two days
Expected result
History query and pagination work even when live connection state expired

Separate storage from delivery

The message write becomes authoritative when the server persists it and assigns a room sequence or another documented order. WebSockets carry low-latency delivery, presence, and acknowledgements. Connections are not the message database. Store (room_id, sequence, message_id, sender_id, body, created_at) and an idempotency key scoped to sender and room. A client must retain the same send-operation ID across retries. A laptop cannot deduplicate a phone's uncertain send unless that pending ID was synchronized. Two independently composed identical texts remain two messages. If a device misses sequence 19, fetch the gap before advancing its cursor. Delivery may be at least once. The UI deduplicates by message ID. “Read” means the client confirms display according to an explicit policy, not merely that a push reached a gateway.

One accepted message

send(room_id, operation_id, body) derives sender identity from authentication. A fenced room owner chooses next_sequence. One DynamoDB transaction checks its ownership epoch and current counter, advances the counter, and stores the message, (sender, room, operation_id) → result + request_hash, and an outbox row. A duplicate key reads the existing result. Different content under that key conflicts. If the epoch or sequence check fails, reload or hand off—do not publish the uncommitted message.

Arrow Acknowledgement means Recovery
Sender → room authority Transaction committed message + sequence + replay + outbox. Same operation ID recovers an uncertain acknowledgement.
Outbox → fanout queue Fanout work is queued, not displayed. Retry relay. Consumers deduplicate (message_id, recipient).
Fanout → authorized connection Delivery attempt only. Recheck membership before private bytes leave. Missing/duplicate broadcast is repaired by authorized history replay.
Device → cursor store Client has received a contiguous range. Never advance over a missing sequence. Retention expiry requires explicit resync.

Failure trace: commit sequence 18 → owner crashes before broadcasting → outbox relays twice → recipient reconnects at cursor 17. History returns message 18 once in the UI. A stale owner fails its epoch condition.

Capacity example: 30,000 messages/s × 50 online recipients = 1.5 million delivery attempts/s, excluding retries. If accepted-to-queued takes 50 ms, queueing 100 ms and live delivery 150 ms, the target is 300 ms for online devices, not for offline readers. Bound queue age and per-room fanout. Retained history, not infinite live buffering, handles slow clients.

Make the boxes accountable

Draw authentication/room membership check → message authority → history store → fanout workers → connection gateways → devices. Membership changes must be checked before fetching private history as well as before live subscriptions. Estimate the largest room's fanout separately from average room size. If media is added, separate encrypted object storage and its authorization from message ordering.

Put the AWS names on the boxes

Why these boxes, and what changes the choice: WebSocket connections deliver low-latency updates, while a DynamoDB log owns acknowledged history. ECS owners may assign room sequence numbers. A database conditional write must still fence old owners. SQS fans out work but can redeliver, so reconnect uses log cursors instead.

Senior follow-up: A fanout worker dies after delivering to half a room. The cursor-based recovery path replays without duplicating visible messages. Show what a restarted worker reads, when it checkpoints, and what happens if a user is removed halfway through a backlog.

Staff follow-up: The same room has members in three Regions. Choose one sequencing authority per room or state the weaker ordering promise. Partition a hot room's fanout without promising independent writers a single total order for free. Specify how ownership transfers and how to check no accepted message disappears.

Practice artifact: Label the durable commit on a before/after diagram. Trace the lost acknowledgement and reconnect. Define retention, per-room ordering, and two tests for replay plus removal.

AWS translation: API Gateway WebSocket or self-managed ECS gateways hold connections. A durable store holds history and cursor. DynamoDB can model PK=room, SK=sequence, but a single huge room may need a different distribution strategy. SQS standard delivery can duplicate and reorder, so it cannot establish room ordering. See SQS standard queue guarantees.

Source note: Original scenario, inspired by recurrent senior-level delivery/consistency themes. No company attribution.

Sources and further reading · 1