Compute trending topics from duplicate and late events

Application background

A sports app shows the most discussed topics from recent reactions. When a goal is scored, many people react to the same topic at once. The leaderboard should describe recent activity rather than count every reaction since the app launched.

Choose a time window, such as the last few minutes, and count each reaction in the appropriate window. Repeated messages and reactions delivered late can otherwise make a topic look more popular than it is.

Example walkthrough

A hot key is one topic receiving a disproportionate share of work. Splitting that work can help capacity, but the partial counts still need a clear combination rule.

Your assignment

Deliver: Build a leaderboard for a declared recent time window. Handle repeated and late reactions, and tell readers how current the displayed ranking is.

Required behavior: Return the top 20 topics for a named event-time window and region, with as-of time and revision. The exercise permits two minutes of lateness and targets updates within thirty seconds for timely events.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/trending_counts.py

Supplied file: examples/architecture-starts/trending_counts.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for trending-counts. No AWS resources are created."""
import hashlib
seen=set(); partials={}
for eid,topic in [('e1','goal'),('e2','goal'),('e1','goal'),('e3','save')]:
    if eid in seen: continue
    seen.add(eid); shard=int(hashlib.sha256(eid.encode()).hexdigest(),16)%4
    key=(topic,shard); partials[key]=partials.get(key,0)+1
totals={}
for (topic,shard),count in partials.items(): totals[topic]=totals.get(topic,0)+count
print('Partials:',partials); print('Ranking:',sorted(totals.items(),key=lambda x:(-x[1],x[0])))

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

Partials: {('goal', 2): 1, ('goal', 3): 1, ('save', 1): 1}
Ranking: [('goal', 2), ('save', 1)]

Set up your implementation workspace

Create work/trending-counts/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Specify what one count means

Choose event types, bot/duplicate treatment, topic normalization and region source. Define window boundaries and a deterministic tie-breaker. Store event and received times so delayed transport is not mistaken for a new reaction.

2. Aggregate in two stages

Assign each event to a deterministic shard, deduplicate within the supported replay horizon and compute partial counts. Merge complete partials for the exact baseline. If you later send only local top-K candidates, prove the approximation behavior rather than claiming exact global top-K.

3. Publish immutable leaderboard revisions

Build a complete snapshot with as-of and finality metadata, then move one serving pointer. Late accepted events create a new revision. Reject stale writers so a delayed merge cannot overwrite a newer leaderboard.

4. Bound overload and history

Prioritize current windows, retain enough state for the two-minute lateness rule and send older corrections to a defined path. If sampling is introduced under overload, show that mode and its error implications to the consumer.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Partial aggregation absorbs hot-topic writes. A versioned snapshot avoids serving half of a new ranking mixed with half of an old one.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Use approximate trends without pretending every rank is exact

A topic near rank ten may be omitted by a small candidate summary or move after late events. A compact counting structure alone does not define which topic IDs you retain.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Choose and document both a candidate-discovery algorithm and a count estimator. Keep the algorithm's error conditions and merge rules with the result schema. For this exercise, use approximate candidates for a provisional display and retain raw events for a later exact top-ten calculation. Compare the candidate set against exact counts on a bounded slice. Do not use approximate public rankings as billing evidence. Preserve the existing event-time and late-arrival policy.

Walk through the result. Construct a window where ranks ten and eleven differ by two events. Add a late event and show both the provisional result and the finalized exact result. Report candidate recall and count error separately. If the approximation cannot distinguish the boundary, label the display provisional rather than inventing a stable rank.

Limit memory with approximate heavy-hitter sketches. Specify false-positive/false-negative and count-error behavior, then decide whether the product can display an approximate ranking without misleading users.