Aggregate click events with late arrivals and reconciliation

Application background

An advertiser opens a dashboard to see how many people clicked each campaign. Click messages include an event ID and the time the click happened. They may arrive twice or reach your service minutes after the actual click.

The dashboard should update quickly, but finance needs a more carefully reconciled count for billing. Those outputs can have different completion rules as long as the interface makes the difference visible.

Example walkthrough

Event time is when the click happened. Arrival time is when your service received it. Windowed reports must choose which time they use and how late corrections work.

Your assignment

Deliver: Build campaign counts that handle repeated and late clicks. Address unusually busy campaigns and distinguish provisional dashboard values from reconciled billing evidence.

Required behavior: Aggregate by campaign and event-time window with a documented duplicate window and late-arrival policy. Dashboard values are provisional until finalized. Reporting counts are not automatically a charge ledger.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/ad_click_aggregator.py

Supplied file: examples/architecture-starts/ad_click_aggregator.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for ad-click-aggregator. No AWS resources are created."""
seen=set(); windows={}; watermark=120; lateness=120
for eid,campaign,at in [('e1','c7',61),('e1','c7',61),('e2','c7',10),('e3','c7',-180)]:
    if eid in seen: print(eid,'duplicate'); continue
    if at<watermark-lateness: print(eid,'too late: reconciliation path'); continue
    seen.add(eid); key=(campaign,(at//60)*60); windows[key]=windows.get(key,0)+1
print('Provisional counts:',windows)

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

e1 duplicate
e3 too late: reconciliation path
Provisional counts: {('c7', 60): 1, ('c7', 0): 1}

Set up your implementation workspace

Create work/ad-click-aggregator/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Ingest a stable click envelope

Preserve event ID from the event producer, campaign and event timestamp, and add received time at ingestion. Validate timestamp bounds so a future-dated event cannot advance watermarks arbitrarily. Keep attribution policy version with derived results.

2. Aggregate event-time windows

Define window boundaries, watermark source and allowed lateness. Deduplicate within the declared replay horizon, checkpoint state with progress, and emit revisions when a late accepted event changes a prior window. Show provisional/final status in the API.

3. Shard hot campaigns

Split one campaign into deterministic partial counters and merge them for serving. Estimate both write distribution and merge work. A random shard suffix does not solve duplicate identity unless every replay chooses the same shard.

4. Reconcile billing separately

Preserve the evidence needed for invalid-traffic filtering, attribution corrections and financial reconciliation. A fast dashboard may undercount during lag. Do not convert an approximate or revisable count directly into irreversible charges.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

The stream processor owns provisional event-time state, the serving table owns a particular published revision, and the archive supports rebuilding. Those are distinct guarantees from a billing ledger.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Separate revisable click reports from finalized invoices

An advertiser paid for yesterday's clicks using fraud policy 3. Recomputing the dashboard with policy 4 may lower the count, but overwriting the invoice would erase what was actually charged.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Keep raw event IDs and versioned metric definitions. At invoice finalization, retain the time range, attribution policy, fraud version and evidence manifest used for its lines. Later corrections create adjustment entries that reference the original invoice. The dashboard can show a newer provisional generation with its own watermark. Use the warehouse for recomputation and a transactional billing ledger for final documents and adjustments.

Walk through the result. Invoice I contains 1,000 billable clicks under policy 3. Policy 4 excludes 100 of those clicks. Show the original invoice unchanged, a separately approved adjustment for 100 and a dashboard count of 900 under policy 4. Deliver the three linked records and explain which is authoritative for each question.

An advertiser disputes yesterday’s invoice after fraud filtering changes. Define immutable invoice evidence, correction entries and the relationship between a revisable dashboard and a finalized financial document.