Ingest and query metrics with bounded cardinality

Application background

Machines report measurements such as CPU use and request count. A metrics service stores those values over time so engineers can draw dashboards and compare current behavior with past capacity. Each measurement has a name, timestamp, value and identifying labels.

Labels distinguish measurements, for example service=orders and region=west. Adding a different request ID to every measurement creates a huge number of separate histories, even if the number of requests has not changed.

An input that changes the amount of stored state

These two illustrative samples contain the same measurement but different labels:

{"name":"request_duration_ms","value":12,"labels":{"service":"checkout","region":"west"}}
{"name":"request_duration_ms","value":12,"labels":{"service":"checkout","region":"west","request_id":"req-987"}}

With a unique request_id on every sample, the second form can create a new series for every request. The supplied label-policy model rejects that extra label. Put request-level investigation details in a log or trace rather than this metric identity.

A time series is one such history for a particular metric name and label combination. Cardinality is the number of distinct series, which directly affects storage and query work.

Sizing that affects this decision

300,000 hosts reporting every ten seconds produce 30,000 host reports/s. At 100 series per report, that is three million samples/s and thirty million active host-series combinations before additional labels. A unique request-ID label would change the series count again, not just the size of one sample.

These are exercise assumptions. The estimation reference explains the units and approximations. They do not establish the local demo's measured capacity.

Your assignment

Deliver: Build metric acceptance and storage rules, recent queries and retained aggregate history. Demonstrate an operating response to an unexpected increase in distinct label combinations.

Required behavior: Accept timestamped numeric samples with a bounded label schema, serve recent range queries, and retain downsampled history with explicit aggregation semantics. Missing data remains missing. It is not automatically zero.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/metrics_platform.py

Supplied file: examples/architecture-starts/metrics_platform.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for metrics-platform. No AWS resources are created."""
buckets=[{'count':1,'sum':100},{'count':9,'sum':90}]
wrong=sum(b['sum']/b['count'] for b in buckets)/len(buckets)
correct=sum(b['sum'] for b in buckets)/sum(b['count'] for b in buckets)
print('Unweighted average of averages:',wrong)
print('Correct weighted average:',correct)
labels={'service':'checkout','request_id':'unique-123'}
print('Rejected labels:',sorted(set(labels)-{'service','region','status'}))

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

Unweighted average of averages: 55.0
Correct weighted average: 19.0
Rejected labels: ['request_id']

Set up your implementation workspace

Create work/metrics-platform/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Define metric types and admission

Distinguish counters, gauges and histograms. Validate sample timestamps and allowed labels, enforce per-tenant active-series budgets and report rejected samples. Request IDs belong in trace/log context, not an unbounded metric dimension.

2. Build recent ingestion and queries

Batch samples, partition by stable series identity and handle retries according to the storage engine’s duplicate semantics. Bound query time range, series expansion and concurrent expensive queries. Measure ingestion delay independently from dashboard refresh time.

3. Downsample with valid algebra

Keep sum and count for weighted averages, and preserve the histogram representation needed for supported percentile queries. Counter resets require explicit handling. Store rollup generation and source coverage so a partial bucket is not presented as complete.

4. Plan retention and overload

Use hot retention for operational queries and S3-backed archived rollups for the year-long exercise. Verify managed-store retention/ingestion quotas rather than assuming every tier has the same limits. Degrade expensive historical queries before losing urgent recent ingestion.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Prometheus-compatible storage handles recent operational queries. A separate rollup/archive path provides the stated long-term history. The diagram includes an ingestion adapter because a stream does not write itself into a metric store.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Budget temporary diagnostic metrics separately

An optional label with thousands of values can multiply every existing series. A time-to-live on samples does not prevent the ingestion and indexing cost incurred before they expire.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Create a separate diagnostic namespace with an owner, expiry, label allowlist, maximum active series and ingestion-rate budget. Enforce admission before expensive storage allocation. Expiry stops new writes and removes the temporary dashboard when its retention ends. Keep monitoring-path health separate from the application's health. A separate CloudWatch namespace or backend tenant helps identify cost, but application admission must enforce the chosen budget.

Walk through the result. Grant an illustrative 5,000-series diagnostic allowance for 30 minutes. Attempt to introduce the 5,001st active series and show rejection with a bounded counter. Stop the collector and display telemetry missing, not service healthy. Hand over the request record, admission result and expiry behavior.

Allow ad hoc labels for one debugging session. Design an expiring, separately budgeted diagnostic namespace so temporary investigation does not permanently multiply production series.