Release invoice changes with stable cohorts and rollback

Application background

A subscription product has a new way to calculate invoices. The team wants a small set of customers to use it first while everyone else keeps the old behavior. A feature setting decides which version each customer receives after the code is deployed.

Customers should not switch unpredictably between versions on every request. Also, an interactive purchase might run immediately while the monthly renewal job runs hours later. A healthy web request does not demonstrate that delayed work is healthy.

Example walkthrough

A cohort is the selected group of customers. Deployment installs code. Rollout decides who uses a behavior. Rollback must consider any new data the code has already written.

Your assignment

Deliver: Build stable customer selection for a new invoice calculation. Version the selection settings and demonstrate rollback for both immediate requests and delayed renewal jobs.

Required behavior: Evaluate a versioned flag against a stable subject identity. Exposure is deterministic for a given flag salt and subject. Turning the flag off restores the supported old path. It cannot undo irreversible writes already made by the new path.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/feature_rollout.py

Supplied file: examples/architecture-starts/feature_rollout.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for feature-rollout. No AWS resources are created."""
import hashlib
def exposed(subject,percent):
    bucket=int(hashlib.sha256(('invoice-v1:'+subject).encode()).hexdigest(),16)%10000
    return bucket<percent*100
users=[f'user-{i}' for i in range(200)]
users=[u for u in users if exposed(u,5)][:2]+[u for u in users if exposed(u,25) and not exposed(u,5)][:2]
for user in users:
    print(user,'5%',exposed(user,5),'25%',exposed(user,25),'repeat',exposed(user,5))
print('Emergency off:',[exposed(u,0) for u in ['ana','ben','chen']])

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

user-4 5% True 25% True repeat True
user-11 5% True 25% True repeat True
user-5 5% False 25% True repeat False
user-9 5% False 25% True repeat False
… (more output follows)

Set up your implementation workspace

Create work/feature-rollout/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Keep both paths compatible

Introduce the new code behind a flag while retaining the old reader/writer contract. Inventory web requests, scheduled jobs, retries and admin tools that use the behavior. Record any data change the old code cannot understand before calling the flag a rollback mechanism.

2. Assign and record stable cohorts

Hash a stable subject with a fixed flag salt. Increasing percentage should include the previous cohort. Record actual exposure at execution time, not only configuration assignment. Keep employee overrides and emergency-off precedence explicit.

3. Observe the relevant work cycle

Compare errors, latency and business outcomes for equivalent exposed/control cohorts. Wait through the daily renewal path and representative traffic before widening exposure. Low traffic may require a longer observation window rather than a claim that zero errors proves safety.

4. Exercise rollback and cleanup

Turn the flag off and confirm the old path processes records written during exposure. Reconcile external effects that cannot be undone. After adoption, remove dead code and retire the flag under a named owner/date so configurations do not accumulate indefinitely.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

AppConfig distributes the policy. Application code evaluates exposure and preserves compatible behavior. A flag cannot reverse an external charge or repair incompatible stored data.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Change experiment cohorts without mixing two experiments

A user who saw the treatment yesterday may become control today. Combining both exposures under one experiment name makes outcomes difficult to attribute and can hide incompatibility between services.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Persist experiment ID, salt version, allocation and behavior version in configuration. Record actual exposure, not just eligibility. Treat a salt change as a new experiment and keep old outcome records attached to the original assignment. During a cross-service rollout, accept compatible configuration versions before increasing exposure. Retain an emergency disable independent of cohort assignment.

Walk through the result. Compute assignments for the same small user set under salts 1 and 2 and display who moved. An outcome caused by a salt-1 exposure remains in experiment 1. Roll one service back while another still runs the newer reader and demonstrate that the shared contract remains valid. Deliver the exposure record and version compatibility table.

Let product change the hash salt mid-rollout. Explain cohort churn and experiment contamination, then make salt changes an explicit new experiment rather than a harmless configuration edit.

import hashlib
import json

def bucket(account_id: str, assignment_salt: str) -> int:
    payload = json.dumps([account_id, assignment_salt],
                         ensure_ascii=False, separators=(",", ":")).encode("utf-8")
    return int.from_bytes(hashlib.sha256(payload).digest()[:8], "big") % 10000

accounts = [f"account-{i}" for i in range(1000)]
assigned = {a: bucket(a, "checkout-A") for a in accounts}
cohort_5 = {a for a, value in assigned.items() if value < 500}
cohort_50 = {a for a, value in assigned.items() if value < 5000}
assert cohort_5 and cohort_5 <= cohort_50
assert all(bucket(a, "checkout-A") == assigned[a] for a in accounts)
assert all(0 <= value < 10000 for value in assigned.values())
assert any(bucket(a, "checkout-B") != assigned[a] for a in accounts)