Measure whether existing checks detect real defects

Application background

A bookmark application already has automated checks that report success. Its maintainers want to know whether those checks would actually notice a serious mistake, such as letting Bob edit Alice's bookmark. A green result alone does not answer that question.

In a disposable copy, deliberately introduce one small defect at a time and observe which existing checks detect it. This is an exercise in evaluating the evidence you already have. The result is a report about detected and missed behavior changes.

Example walkthrough

A mutation is a deliberate small code change used to challenge the checks. Sensitivity describes whether the available evidence responds when the protected behavior breaks.

Your assignment

Deliver: Produce a report showing which deliberate defects the existing checks detect and which they miss. Do the experiment in a disposable copy and keep each observation tied to the change that caused it.

Required behavior: Distinguish meaningful behavior changes from equivalent edits. Report which selected defects are detected, which survive and which time out. A percentage over a small selected set is evidence about that set, not proof of all correctness.

The primary deliverable is the report or operational procedure named above, backed by a reproducible local demonstration. Build the smallest supporting code needed to make that evidence visible.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/01_the_suite_that_can_fail.py

Supplied file: examples/architecture-starts/01_the_suite_that_can_fail.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for 01-the-suite-that-can-fail. No AWS resources are created."""
records=[('owner filter','semantic','detected'),('expiry boundary','semantic','survived'),('comment','equivalent','excluded'),('local rename','equivalent','excluded')]
relevant=[r for r in records if r[1]=='semantic']
print('Selected semantic defects:',len(relevant))
print('Detected:',sum(r[2]=='detected' for r in relevant))
print('Needs investigation:',[r[0] for r in relevant if r[2]=='survived'])

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

Selected semantic defects: 2
Detected: 1
Needs investigation: ['expiry boundary']

Set up your implementation workspace

Create work/01-the-suite-that-can-fail/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Choose the protected behavior

Open an existing ownership/expiry implementation and write its exact examples: Ben must not see Ana’s record. A record is expired at now >= expires_at. Keep the original program and environment fixed while changing one behavior at a time.

2. Classify candidate edits

Remove the owner predicate and change the expiry comparison independently. Exclude comments and equivalent renames from the semantic denominator. Inspect whether the edit is reachable and actually changes the contract before calling it a surviving defect.

3. Capture isolated outcomes

Run each selected version in a disposable directory with a fixed input and deadline. Record normal failure, timeout, execution error and survival separately. Keep the smallest input that exposes a missed behavior. Do not manufacture a failure when the current evidence already catches the defect.

4. Write the engineering conclusion

Name the uncovered boundary and the evidence needed to protect it. Report sampled scope and limitations. Keep this as a learning exercise and review artifact. No CI requirement or automatic website-publishing check is introduced.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

The cloud version is a bounded experiment runner with durable evidence. S3 or ECS does not decide whether a mutation is meaningful. That judgment comes from the stated behavior contract.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

All twenty selected defects are detected. Report that result and the sampled risks honestly. Expand coverage only for a concrete uncovered behavior, not to meet a quota of failures.