Bound retry load and reconcile a lost payment response

Calculate amplification before choosing an SDK policy

Our toy call graph has three layers, each allowing three total attempts, including its original attempt. If every failure traverses all layers, one user request can create 3 × 3 × 3 = 27 leaf attempts. For 100 requests, that is 2,700 attempts. This is a worst-case bound, not a measurement from GitHub. Many real failures stop earlier.

Choose one layer to own retries for a dependency. Give each logical request one absolute deadline; attempts consume the remaining time rather than resetting the clock. Also cap concurrent requests and the aggregate retry rate. An attempt limit per request still permits a huge storm across a million requests.

AWS implementation exercise

For an ECS or Lambda handler calling DynamoDB, inspect the SDK's own retry behavior before adding an outer retry loop. Set a bounded SDK policy and explicit request timeouts. In Python botocore, Config(retries={"mode": "standard", "total_max_attempts": 3}) means three total attempts. Application concurrency, deadline propagation, and retry-rate protection remain separate responsibilities. AWS SDK retry settings.

Use DynamoDB throttling as a test classification, and inject a deterministic timeout in a fake client locally. Do not retry validation errors or access denial indefinitely. For SQS workers, account for SDK attempts, Lambda invocation retries through message visibility, and redrive count as separate layers. The queue lab makes the durable result conditional so a lost acknowledgement does not create a second result item.

Toy budget: base arrival rate 100/s, retry allowance 20/s, downstream safe capacity 150/s. Maximum admitted offered load is 120/s if the budget is enforced globally for this dependency. If 20 processes each independently allow 20 retries/s, the aggregate bound is 500/s. Specify whether a budget is global, per instance, or partitioned.

Operational evidence

Track logical requests separately from attempts: attempts / logical_requests, retries denied, deadline exhaustion, in-flight calls, downstream throttles, and successful user outcomes. If attempts fall while successful outcomes rise, shedding useless work can be helping. A falling error count alone may simply mean you stopped accepting users.

a payment provider times out after charging the customer. Use a stable provider idempotency key and reconciliation query when supported; a local retry counter cannot establish what happened remotely. If no deduplication or status query exists, represent an uncertain outcome for reconciliation instead of claiming exactly-once execution.

Follow-ups that change the design

Senior: conditional 429. Use the 1,500 ms remaining-time example above. Retry only with safe repeat semantics, a token and the required wait plus bounded call time. At 1,499 ms, return/degrade without another attempt. A 400 validation failure or 403 denial does not enter this transient loop. The executable policy fixtures (download file, source below) test all three exclusions and budget exhaustion.

import unittest
import csv
from pathlib import Path
from fractions import Fraction as F
from model import sli, burn, alarm, latched, leaf_attempts, retry_allowed, admit, capacity, drain


class ReliabilityTests(unittest.TestCase):
    def test_incident_csv_windows_and_observed_queue(self):
        with (Path(__file__).parent/'incident.csv').open() as f:
            rows=[{k:int(v) for k,v in row.items()} for row in csv.DictReader(f)]
        def window(start,end):
            return [(r['good_reads'],r['eligible_reads']) for r in rows[start:end+1]]
        self.assertEqual(burn(window(3,4)),45)
        self.assertEqual(burn(window(0,4)),22)
        self.assertTrue(alarm(burn(window(3,4)),burn(window(0,4))))
        self.assertEqual(burn(window(5,6)),0)
        self.assertEqual(burn(window(2,6)),22)
        self.assertFalse(alarm(burn(window(5,6)),burn(window(2,6))))
        for previous,current in zip(rows,rows[1:]):
            growth=(current['offered_attempts_per_s']-current['completed_attempts_per_s'])*60
            self.assertEqual(current['queue_items']-previous['queue_items'],growth)
        self.assertEqual(drain(rows[5]['queue_items'],80,100),1020)

    def test_variable_traffic_budget_not_minutes(self):
        rows = [(990000, 990000), (0, 10000)]
        self.assertEqual(sli(rows), F(99,100))
        self.assertEqual(burn(rows), 10)
        self.assertNotEqual(sli(rows), F(1,2))  # averaging interval ratios is wrong
        self.assertEqual(F(30*24*60,1000), F(216,5))  # 43.2 time-based minutes only

    def test_no_eligible_requests_and_invalid_counters(self):
        self.assertIsNone(sli([(0,0)]))
        self.assertIsNone(burn([]))
        self.assertIsNone(alarm(None,20))
        self.assertFalse(alarm(None,0))
        with self.assertRaises(ValueError):
            sli([(2,1)])

    def test_boolean_recovery_both_directions_and_latch(self):
        self.assertTrue(alarm(20,20))
        self.assertFalse(alarm(0,20))
        self.assertFalse(alarm(20,0))
        self.assertFalse(alarm(0,0))
        self.assertTrue(latched(True,False,True))
        self.assertTrue(latched(True,True,False))
        self.assertFalse(latched(True,False,False))
        self.assertFalse(latched(False,False,True))

    def test_attempt_units_and_layer_scope(self):
        self.assertEqual(leaf_attempts([3,3,3]),27)
        self.assertEqual(leaf_attempts([3,3,3],counts='retries'),64)
        self.assertEqual(leaf_attempts([1,3,1]),3)
        self.assertEqual(100+20*20,500)  # per-process is not global retry budget

    def test_priority_cannot_create_capacity(self):
        a=admit(120,30,100)
        self.assertEqual((a.critical,a.bulk,a.critical_refused,a.bulk_refused),(100,0,20,30))
        a=admit(80,40,100)
        self.assertEqual((a.critical,a.bulk,a.critical_refused,a.bulk_refused),(80,20,0,20))

    def test_retry_policy_has_countermodels(self):
        p=dict(safe=True,remaining_ms=1500,wait_ms=1000,attempt_ms=500,tokens=1)
        self.assertTrue(retry_allowed(429,**p))
        for status in (400,401,403,404,422):
            self.assertFalse(retry_allowed(status,**p))
        for changed in ({'safe':False},{'tokens':0},{'remaining_ms':1499}):
            self.assertFalse(retry_allowed(429,**(p|changed)))

    def test_queue_growth_recovery_and_closed_loop(self):
        self.assertEqual(capacity(20,.2),100)
        self.assertEqual(capacity(20,2),10)
        backlog=(80-10)*60
        self.assertEqual(backlog,4200)
        self.assertEqual(drain(backlog,80,100),210)
        self.assertIsNone(drain(backlog,100,100))
        self.assertEqual(capacity(20,2),10)  # 20 closed-loop clients self-throttle to 10/s
        self.assertEqual((80-capacity(20,2))*60,4200)  # open arrivals still accumulate


if __name__=='__main__':
    unittest.main(verbosity=2)

Lead: the payment succeeded but acknowledgement was lost. A stable provider key can collapse repeated attempts when the provider supports it; otherwise an uncertain local outcome must trigger reconciliation, not an invented failure followed by a new payment. Predict the action after the timeout before reading the state transitions.

For the cross-team version, twenty independently enforced 20/s budgets permit 100+20×20=500/s. A 150/s dependency cannot accept that bound. Partition a total 20/s retry allowance, or enforce one authoritative budget near the dependency; define behavior when budget coordination fails. A per-client SDK bucket is not that authority.

Run python -m unittest discover -s curriculum/03-production/05-reliability/labs/reliability -p 'test_*.py' -v from the repository root. These local models verify counts and policy, not remote payment behavior. For the build extension, submit logs linking logical ID to attempt ID, a lost-ack test and an explicit uncertain-state recovery; never present a fake client as a real provider guarantee.