""" Models what a 529 overloaded_error does to a fan-out of agent requests, and why the retry policy you pick decides whether the burst drains or amplifies. Artifact for /answers/anthropic-529-overloaded-retry. No network, no API key: this is a discrete-time simulation of a capacity-limited endpoint, not a measurement of Anthropic's real capacity or 529 rate (no editor publishes that, and this page does not pretend to). What it reproduces deterministically is the *retry dynamics*: fire N requests in one burst at an endpoint that can only serve C per tick, and watch each retry policy either drain the backlog or beat the same wall in lockstep. Model: - N tasks each need ONE successful request. All fire at tick 0 (the fan-out burst: an orchestrator spawning N subagents at once). - The endpoint serves at most C requests per tick. Arrivals beyond C that tick get a 529 and retry per policy. The C served are the lowest task ids, so the run is reproducible byte for byte. - A task that exhausts its attempt budget is dropped (surfaces to the user as a hard 529, the failure the issues on this page report). Retry policies compared, all starting from the same base delay: A. none - one attempt, no retry (SDK max_retries=0 / a wrapper that swallows the SDK's own retry) B. fixed, no jitter - retry every `base` ticks C. exponential, no jitter - base * 2**(attempt-1); the textbook answer, and the one that keeps the fleet synchronized D. exponential + full jitter - uniform(0, base * 2**(attempt-1)); breaks lockstep Run: python anthropic-529-retry-budget.py Measured output (seed=1729, N=64, C=8, base=1 tick, horizon=200): max_attempts = 3 (the Anthropic SDK default: max_retries=2) policy | served | dropped | attempts | ticks to drain --------------------------|--------|---------|----------|--------------- A. none | 8 | 56 | 64 | did not drain B. fixed, no jitter | 24 | 40 | 168 | did not drain C. exponential, no jitter | 24 | 40 | 168 | did not drain D. exp + full jitter | 24 | 40 | 168 | did not drain max_attempts = 8 policy | served | dropped | attempts | ticks to drain --------------------------|--------|---------|----------|--------------- A. none | 8 | 56 | 64 | did not drain B. fixed, no jitter | 64 | 0 | 288 | 7 C. exponential, no jitter | 64 | 0 | 288 | 127 D. exp + full jitter | 64 | 0 | 225 | 14 What it establishes: 1. At the SDK's default 3 attempts, the backoff *shape* is irrelevant. B, C and D all land on 24 served / 40 dropped / 168 attempts - identical - because the budget, not the timing, is the binding constraint: every request that fails burns all three attempts whenever it makes them. A 64-wide burst against an 8/tick endpoint drops 40 tasks no matter how clever the backoff. The lever that saves the run is raising the attempt budget, and only then does timing start to matter. 2. Once the budget is large enough to drain (8 attempts), the policies separate on *cost*. Full jitter clears the burst in the fewest total attempts (225 vs 288), because spreading retries across ticks keeps the served slots full every tick instead of arriving in colliding waves. Fewer attempts is less load you put back on the endpoint you are trying to unblock. 3. Exponential-no-jitter is the slowest to drain (127 ticks vs 14 for jitter): every request at the same attempt count retries on the same tick, so the fleet moves in lockstep and the waves get further apart as the exponent grows, leaving capacity idle between them. Backoff without jitter trades a retry storm for a retry convoy. 4. "none" is the shape of every reported bug: 8 served, 56 dropped, and the 56 surface as hard 529s mid-task. That is a client that never retried, or a wrapper that ate the SDK's retry - not the API being down. """ import random def next_delay(policy: str, attempt: int, base: int, rng: random.Random) -> int: """Ticks to wait before the next attempt, given the attempt just failed.""" if policy == "fixed": return base exp = base * (2 ** (attempt - 1)) if policy == "exponential": return exp if policy == "jitter": # Full jitter: uniform(0, exp), floored at 1 tick so it always advances. return max(1, int(rng.uniform(0, exp))) raise ValueError(policy) def simulate(policy: str, n: int, capacity: int, base: int, max_attempts: int, horizon: int, seed: int) -> dict: rng = random.Random(seed) # scheduled[tick] = list of (task_id, attempt_number_about_to_be_made) scheduled: dict[int, list[tuple[int, int]]] = {0: [(i, 1) for i in range(n)]} served = 0 attempts = 0 drain_tick = None for tick in range(horizon): arrivals = scheduled.pop(tick, []) if not arrivals: continue # Deterministic tie-break: lowest task id wins a served slot this tick. arrivals.sort() for rank, (task_id, attempt) in enumerate(arrivals): attempts += 1 if rank < capacity: served += 1 # this attempt got a slot: task done continue # 529 for this attempt. Retry if budget remains. if attempt >= max_attempts: continue # dropped: surfaces as a hard 529 delay = next_delay(policy, attempt, base, rng) scheduled.setdefault(tick + delay, []).append((task_id, attempt + 1)) if served == n and drain_tick is None: drain_tick = tick return { "served": served, "dropped": n - served, "attempts": attempts, "drain": drain_tick, } def run_table(max_attempts: int, n: int, capacity: int, base: int, horizon: int, seed: int) -> None: policies = [ ("A. none", "none", 1), ("B. fixed, no jitter", "fixed", max_attempts), ("C. exponential, no jitter", "exponential", max_attempts), ("D. exp + full jitter", "jitter", max_attempts), ] print(f"max_attempts = {max_attempts}") print(f"{'policy':<26}| served | dropped | attempts | ticks to drain") print(f"{'-' * 26}|--------|---------|----------|---------------") for label, policy, attempts_cap in policies: r = simulate(policy, n, capacity, base, attempts_cap, horizon, seed) drain = r["drain"] if r["drain"] is not None else "did not drain" print(f"{label:<26}| {r['served']:>6} | {r['dropped']:>7} | " f"{r['attempts']:>8} | {drain}") print() if __name__ == "__main__": SEED, N, CAPACITY, BASE, HORIZON = 1729, 64, 8, 1, 200 print() print(f"N={N} fan-out requests, endpoint serves C={CAPACITY}/tick, " f"base delay={BASE} tick, seed={SEED}") print() run_table(max_attempts=3, n=N, capacity=CAPACITY, base=BASE, horizon=HORIZON, seed=SEED) run_table(max_attempts=8, n=N, capacity=CAPACITY, base=BASE, horizon=HORIZON, seed=SEED) print("A dropped count is requests that surfaced as a hard 529 to the user.") print("At 3 attempts the budget is the whole story: B, C and D tie. At 8, they") print("separate on cost - jitter drains in 225 attempts, exponential-no-jitter") print("in 288 and 127 ticks. Raise the budget first, then jitter it.")