A 14-upstream aggregate on bunny sent Maven Central over a thousand 429s
an hour: 4463 upstream requests for 1336 distinct URLs, one missing
.module walked through every upstream 42 times. Four things multiplied
the traffic, and each is fixed here.
A miss is remembered. When every upstream says 404 the answer is kept in
memory for cache.not_found_ttl (default 10m, negative disables; capped
at metadata_ttl for SNAPSHOT and metadata paths) and repeated without
asking anyone. Only definite misses count: a throttled, unreachable or
gateway-erroring upstream on the way means the 404 is relayed but not
kept, so an outage is never cached as absence.
A throttled upstream is benched. After a 429 it is skipped for every
path for its Retry-After, clamped to 10s-5m (30s without one), shared by
every vrepo that lists the same URL, egress and account, and only ever
extended. While benched it still counts as rate limited, so a path no
one else serves is a 429, not a 404. Skips are counted in
tinyproxy_upstream_cooldown_skips_total.
A HEAD miss is stored. Maven upstreams are asked with GET, the client
gets the headers at once, and the body lands in the background under the
key lock and a context detached from the request, at most 16 at a time;
past that, and through cache: false upstreams, a HEAD stays a HEAD.
Plain Serve (docker and the rest) is unchanged.
Chained tiny-proxies read each other right. X-Tinyproxy-Answer marks an
unsure 404 (uncertain), so the tier in front does not remember it, and a
relayed 429 (relayed), so it does not bench a proxy that still serves
what it has cached. A 404 from memory carries X-Tinyproxy-Miss-Ttl, and
the tier in front remembers it no longer than that. Answers from memory
are counted in tinyproxy_cache_remembered_misses_total, not as hits.
When an upstream was throttled a trailing 401/403 is also reported as
the 429 the build can retry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MRLokopandClaude Opus 5.5
06d303c58e