A 14-upstream aggregate on bunny sent Maven Central over a thousand 429s
an hour: 4463 upstream requests for 1336 distinct URLs, one missing
.module walked through every upstream 42 times. Four things multiplied
the traffic, and each is fixed here.
A miss is remembered. When every upstream says 404 the answer is kept in
memory for cache.not_found_ttl (default 10m, negative disables; capped
at metadata_ttl for SNAPSHOT and metadata paths) and repeated without
asking anyone. Only definite misses count: a throttled, unreachable or
gateway-erroring upstream on the way means the 404 is relayed but not
kept, so an outage is never cached as absence.
A throttled upstream is benched. After a 429 it is skipped for every
path for its Retry-After, clamped to 10s-5m (30s without one), shared by
every vrepo that lists the same URL, egress and account, and only ever
extended. While benched it still counts as rate limited, so a path no
one else serves is a 429, not a 404. Skips are counted in
tinyproxy_upstream_cooldown_skips_total.
A HEAD miss is stored. Maven upstreams are asked with GET, the client
gets the headers at once, and the body lands in the background under the
key lock and a context detached from the request, at most 16 at a time;
past that, and through cache: false upstreams, a HEAD stays a HEAD.
Plain Serve (docker and the rest) is unchanged.
Chained tiny-proxies read each other right. X-Tinyproxy-Answer marks an
unsure 404 (uncertain), so the tier in front does not remember it, and a
relayed 429 (relayed), so it does not bench a proxy that still serves
what it has cached. A 404 from memory carries X-Tinyproxy-Miss-Ttl, and
the tier in front remembers it no longer than that. Answers from memory
are counted in tinyproxy_cache_remembered_misses_total, not as hits.
When an upstream was throttled a trailing 401/403 is also reported as
the 429 the build can retry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The counters were process-wide globals: one upstream_requests_total, one
cache hit rate, one pair of byte counters. With several vrepos and
fallback upstream lists that answers none of the questions that actually
come up — which mirror is slow, which one is 404-ing everything so every
miss costs an extra round trip, what each one costs in bandwidth.
Everything attributable now carries a label: vrepo for what the client
asked for, vrepo plus the upstream URL for what the proxy had to fetch.
Series are registered from the config at startup, so a label value can
only ever be a configured name — a client asking for random paths lands
in a vrepo="" bucket instead of minting series. Handlers mark the repo
once they have parsed the path and passed auth; proxycache reads it back
off the context, so no call site had to change shape.
Bytes are now counted at the socket, where they are real: each upstream
gets a counting dialer and the server runs on a counting listener, so the
new wire counters include request lines, headers, chunked framing and the
TLS records and handshake. The body counters stay beside them — one is
what the artifact weighs, the other is what the link is billed for. A
cold 68KB jar reads as 68605 body bytes against 75562 on the wire. Client
sockets keep requests for several vrepos on one keep-alive connection, so
those totals stay process-wide rather than split on a guess.
Timings come from httptrace, per wire attempt: connect (DNS included),
TLS handshake, time-to-first-byte and body streaming, plus the inbound
request duration per vrepo. They are histograms, since the average of a
download time hides exactly the tail worth knowing about. Connection
hooks only fire on a real dial, so a quiet connect histogram beside a
busy request counter means the pool is working.
Two knock-on fixes: a transport with a custom DialContext gets no
automatic HTTP/2, so ForceAttemptHTTP2 is now explicit rather than every
upstream silently dropping to HTTP/1.1; and the counting conn implements
ReadFrom so serving a cached file still uses sendfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apt kept failing on a cached .deb with "Error reading from server. Remote
end closed connection", and no amount of retrying fixed it. Two bugs were
compounding.
The tee loop in serveAndStore broke on any read error without inspecting
it, then closed the cache pipe cleanly — so a connection that died
mid-body handed Store.Put a clean EOF and the truncated file was renamed
into place as if it were whole. And metaFromResponse recorded the length
the upstream had advertised rather than the bytes that arrived, so the
sidecar promised more than the file held. Every later hit then set that
Content-Length, streamed fewer bytes, and had its connection cut. Since
released artifacts never expire, the poisoned entry was served forever:
one dropped packet, one permanently broken package.
Now a read error or a short body aborts the pipe with an error, so the
store discards its temp file, and the store records what actually landed.
LocalFS refuses a Put whose body disagrees with its declared size, and
commits the payload before the sidecar so a crash between the two leaves
an invisible object rather than metadata pointing at a missing file. As a
backstop Stat cross-checks each entry against its file and discards any
that disagree — which heals the entries written before this existed,
without manual cleanup.
Retries now cover more than 429. A request that never produced a response
(refused, reset, TLS or DNS failure) is retried the same way, and a
transfer that dies after the headers — past the point any request-level
retry can help — is resumed: proxycache re-requests the remainder from
the upstream that answered, with Range and If-Range, and carries on
streaming where it left off. A resumed response is accepted only as a 206
whose Content-Range starts exactly where the transfer stopped; a 200
means the upstream is restarting from zero, and splicing that onto bytes
already sent would corrupt the object, so it is refused. Handlers build
the resume callback with upstream.Resumer, which binds the client and URL
that served the first byte — a maven or distro resume must never land on
a different mirror mid-file.
Also adds the observability this took to find. Every request gets a trace
id, adopted from the caller's X-Request-Id or traceparent when it sent
one, echoed back, attached to every log record the request produces via a
context-aware slog handler, and forwarded to the upstreams it calls, so
one grep follows a request end to end and across chained proxies. New
counters cover client and upstream requests, bytes in and out on both
sides, upstream errors, retries by reason, resumes by outcome, discarded
transfers and corrupt entries. A log config section tunes level, format
and the two access logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An upstream may now set `cache: false` to be proxied through without
being stored under the vrepo that fetched it. It is meant for chained
setups where the upstream is another vrepo of this proxy: without it
both layers keep their own copy of every artifact.
The decision has to travel with the response — a maven, debian or pacman
vrepo only learns which upstream answered once fallthrough resolves — so
proxycache fetch callbacks now return a Result{Resp, NoStore}. Reads are
unaffected: anything already cached under the vrepo is still served from
storage.
The UI's upstream column now shows each upstream's host and path, since
several vrepos routinely share a host (deb.debian.org/debian vs
/debian-security, chained vrepos of this proxy), and marks the ones that
are not cached.
Also includes an upstream retry transport for 429 responses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
If Store.Put returns an error before draining the pipe (e.g. unwritable
cache dir), the tee writer blocked forever and the client hung with zero
bytes. Close the pipe reader on Put error, and replace io.TeeReader with
a hand-rolled loop: TeeReader consumes an upstream chunk before the pipe
write reports the failure, silently dropping it from the client stream.
Caching is now best-effort — the client always gets the full body.
Co-Authored-By: Claude Code <noreply@anthropic.com>
MRLokopandClaude Opus 5.5