A home proxy that chains to a remote tiny-proxy and falls back to the
real upstreams needs two things it did not have.
502, 503 and 504 now fall through to the next upstream in the maven,
debian, pacman and docker walks. They are what an ingress answers when a
chained tiny-proxy is down, and what tiny-proxy itself answers (502
upstream error) when its own upstream fails — either way the object is
not bad, the path to it is. A 500 is still the upstream's own answer and
is returned as-is, as is anything the last upstream says, and a throttle
seen earlier still turns a trailing 404 into a 429.
Docker vrepos take `upstreams:` like the others. Each entry brings its
own client, cache flag and retry budget, and a blob resume goes back to
the upstream that served it. An upstream URL whose path starts with /v2/
is a repository root — another tiny-proxy's vrepo — so the image path is
appended as-is rather than getting a second /v2/. The library/ prefix is
now decided by the upstream's host instead of a substring of its whole
URL, so a chained proxy is not handed library/ twice. When nothing
answers at all the last transport error is returned, so a single
unreachable registry still reads as 502 rather than as a missing image.
github and gitea stay single-upstream: they speak a REST API, not a
path-for-path mirror.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The counters were process-wide globals: one upstream_requests_total, one
cache hit rate, one pair of byte counters. With several vrepos and
fallback upstream lists that answers none of the questions that actually
come up — which mirror is slow, which one is 404-ing everything so every
miss costs an extra round trip, what each one costs in bandwidth.
Everything attributable now carries a label: vrepo for what the client
asked for, vrepo plus the upstream URL for what the proxy had to fetch.
Series are registered from the config at startup, so a label value can
only ever be a configured name — a client asking for random paths lands
in a vrepo="" bucket instead of minting series. Handlers mark the repo
once they have parsed the path and passed auth; proxycache reads it back
off the context, so no call site had to change shape.
Bytes are now counted at the socket, where they are real: each upstream
gets a counting dialer and the server runs on a counting listener, so the
new wire counters include request lines, headers, chunked framing and the
TLS records and handshake. The body counters stay beside them — one is
what the artifact weighs, the other is what the link is billed for. A
cold 68KB jar reads as 68605 body bytes against 75562 on the wire. Client
sockets keep requests for several vrepos on one keep-alive connection, so
those totals stay process-wide rather than split on a guess.
Timings come from httptrace, per wire attempt: connect (DNS included),
TLS handshake, time-to-first-byte and body streaming, plus the inbound
request duration per vrepo. They are histograms, since the average of a
download time hides exactly the tail worth knowing about. Connection
hooks only fire on a real dial, so a quiet connect histogram beside a
busy request counter means the pool is working.
Two knock-on fixes: a transport with a custom DialContext gets no
automatic HTTP/2, so ForceAttemptHTTP2 is now explicit rather than every
upstream silently dropping to HTTP/1.1; and the counting conn implements
ReadFrom so serving a cached file still uses sendfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apt kept failing on a cached .deb with "Error reading from server. Remote
end closed connection", and no amount of retrying fixed it. Two bugs were
compounding.
The tee loop in serveAndStore broke on any read error without inspecting
it, then closed the cache pipe cleanly — so a connection that died
mid-body handed Store.Put a clean EOF and the truncated file was renamed
into place as if it were whole. And metaFromResponse recorded the length
the upstream had advertised rather than the bytes that arrived, so the
sidecar promised more than the file held. Every later hit then set that
Content-Length, streamed fewer bytes, and had its connection cut. Since
released artifacts never expire, the poisoned entry was served forever:
one dropped packet, one permanently broken package.
Now a read error or a short body aborts the pipe with an error, so the
store discards its temp file, and the store records what actually landed.
LocalFS refuses a Put whose body disagrees with its declared size, and
commits the payload before the sidecar so a crash between the two leaves
an invisible object rather than metadata pointing at a missing file. As a
backstop Stat cross-checks each entry against its file and discards any
that disagree — which heals the entries written before this existed,
without manual cleanup.
Retries now cover more than 429. A request that never produced a response
(refused, reset, TLS or DNS failure) is retried the same way, and a
transfer that dies after the headers — past the point any request-level
retry can help — is resumed: proxycache re-requests the remainder from
the upstream that answered, with Range and If-Range, and carries on
streaming where it left off. A resumed response is accepted only as a 206
whose Content-Range starts exactly where the transfer stopped; a 200
means the upstream is restarting from zero, and splicing that onto bytes
already sent would corrupt the object, so it is refused. Handlers build
the resume callback with upstream.Resumer, which binds the client and URL
that served the first byte — a maven or distro resume must never land on
a different mirror mid-file.
Also adds the observability this took to find. Every request gets a trace
id, adopted from the caller's X-Request-Id or traceparent when it sent
one, echoed back, attached to every log record the request produces via a
context-aware slog handler, and forwarded to the upstreams it calls, so
one grep follows a request end to end and across chained proxies. New
counters cover client and upstream requests, bytes in and out on both
sides, upstream errors, retries by reason, resumes by outcome, discarded
transfers and corrupt entries. A log config section tunes level, format
and the two access logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An upstream may now set `cache: false` to be proxied through without
being stored under the vrepo that fetched it. It is meant for chained
setups where the upstream is another vrepo of this proxy: without it
both layers keep their own copy of every artifact.
The decision has to travel with the response — a maven, debian or pacman
vrepo only learns which upstream answered once fallthrough resolves — so
proxycache fetch callbacks now return a Result{Resp, NoStore}. Reads are
unaffected: anything already cached under the vrepo is still served from
storage.
The UI's upstream column now shows each upstream's host and path, since
several vrepos routinely share a host (deb.debian.org/debian vs
/debian-security, chained vrepos of this proxy), and marks the ones that
are not cached.
Also includes an upstream retry transport for 429 responses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MRLokopandClaude Opus 5.5