dirBrowse rendered a single directory level but paid for a full
recursive listing of the repo — on CephFS-backed LocalFS that is a
walk of the entire storage root plus a sidecar read per object, which
took Ceph down on large repos. Storage grows a ListDir primitive
(os.ReadDir on localfs, Delimiter on S3) and the UI and the debian
dists fallback use it; the recursive List stays for the docker page
and now starts its walk at the requested prefix.
The per-vrepo storage gauges on /metrics did the same full listing on
every scrape; they are removed — the backend's own accounting (du,
bucket metrics) covers capacity.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The counters were process-wide globals: one upstream_requests_total, one
cache hit rate, one pair of byte counters. With several vrepos and
fallback upstream lists that answers none of the questions that actually
come up — which mirror is slow, which one is 404-ing everything so every
miss costs an extra round trip, what each one costs in bandwidth.
Everything attributable now carries a label: vrepo for what the client
asked for, vrepo plus the upstream URL for what the proxy had to fetch.
Series are registered from the config at startup, so a label value can
only ever be a configured name — a client asking for random paths lands
in a vrepo="" bucket instead of minting series. Handlers mark the repo
once they have parsed the path and passed auth; proxycache reads it back
off the context, so no call site had to change shape.
Bytes are now counted at the socket, where they are real: each upstream
gets a counting dialer and the server runs on a counting listener, so the
new wire counters include request lines, headers, chunked framing and the
TLS records and handshake. The body counters stay beside them — one is
what the artifact weighs, the other is what the link is billed for. A
cold 68KB jar reads as 68605 body bytes against 75562 on the wire. Client
sockets keep requests for several vrepos on one keep-alive connection, so
those totals stay process-wide rather than split on a guess.
Timings come from httptrace, per wire attempt: connect (DNS included),
TLS handshake, time-to-first-byte and body streaming, plus the inbound
request duration per vrepo. They are histograms, since the average of a
download time hides exactly the tail worth knowing about. Connection
hooks only fire on a real dial, so a quiet connect histogram beside a
busy request counter means the pool is working.
Two knock-on fixes: a transport with a custom DialContext gets no
automatic HTTP/2, so ForceAttemptHTTP2 is now explicit rather than every
upstream silently dropping to HTTP/1.1; and the counting conn implements
ReadFrom so serving a cached file still uses sendfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
log.upstream is an explicit switch, so turning it on should be enough to
see the lines — dropping the whole process to debug to get them defeats
the point of having the switch. At the default level the proxy now emits
one line per finished client request and one per upstream attempt, which
is what makes a trace id useful. Debug adds the request-arrival line,
which only matters for a request that never finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apt kept failing on a cached .deb with "Error reading from server. Remote
end closed connection", and no amount of retrying fixed it. Two bugs were
compounding.
The tee loop in serveAndStore broke on any read error without inspecting
it, then closed the cache pipe cleanly — so a connection that died
mid-body handed Store.Put a clean EOF and the truncated file was renamed
into place as if it were whole. And metaFromResponse recorded the length
the upstream had advertised rather than the bytes that arrived, so the
sidecar promised more than the file held. Every later hit then set that
Content-Length, streamed fewer bytes, and had its connection cut. Since
released artifacts never expire, the poisoned entry was served forever:
one dropped packet, one permanently broken package.
Now a read error or a short body aborts the pipe with an error, so the
store discards its temp file, and the store records what actually landed.
LocalFS refuses a Put whose body disagrees with its declared size, and
commits the payload before the sidecar so a crash between the two leaves
an invisible object rather than metadata pointing at a missing file. As a
backstop Stat cross-checks each entry against its file and discards any
that disagree — which heals the entries written before this existed,
without manual cleanup.
Retries now cover more than 429. A request that never produced a response
(refused, reset, TLS or DNS failure) is retried the same way, and a
transfer that dies after the headers — past the point any request-level
retry can help — is resumed: proxycache re-requests the remainder from
the upstream that answered, with Range and If-Range, and carries on
streaming where it left off. A resumed response is accepted only as a 206
whose Content-Range starts exactly where the transfer stopped; a 200
means the upstream is restarting from zero, and splicing that onto bytes
already sent would corrupt the object, so it is refused. Handlers build
the resume callback with upstream.Resumer, which binds the client and URL
that served the first byte — a maven or distro resume must never land on
a different mirror mid-file.
Also adds the observability this took to find. Every request gets a trace
id, adopted from the caller's X-Request-Id or traceparent when it sent
one, echoed back, attached to every log record the request produces via a
context-aware slog handler, and forwarded to the upstreams it calls, so
one grep follows a request end to end and across chained proxies. New
counters cover client and upstream requests, bytes in and out on both
sides, upstream errors, retries by reason, resumes by outcome, discarded
transfers and corrupt entries. A log config section tunes level, format
and the two access logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An upstream may now set `cache: false` to be proxied through without
being stored under the vrepo that fetched it. It is meant for chained
setups where the upstream is another vrepo of this proxy: without it
both layers keep their own copy of every artifact.
The decision has to travel with the response — a maven, debian or pacman
vrepo only learns which upstream answered once fallthrough resolves — so
proxycache fetch callbacks now return a Result{Resp, NoStore}. Reads are
unaffected: anything already cached under the vrepo is still served from
storage.
The UI's upstream column now shows each upstream's host and path, since
several vrepos routinely share a host (deb.debian.org/debian vs
/debian-security, chained vrepos of this proxy), and marks the ones that
are not cached.
Also includes an upstream retry transport for 429 responses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some registries (Gitea packages, private repos) answer 401/403 for every
anonymous read instead of 404. Returning that as-is broke aggregation for
any path not served by an earlier upstream. Mirror Reposilite's behavior:
treat 401/403 like 404 when more eligible upstreams remain; the last
upstream's status is still returned as-is.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Maven vrepos now accept an upstreams list tried in order: 404s and
connection errors fall through to the next upstream. Each upstream can
be restricted with wildcard group prefixes, artifact name globs, or
snapshots_only. Also includes github handler, upstream client and UI
updates.
Co-Authored-By: Claude Code <noreply@anthropic.com>
If Store.Put returns an error before draining the pipe (e.g. unwritable
cache dir), the tee writer blocked forever and the client hung with zero
bytes. Close the pipe reader on Put error, and replace io.TeeReader with
a hand-rolled loop: TeeReader consumes an upstream chunk before the pipe
write reports the failure, silently dropping it from the client stream.
Caching is now best-effort — the client always gets the full body.
Co-Authored-By: Claude Code <noreply@anthropic.com>
MRLokopandClaude Code