A go vrepo serves the GOPROXY protocol at /go/<vrepo>/. Its upstreams
are ordered rules matched by module path (GOPRIVATE-style patterns);
each one relays to another module proxy, or builds .info/.mod/.zip,
@v/list and @latest itself from a git repository: through the Gitea
or GitLab REST API, or over git smart-HTTP (go-git) for any other
host, without a git binary. Module layout follows the go command:
<dir>/vX.Y.Z tags, /vN major subdirectories, +incompatible, nested
modules and vendor/ left out of zips. go.mod is served as committed,
so forks keeping their upstream module path work through replace.
Repo templates map captured pattern elements to repository paths
({1}, {2}...). vanity_hosts answers ?go-get=1 with a go-import meta
tag pointing at this proxy or at the repository.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 14-upstream aggregate on bunny sent Maven Central over a thousand 429s
an hour: 4463 upstream requests for 1336 distinct URLs, one missing
.module walked through every upstream 42 times. Four things multiplied
the traffic, and each is fixed here.
A miss is remembered. When every upstream says 404 the answer is kept in
memory for cache.not_found_ttl (default 10m, negative disables; capped
at metadata_ttl for SNAPSHOT and metadata paths) and repeated without
asking anyone. Only definite misses count: a throttled, unreachable or
gateway-erroring upstream on the way means the 404 is relayed but not
kept, so an outage is never cached as absence.
A throttled upstream is benched. After a 429 it is skipped for every
path for its Retry-After, clamped to 10s-5m (30s without one), shared by
every vrepo that lists the same URL, egress and account, and only ever
extended. While benched it still counts as rate limited, so a path no
one else serves is a 429, not a 404. Skips are counted in
tinyproxy_upstream_cooldown_skips_total.
A HEAD miss is stored. Maven upstreams are asked with GET, the client
gets the headers at once, and the body lands in the background under the
key lock and a context detached from the request, at most 16 at a time;
past that, and through cache: false upstreams, a HEAD stays a HEAD.
Plain Serve (docker and the rest) is unchanged.
Chained tiny-proxies read each other right. X-Tinyproxy-Answer marks an
unsure 404 (uncertain), so the tier in front does not remember it, and a
relayed 429 (relayed), so it does not bench a proxy that still serves
what it has cached. A 404 from memory carries X-Tinyproxy-Miss-Ttl, and
the tier in front remembers it no longer than that. Answers from memory
are counted in tinyproxy_cache_remembered_misses_total, not as hits.
When an upstream was throttled a trailing 401/403 is also reported as
the 429 the build can retry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A home proxy that chains to a remote tiny-proxy and falls back to the
real upstreams needs two things it did not have.
502, 503 and 504 now fall through to the next upstream in the maven,
debian, pacman and docker walks. They are what an ingress answers when a
chained tiny-proxy is down, and what tiny-proxy itself answers (502
upstream error) when its own upstream fails — either way the object is
not bad, the path to it is. A 500 is still the upstream's own answer and
is returned as-is, as is anything the last upstream says, and a throttle
seen earlier still turns a trailing 404 into a 429.
Docker vrepos take `upstreams:` like the others. Each entry brings its
own client, cache flag and retry budget, and a blob resume goes back to
the upstream that served it. An upstream URL whose path starts with /v2/
is a repository root — another tiny-proxy's vrepo — so the image path is
appended as-is rather than getting a second /v2/. The library/ prefix is
now decided by the upstream's host instead of a substring of its whole
URL, so a chained proxy is not handed library/ twice. When nothing
answers at all the last transport error is returned, so a single
unreachable registry still reads as 502 rather than as a missing image.
github and gitea stay single-upstream: they speak a REST API, not a
path-for-path mirror.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dirBrowse rendered a single directory level but paid for a full
recursive listing of the repo — on CephFS-backed LocalFS that is a
walk of the entire storage root plus a sidecar read per object, which
took Ceph down on large repos. Storage grows a ListDir primitive
(os.ReadDir on localfs, Delimiter on S3) and the UI and the debian
dists fallback use it; the recursive List stays for the docker page
and now starts its walk at the requested prefix.
The per-vrepo storage gauges on /metrics did the same full listing on
every scrape; they are removed — the backend's own accounting (du,
bucket metrics) covers capacity.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The counters were process-wide globals: one upstream_requests_total, one
cache hit rate, one pair of byte counters. With several vrepos and
fallback upstream lists that answers none of the questions that actually
come up — which mirror is slow, which one is 404-ing everything so every
miss costs an extra round trip, what each one costs in bandwidth.
Everything attributable now carries a label: vrepo for what the client
asked for, vrepo plus the upstream URL for what the proxy had to fetch.
Series are registered from the config at startup, so a label value can
only ever be a configured name — a client asking for random paths lands
in a vrepo="" bucket instead of minting series. Handlers mark the repo
once they have parsed the path and passed auth; proxycache reads it back
off the context, so no call site had to change shape.
Bytes are now counted at the socket, where they are real: each upstream
gets a counting dialer and the server runs on a counting listener, so the
new wire counters include request lines, headers, chunked framing and the
TLS records and handshake. The body counters stay beside them — one is
what the artifact weighs, the other is what the link is billed for. A
cold 68KB jar reads as 68605 body bytes against 75562 on the wire. Client
sockets keep requests for several vrepos on one keep-alive connection, so
those totals stay process-wide rather than split on a guess.
Timings come from httptrace, per wire attempt: connect (DNS included),
TLS handshake, time-to-first-byte and body streaming, plus the inbound
request duration per vrepo. They are histograms, since the average of a
download time hides exactly the tail worth knowing about. Connection
hooks only fire on a real dial, so a quiet connect histogram beside a
busy request counter means the pool is working.
Two knock-on fixes: a transport with a custom DialContext gets no
automatic HTTP/2, so ForceAttemptHTTP2 is now explicit rather than every
upstream silently dropping to HTTP/1.1; and the counting conn implements
ReadFrom so serving a cached file still uses sendfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
log.upstream is an explicit switch, so turning it on should be enough to
see the lines — dropping the whole process to debug to get them defeats
the point of having the switch. At the default level the proxy now emits
one line per finished client request and one per upstream attempt, which
is what makes a trace id useful. Debug adds the request-arrival line,
which only matters for a request that never finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apt kept failing on a cached .deb with "Error reading from server. Remote
end closed connection", and no amount of retrying fixed it. Two bugs were
compounding.
The tee loop in serveAndStore broke on any read error without inspecting
it, then closed the cache pipe cleanly — so a connection that died
mid-body handed Store.Put a clean EOF and the truncated file was renamed
into place as if it were whole. And metaFromResponse recorded the length
the upstream had advertised rather than the bytes that arrived, so the
sidecar promised more than the file held. Every later hit then set that
Content-Length, streamed fewer bytes, and had its connection cut. Since
released artifacts never expire, the poisoned entry was served forever:
one dropped packet, one permanently broken package.
Now a read error or a short body aborts the pipe with an error, so the
store discards its temp file, and the store records what actually landed.
LocalFS refuses a Put whose body disagrees with its declared size, and
commits the payload before the sidecar so a crash between the two leaves
an invisible object rather than metadata pointing at a missing file. As a
backstop Stat cross-checks each entry against its file and discards any
that disagree — which heals the entries written before this existed,
without manual cleanup.
Retries now cover more than 429. A request that never produced a response
(refused, reset, TLS or DNS failure) is retried the same way, and a
transfer that dies after the headers — past the point any request-level
retry can help — is resumed: proxycache re-requests the remainder from
the upstream that answered, with Range and If-Range, and carries on
streaming where it left off. A resumed response is accepted only as a 206
whose Content-Range starts exactly where the transfer stopped; a 200
means the upstream is restarting from zero, and splicing that onto bytes
already sent would corrupt the object, so it is refused. Handlers build
the resume callback with upstream.Resumer, which binds the client and URL
that served the first byte — a maven or distro resume must never land on
a different mirror mid-file.
Also adds the observability this took to find. Every request gets a trace
id, adopted from the caller's X-Request-Id or traceparent when it sent
one, echoed back, attached to every log record the request produces via a
context-aware slog handler, and forwarded to the upstreams it calls, so
one grep follows a request end to end and across chained proxies. New
counters cover client and upstream requests, bytes in and out on both
sides, upstream errors, retries by reason, resumes by outcome, discarded
transfers and corrupt entries. A log config section tunes level, format
and the two access logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An upstream may now set `cache: false` to be proxied through without
being stored under the vrepo that fetched it. It is meant for chained
setups where the upstream is another vrepo of this proxy: without it
both layers keep their own copy of every artifact.
The decision has to travel with the response — a maven, debian or pacman
vrepo only learns which upstream answered once fallthrough resolves — so
proxycache fetch callbacks now return a Result{Resp, NoStore}. Reads are
unaffected: anything already cached under the vrepo is still served from
storage.
The UI's upstream column now shows each upstream's host and path, since
several vrepos routinely share a host (deb.debian.org/debian vs
/debian-security, chained vrepos of this proxy), and marks the ones that
are not cached.
Also includes an upstream retry transport for 429 responses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some registries (Gitea packages, private repos) answer 401/403 for every
anonymous read instead of 404. Returning that as-is broke aggregation for
any path not served by an earlier upstream. Mirror Reposilite's behavior:
treat 401/403 like 404 when more eligible upstreams remain; the last
upstream's status is still returned as-is.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Maven vrepos now accept an upstreams list tried in order: 404s and
connection errors fall through to the next upstream. Each upstream can
be restricted with wildcard group prefixes, artifact name globs, or
snapshots_only. Also includes github handler, upstream client and UI
updates.
Co-Authored-By: Claude Code <noreply@anthropic.com>
If Store.Put returns an error before draining the pipe (e.g. unwritable
cache dir), the tee writer blocked forever and the client hung with zero
bytes. Close the pipe reader on Put error, and replace io.TeeReader with
a hand-rolled loop: TeeReader consumes an upstream chunk before the pipe
write reports the failure, silently dropping it from the client stream.
Caching is now best-effort — the client always gets the full body.
Co-Authored-By: Claude Code <noreply@anthropic.com>
MRLokopandClaude Opus 5.5