apt kept failing on a cached .deb with "Error reading from server. Remote
end closed connection", and no amount of retrying fixed it. Two bugs were
compounding.
The tee loop in serveAndStore broke on any read error without inspecting
it, then closed the cache pipe cleanly — so a connection that died
mid-body handed Store.Put a clean EOF and the truncated file was renamed
into place as if it were whole. And metaFromResponse recorded the length
the upstream had advertised rather than the bytes that arrived, so the
sidecar promised more than the file held. Every later hit then set that
Content-Length, streamed fewer bytes, and had its connection cut. Since
released artifacts never expire, the poisoned entry was served forever:
one dropped packet, one permanently broken package.
Now a read error or a short body aborts the pipe with an error, so the
store discards its temp file, and the store records what actually landed.
LocalFS refuses a Put whose body disagrees with its declared size, and
commits the payload before the sidecar so a crash between the two leaves
an invisible object rather than metadata pointing at a missing file. As a
backstop Stat cross-checks each entry against its file and discards any
that disagree — which heals the entries written before this existed,
without manual cleanup.
Retries now cover more than 429. A request that never produced a response
(refused, reset, TLS or DNS failure) is retried the same way, and a
transfer that dies after the headers — past the point any request-level
retry can help — is resumed: proxycache re-requests the remainder from
the upstream that answered, with Range and If-Range, and carries on
streaming where it left off. A resumed response is accepted only as a 206
whose Content-Range starts exactly where the transfer stopped; a 200
means the upstream is restarting from zero, and splicing that onto bytes
already sent would corrupt the object, so it is refused. Handlers build
the resume callback with upstream.Resumer, which binds the client and URL
that served the first byte — a maven or distro resume must never land on
a different mirror mid-file.
Also adds the observability this took to find. Every request gets a trace
id, adopted from the caller's X-Request-Id or traceparent when it sent
one, echoed back, attached to every log record the request produces via a
context-aware slog handler, and forwarded to the upstreams it calls, so
one grep follows a request end to end and across chained proxies. New
counters cover client and upstream requests, bytes in and out on both
sides, upstream errors, retries by reason, resumes by outcome, discarded
transfers and corrupt entries. A log config section tunes level, format
and the two access logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MRLokopandClaude Opus 5
0df385fa57