# tiny-proxy

A single-binary Go service that proxies and caches OCI/Docker images, Maven artifacts, GitHub and Gitea release downloads, and Debian/APT and Pacman package mirrors through named **virtual repositories**, each with its own upstream, upstream credentials, egress proxy (HTTP/SOCKS5), client auth mode, and cache TTLs. Cached objects live on the local filesystem or in S3 (or any S3-compatible store, e.g. MinIO).

## Run

```sh
go build -o tiny-proxy .
cp config.example.yaml config.yaml   # edit to taste
./tiny-proxy -config config.yaml
```

## Usage

### Docker / OCI

The first path segment after `/v2/` names the virtual repo:

```sh
docker pull localhost:8080/dockerhub/library/nginx:1.27
```

Note: Docker treats plain HTTP registries as insecure — add the proxy host to `insecure-registries` in daemon.json, or put the proxy behind TLS.

### Maven

```
GET https://localhost:8080/maven/central/com/google/guava/guava/33.0.0-jre/guava-33.0.0-jre.jar
```

In `settings.xml`:

```xml
<mirror>
  <id>tiny-proxy</id>
  <mirrorOf>central</mirrorOf>
  <url>http://localhost:8080/maven/central</url>
</mirror>
```

### GitHub releases

Paths are `/github/<vrepo>/<owner>/<repo>/<selector>` where selector is a tag
or an alias — `latest` (latest stable release) or `snapshot` (most recent
prerelease):

| request | response |
|---|---|
| `/github/gh/golang/go/latest` | 302 → the resolved tag (or content in place when `redirect: false`) |
| `/github/gh/golang/go/v1.2.3.json` | release metadata: the GitHub API release object with each asset's `browser_download_url` rewritten to this proxy |
| `/github/gh/golang/go/latest.json` | same, resolved via the alias |
| `/github/gh/golang/go/v1.2.3/app.zip` | asset download, cached forever |
| `/github/gh/golang/go/latest/app.zip` | 302 → the asset under the resolved tag |

```sh
curl -s http://localhost:8080/github/gh/golang/go/latest.json | jq -r '.tag_name'
curl -LO http://localhost:8080/github/gh/golang/go/latest/go1.23.4.linux-amd64.tar.gz
```

- Metadata and alias resolutions are cached for `cache.metadata_ttl`; assets
  are immutable and cached forever. Aliases share the tag's cache entries.
- `allow: ["myorg/*", "foo/bar"]` restricts which `owner/repo` pairs may be
  proxied (`path.Match` globs). Omit it to allow everything.
- `redirect: false` serves alias URLs in place instead of 302-redirecting to
  the resolved tag (cache keys are shared either way).
- Set `upstream.username`/`password` to a GitHub user + PAT for private repos
  and to avoid the unauthenticated API rate limit (60 req/h/IP). For GitHub
  Enterprise point `upstream.url` at your instance (API `<url>/api/v3`).
- Tags containing `/` work in the `.json` and asset forms; `latest` and
  `snapshot` are reserved names.
- Assets whose names end in `.json` work too: the proxy tries the tag
  metadata first and serves the asset form when no such tag exists.
- `browser_download_url` values in `.json` metadata are root-relative paths
  on this proxy (e.g. `/github/gh/o/r/v1.2.3/app.zip`), not absolute URLs.
- Assets are cached forever, matching the immutable-artifact assumption of
  the other modules; if an upstream asset is re-uploaded under the same
  name, delete the key under `data/github/…` to refresh.

### Gitea releases

Identical to the GitHub type — same URL grammar under `/gitea/<vrepo>/...`
(aliases, `.json` metadata with rewritten asset URLs, immutable asset
cache). Point `upstream.url` at your Gitea instance; its API is served at
`<url>/api/v1`. There is no default upstream. Set
`upstream.username`/`password` to a user + access token for private repos.

```sh
curl -s http://localhost:8080/gitea/gt/myorg/mytool/latest.json | jq -r '.tag_name'
curl -LO http://localhost:8080/gitea/gt/myorg/mytool/latest/mytool_linux-amd64
```

### Debian / APT

A pass-through cache for APT repositories (Debian, Ubuntu, or any mirror):

```
GET http://localhost:8080/debian/deb/dists/bookworm/Release
GET http://localhost:8080/debian/deb/pool/main/h/hello/hello_1.0-1_amd64.deb
```

In `sources.list`:

```
deb http://localhost:8080/debian/deb bookworm main contrib non-free-firmware
```

The UI page for a debian vrepo (`/ui/debian/<vrepo>/`) generates these lines
for you: it probes the upstream mirror's `dists/` index and each suite's
`Release` file, then lists one ready-to-paste `deb` line per distro version
with that suite's real component list, plus the architectures each version
carries. Works for any APT archive — Debian's `bookworm`/`trixie`/`sid`,
Ubuntu's `noble`/`jammy`, `*-updates`, `*-backports` and friends. The probe
result is memoized per vrepo for `cache.metadata_ttl` (failures for a
minute), so browsing the page doesn't re-hit the mirror; `Release` files are
fetched in a bounded worker pool. When the mirror is unreachable the page
falls back to listing the dists already in the cache.

- Metadata under `dists/**` (Release, InRelease, Packages.*, Sources.*,
  Contents-*, Translation-*) is cached for `cache.metadata_ttl`; anything
  under `pool/` or `by-hash/` is content-addressed and cached forever. This
  keeps `apt update` correct for by-hash-capable clients (apt 1.2+).
- Signatures need no special handling: `InRelease` and `Release.gpg` are
  passed through byte-identical, so apt verifies them against the distro's
  regular keyring. The proxy neither strips nor re-signs anything, and no
  key needs to be installed on clients.
- All architectures a suite publishes are served (`binary-amd64`,
  `binary-arm64`, …); pin one with `deb [arch=amd64] <uri> <suite> …`.
  `deb-src` lines work the same way.
- Multiple `upstreams` may be configured as mirrors: a 404, 401, 403, 429 or
  connection error falls through to the next one.
- `If-Modified-Since` revalidation is not forwarded; `metadata_ttl` is the
  freshness knob. Keep it short (minutes) so `InRelease` and `Packages.*`
  don't drift stale-inconsistent.

### Pacman

A pass-through cache for Arch Linux package mirrors:

```
GET http://localhost:8080/pacman/arch/core/os/x86_64/core.db
GET http://localhost:8080/pacman/arch/core/os/x86_64/linux-6.10.2-1-x86_64.pkg.tar.zst
```

In `pacman.conf`:

```
[core]
Server = http://localhost:8080/pacman/arch/$repo/os/$arch
```

- Repo databases (`<repo>.db`, `<repo>.files`, in any compression form, plus
  their `.sig` and the `lastupdate` stamp) are cached for
  `cache.metadata_ttl`; packages (`*.pkg.tar.*` and signatures) are
  immutable and cached forever.
- Multiple `upstreams` may be configured as mirrors with
  404/401/403/429/error fall-through.
- pacman has no native HTTP-auth support; run pacman vrepos in `public`
  mode on a trusted network, or behind an auth-injecting local mirror.

## Configuration

See `config.example.yaml`. Per virtual repo:

| field | meaning |
|---|---|
| `type` | `docker`, `maven`, `github`, `gitea`, `debian` or `pacman` |
| `upstream.url` | registry / repository base URL (github: defaults to `https://github.com`; gitea: required, API at `<url>/api/v1`) |
| `upstream.username` / `password` | optional upstream credentials. For Bearer-challenge registries (Docker Hub, GHCR) the proxy performs the token exchange automatically. |
| `upstream.proxy` | optional egress proxy: `http://`, `https://` or `socks5://[user:pass@]host:port` |
| `upstreams` | maven, debian, pacman only: ordered fallback list, replacing `upstream`. Each entry takes the same `url`/`username`/`password`/`proxy` fields plus (maven only) filters below. A 404, 401, 403, 429 or connection error falls through to the next upstream; any other status is returned as-is. |
| `upstreams[].groups` | maven only: wildcard group allowlist, e.g. `com.google.*` (`*` crosses dots; bare `com.google` also covers the subtree). Empty = all groups. |
| `upstreams[].artifacts` | maven only: wildcard artifact-name allowlist, e.g. `my-lib-*`. Empty = all artifacts. |
| `upstreams[].snapshots_only` | maven only: only `-SNAPSHOT` paths are requested from this upstream. |
| `upstream.cache` / `upstreams[].cache` | `true` (default) stores what this upstream serves; `false` proxies it through without writing to storage. Use it when the upstream already caches for you — typically another vrepo of this proxy. Reads are unaffected: anything already cached under the vrepo is still served from storage. |
| `upstream.retry.attempts` / `upstreams[].retry.attempts` | retries for this upstream, before the request falls through to the next one. Covers a `429 Too Many Requests`, a connection that fails before any response arrives, and a transfer that dies mid-body (resumed with a byte range). Default `2`; `0` disables retrying. |
| `upstream.retry.max_wait` | cap on a single wait between those retries, a `Retry-After` sent by the upstream included. Default `5s`. |
| `auth.mode` | `public` (no auth), `password` (any login + this password), `users` (map of login: password) |
| `cache.manifest_ttl` / `metadata_ttl` | how long tags / `maven-metadata.xml` / SNAPSHOTs / release metadata / APT and pacman indexes count as fresh. Blobs and released artifacts are immutable and cached forever. `0` = always revalidate. |
| `allow` | github, gitea only: glob allowlist of `owner/repo` pairs; empty = allow all |
| `redirect` | github, gitea only: `true` (default) 302s aliases to the resolved tag; `false` serves content under the alias URL |

### Retries, rate limits and interrupted transfers

Upstreams fail in three ways worth retrying, and `retry.attempts` /
`retry.max_wait` govern all three.

**Rate limits (429).** Upstreams throttle busy clients — `Received status code
429 from server: Too Many Requests` in a Maven or Gradle build. The proxy
absorbs it in two steps:

1. The upstream client retries the request `retry.attempts` times, waiting
   the upstream's `Retry-After` (or an exponential backoff with jitter when
   it sends none), capped at `retry.max_wait` per wait.
2. If it is still throttled, the request falls through to the next
   `upstreams` entry, exactly like a 404.

A 429 is only returned to the client when no upstream could serve the path.
That answer wins over a 404 from another mirror — a throttled upstream may
well hold the artifact, so reporting it missing would make the build fail
for the wrong reason — and carries the upstream's `Retry-After` when it sent
one. Nothing about a 429 is cached.

Long throttling windows are not waited out: an upstream asking for minutes
is capped to `max_wait`, the attempts run out, and the next mirror is tried
instead of pinning the build's connection open. Retries are logged at warn
level (`upstream request failed, retrying`, with `reason=rate_limited`).

**Connection failures.** A request that never produces a response — connection
refused or reset, a TLS or DNS failure — is retried the same way
(`reason=transport_error`), then falls through to the next upstream. Only
bodyless `GET`/`HEAD` requests are retried, so a retry never has to replay a
request body. A cancelled request is not retried: the client has already gone.

**Interrupted transfers.** A connection that dies *after* the response headers
— `Error reading from server. Remote end closed connection` in apt — is past
the point any request-level retry can help, so the proxy resumes it instead:
it re-requests the remainder from the same upstream with
`Range: bytes=<n>-` and `If-Range`, and carries on streaming to the client
where it left off. The resumed response is accepted only if it is a `206`
whose `Content-Range` starts exactly where the transfer stopped; a `200`
(the upstream ignoring the range, or the file having changed under us) is
refused rather than spliced into the middle of an object.

**Nothing broken is ever cached.** A transfer that ends short of the full
object — a read error, or a clean close after fewer bytes than the
`Content-Length` promised — is discarded rather than committed, so a
transient network hiccup cannot leave a truncated artifact in the cache.
Because released artifacts never expire, a truncated one used to stay
truncated forever. As a backstop, any stored object whose size disagrees with
its recorded metadata is discarded and refetched
(`corrupt cache entry, discarding`), which heals entries written before this
check existed — no manual cleanup needed.

### Chaining vrepos

A vrepo may list another vrepo of the same proxy as an upstream — an
aggregator in front of a shared cache. Without `cache: false` both layers
store the same bytes; with it the aggregator only routes:

```yaml
- name: central                    # caches Maven Central
  type: maven
  upstream: { url: https://repo1.maven.org/maven2 }

- name: deps                       # aggregator: routes, does not store
  type: maven
  upstreams:
    - url: https://mvn.example.com
    - url: http://localhost:8080/maven/proxy
      cache: false
    - url: http://localhost:8080/maven/central
      cache: false
```

Point the chained upstream at this proxy's own base URL; it is an ordinary
HTTP request, so a vrepo with `auth.mode` other than `public` needs
`username`/`password` on the upstream entry. Keep the chain shallow — each
hop is a real request — and acyclic: a vrepo that reaches itself will
recurse until the client times out.

The UI's Upstream column shows each upstream's host and path, marking the
ones this vrepo does not cache with `(uncached)`.

### Metrics

`GET /metrics` serves Prometheus text exposition. Everything a request can be
blamed on carries a label: `vrepo` for what the client asked for, `vrepo` plus
`upstream` (the URL as configured) for what the proxy had to fetch.

| series | meaning |
|---|---|
| `tinyproxy_requests_total{type,vrepo}` | client requests per proxy type and repo. An empty `vrepo` is a request that reached no repo: unknown name, bad path, or auth refused |
| `tinyproxy_request_duration_seconds{type,vrepo}` | histogram of time spent serving a request in a vrepo — a cache hit and an upstream fetch both land here |
| `tinyproxy_cache_hits_total{type,vrepo}` / `_misses_total` / `_errors_total` | cache lookups served from storage, fetched from upstream, and failed |
| `tinyproxy_transfer_failures_total{type,vrepo}` | transfers that ended short of the full object and were discarded instead of cached |
| `tinyproxy_upstream_requests_total{vrepo,upstream}` | upstream attempts, counted per wire attempt — a retried 429, a token exchange and a resumed range request each count |
| `tinyproxy_upstream_responses_total{vrepo,upstream,status}` | upstream responses by status code |
| `tinyproxy_upstream_errors_total{vrepo,upstream}` | attempts that never produced a response (refused, reset, TLS or DNS failure) |
| `tinyproxy_upstream_received_bytes_total{vrepo,upstream}` / `_sent_bytes_total` | **body** bytes read from and written to upstreams |
| `tinyproxy_upstream_wire_received_bytes_total{vrepo,upstream}` / `_wire_sent_bytes_total` | **socket** bytes: headers, chunked framing, TLS records and handshakes included |
| `tinyproxy_upstream_connect_seconds{vrepo,upstream}` | histogram of dial time, DNS included |
| `tinyproxy_upstream_tls_handshake_seconds{vrepo,upstream}` | histogram of TLS handshake time |
| `tinyproxy_upstream_ttfb_seconds{vrepo,upstream}` | histogram of the wait between sending a request and the first response byte — the upstream's own think time |
| `tinyproxy_upstream_body_seconds{vrepo,upstream}` | histogram of body streaming time, from first byte to EOF or close |
| `tinyproxy_upstream_retries_total{reason}` | retried attempts, by `rate_limited` or `transport_error` |
| `tinyproxy_upstream_resumes_total{result}` | transfers that died mid-body and were re-requested with a byte range, by `ok` or `failed` |
| `tinyproxy_client_requests_total` | client requests across all endpoints, `/ui/` and `/metrics` included |
| `tinyproxy_client_received_bytes_total` / `_sent_bytes_total` | **body** bytes read from and written to clients |
| `tinyproxy_client_wire_received_bytes_total` / `_wire_sent_bytes_total` | **socket** bytes exchanged with clients, headers and framing included |
| `tinyproxy_cache_corrupt_entries_total` | stored objects found to disagree with their metadata; discarded and refetched |
| `tinyproxy_storage_items` / `tinyproxy_storage_bytes` | per-vrepo gauges, collected on scrape via one storage `List` per vrepo |

Plus Go runtime stats (`tinyproxy_uptime_seconds`, goroutines, memstats).

**Body vs wire bytes.** The body counters are payload — what the artifact
weighs. The wire counters are taken at the socket and are the real traffic:
request lines, headers, chunked framing and, upstream, the TLS records and
handshake. Upstream sockets belong to exactly one vrepo/upstream pair, so
those totals are labelled; client sockets are not, because a keep-alive
connection carries requests for several vrepos — so the client wire counters
are process-wide rather than split on a guess.

**Reading the timings.** The connection histograms only record when a
connection is actually opened, so a quiet `connect_seconds` beside a busy
`upstream_requests_total` means the pool is doing its job. `ttfb_seconds` is
the upstream thinking; `body_seconds` is the link. A rise in the first is the
upstream's problem, a rise in the second is usually yours.

Two ratios earn their keep. `tinyproxy_upstream_requests_total` over
`tinyproxy_client_requests_total` rises when an upstream starts throttling,
flapping or truncating transfers. And in a fallback list (maven, debian,
pacman) the `status="404"` share of `tinyproxy_upstream_responses_total` shows
how much of the walk each mirror is absorbing — a mirror that answers nothing
but 404 is costing a round trip on every miss and belongs later in the list.

`metrics.token` protects the endpoint — empty (the default) means open; when
set, clients must send `Authorization: Bearer <token>` or a
`?token=<token>` query param. Use a 15-30s+ scrape interval: each scrape
lists every vrepo's storage prefix, and S3 `List` paginates the whole
prefix.

## How it works

- Read-only pull-through: `GET`/`HEAD` only; pushes get `405`.
- Cache miss → stream from upstream while tee-ing into storage (write-through, single-flight per key). Hit → served from storage.
- `storage.type: none` runs the proxy without any cache: every request goes upstream, nothing is stored, and the UI and storage gauges report empty. `upstream.cache: false` is the same idea scoped to one upstream.
- Docker image names are normalized for Docker Hub (`nginx` → `library/nginx`).
- Docker clients authenticate with plain Basic auth (supported natively by `docker login`); the Bearer token dance happens only between the proxy and the upstream.
- `/healthz` answers liveness probes; `/metrics` exposes Prometheus metrics.
- Every request gets a trace id, carried on every log line it causes and passed to the upstreams it fans out to. See [Logging](#logging).

### Logging

Each inbound request is given a trace id — adopted from the caller's
`X-Request-Id` or W3C `traceparent` when it sent one, minted otherwise — that
is echoed back in `X-Request-Id`, attached to every log record the request
produces, and forwarded to each upstream it calls. Grepping one id shows the
whole life of a request:

```
INFO upstream request trace=9f2a1c8e40b7d5e3 vrepo=debian-security url=https://deb.debian.org/debian-security/... status=200 dur_ms=41
WARN upstream transfer interrupted, resuming trace=9f2a1c8e40b7d5e3 key=debian/debian-security/pool/... offset=131072 attempts_left=1
INFO upstream request trace=9f2a1c8e40b7d5e3 vrepo=debian-security url=https://deb.debian.org/debian-security/... status=206 dur_ms=38
INFO request done trace=9f2a1c8e40b7d5e3 method=GET path=/debian/debian-security/pool/.../libblkid-dev_2.41.5-0+deb13u1_arm64.deb status=200 bytes=289456 dur_ms=402
```

Because chained tiny-proxy hops adopt each other's `X-Request-Id`, one id
follows a request across all of them.

| field | meaning |
|---|---|
| `log.level` | `debug`, `info`, `warn` or `error`. Default `info`, which already gives one line per finished client request and one per upstream attempt. `debug` adds a line when each request *arrives*, which only matters for requests that never finish. |
| `log.format` | `text` (default) or `json` |
| `log.requests` | inbound access log. Default `true`; `false` keeps the trace ids and metrics but drops the lines. |
| `log.upstream` | per-upstream request log. Default `true`. |

## Out of scope

Pushes/deploys, HTTPS termination (use a reverse proxy), cache eviction/GC, admin API.