llmshit

A proxy for the Anthropic Messages API that applies find/replace rules — literal or regex — to prompts as they pass through. Point a client at it instead of api.anthropic.com and everything keeps working: streaming, tool use, extended thinking.

client ──▶ llmshit ──▶ api.anthropic.com
             │
             └── rules from llmshit.yaml

Useful for redacting internal hostnames or secrets before they leave the network, patching a system prompt you do not control, swapping terminology, or A/B-testing prompt edits without touching the client.

Quick start

bun install
cp llmshit.example.yaml llmshit.yaml   # edit the rules
bun run start

Then point any Anthropic client at it:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8787
claude                    # Claude Code

const client = new Anthropic({ baseURL: "http://127.0.0.1:8787" });

Then open http://127.0.0.1:8787/__llmshit/ui to watch requests arrive and see exactly what your rules did to each one.

Docker

docker run -d --name llmshit \
  -p 127.0.0.1:8787:8787 \
  -v "$PWD/llmshit.yaml:/etc/llmshit/llmshit.yaml:ro" \
  git.mrlokop.dev/mrlokop/llmshit:latest

Or with the bundled docker-compose.yml: docker compose up -d.

The image looks for a config at ./llmshit.yaml then /etc/llmshit/llmshit.yaml; mount one at the latter, or run without it for a pass-through proxy. It binds 0.0.0.0 inside the container — 127.0.0.1 would be unreachable — so publishing to loopback is what keeps the viewer private. Publish it more widely only if you mean to, and set ui.allow_remote: true (or ui.enabled: false) to acknowledge it.

Runs as the non-root bun user, with a healthcheck on /__llmshit/health. Build it yourself with docker build -t llmshit ..

CLI

bun run index.ts [--config <path>] [--port <n>] [--host <addr>] [--watch]

  -c, --config <path>  YAML config (default: $LLMSHIT_CONFIG, then ./llmshit.yaml)
  -p, --port <n>       Override server.port
  -H, --host <addr>    Override server.host
  -w, --watch          Reload the config when the file changes

Variable	Effect
LLMSHIT_CONFIG	Config path. Without it, ./llmshit.yaml then /etc/llmshit/llmshit.yaml are tried.
LLMSHIT_LISTEN_HOST	Bind address. A --host flag still wins.
LLMSHIT_LISTEN_PORT	Port. A --port flag still wins. An unparseable value is warned about and ignored — Kubernetes injects <SERVICE_NAME>_PORT=tcp://… for every Service in a namespace, and no proxy should fail to boot over that.

With no config file it runs as a plain pass-through proxy. GET /__llmshit/health reports the loaded config, rule count, and upstream.

What gets proxied

Path	Behaviour
POST /v1/messages	Request rules applied. Response rewritten only if a rule targets response_text (SSE-aware when streaming).
POST /v1/messages/count_tokens	Same request rules, so counts reflect what will actually be sent.
everything else	Byte-for-byte pass-through — models, files, batches, and anything the API adds later.
GET /api/hello	Answered locally with {"message": "hello"}, exactly as api.anthropic.com does, so health probes see the proxy as the real API. Not forwarded, and not recorded — a probe every few seconds would otherwise evict real traffic from the viewer's bounded log.
/__llmshit/*	Served locally, never forwarded: the viewer, its data endpoints, and /__llmshit/health.

The proxy parses JSON only for the two paths above, and only rewrites an explicit allowlist of fields. Unknown request parameters, unknown content-block types, and unknown SSE events are forwarded untouched, so new API features keep working without a proxy update.

Configuration

server:
  host: 127.0.0.1
  port: 8787

upstream:
  base_url: https://api.anthropic.com
  api_key: ${ANTHROPIC_API_KEY}   # optional
  timeout_ms: 600000

logging:
  level: info            # silent | error | info | debug
  log_replacements: true # logs rule name + hit count, never the matched text

ui:
  enabled: true
  max_entries: 200
  max_retained_chars: 8000000
  capture_responses: true
  allow_remote: false

streaming:
  hold_bytes: 256

rules:
  - name: redact-internal-hosts
    enabled: true        # default true
    kind: regex          # regex (default) | literal
    find: 'https?://[a-z0-9.-]+\.corp\.internal\b'
    flags: i             # any of i m s u; g is always applied
    replace: 'https://example.com'
    in: [system, user]   # default [system, user]
    when:                # optional
      model: 'claude-opus-*'
      path: '/v1/messages'
      only_last_user_message: false

${VAR} and ${VAR:-default} are expanded from the environment when the file is read. Unknown keys are rejected rather than ignored, so typos surface at startup instead of silently disabling a rule.

Authentication

By default the caller's x-api-key / Authorization is forwarded unchanged. Set upstream.api_key and the proxy substitutes it instead, so clients can be configured with a dummy key and the real one lives only in the proxy's environment.

kind: regex vs kind: literal

• regex — find is a JavaScript regular expression and replace supports $1, $&, etc.
• literal — both find and replace are taken verbatim. Regex metacharacters and $1 have no special meaning.

The g flag is always applied, so every occurrence is replaced. The sticky flag y is rejected: it would behave differently on a streamed fragment than on a whole string.

Rules run in file order, so a later rule sees the output of an earlier one.

Targets (in)

Target	Reaches
system	the top-level system field (string or text blocks), and mid-conversation {role: "system"} messages
user	text blocks in user messages
assistant	text blocks in assistant messages
tool_result	string and text-block content inside tool_result blocks
tool_definitions	tool description, plus description strings nested in input_schema
tool_use_input	string leaves of tool_use.input — opt-in, see the warning below
response_text	assistant text output, streaming and non-streaming

Conditions (when)

• model — glob against the request's model (claude-opus-*). A rule with a model glob never fires on a request with no model.
• path — glob against the request path, to separate /v1/messages from /v1/messages/count_tokens.
• only_last_user_message — restrict the rule to the final user turn. Applies to the message-content targets only; system, tool_definitions, and response_text are unaffected.

Live request viewer

http://127.0.0.1:8787/__llmshit/ui is a single self-contained page — no CDN, no fonts, no external requests of any kind, because the page displays your prompts and must not hand them to a third party. It streams updates over SSE, so requests appear as they arrive.

The sidebar lists recent requests with their status, model, duration, and how many substitutions fired. Selecting one opens five tabs:

• Overview — status, duration, stop reason, and every rule that fired with its replacement count.
• System — the system prompt, toggling between what the client sent, what went upstream, and a split view of both. This is usually the tab you want.
• Messages — each message's blocks, showing before/after side by side for any block a rule changed.
• Tools — declared tool names and descriptions, as sent upstream.
• Response — the assistant text the client received.

Everything lives in memory and is gone on restart. Nothing is written to disk.

When the event stream cannot get through

Live updates arrive over SSE, which has to survive every proxy between the browser and the pod — and some buffer or drop long-lived responses. If the stream fails repeatedly the viewer stops retrying and polls requests every 2 seconds instead; the header shows polling rather than live and everything else keeps working, since the rest of the page is plain GETs. The SSE response carries no hop-by-hop headers (illegal over HTTP/2) and sets X-Accel-Buffering: no to ask intermediaries not to buffer it.

Limits

Prompts are captured whole — nothing is truncated. Two caps bound the memory instead, and both evict oldest-first:

• ui.max_entries (default 200) caps how many requests are retained.
• ui.max_retained_chars (default 8,000,000) caps the total text retained across all of them, measured after deduplication.

The newest request is never evicted, so one larger than the whole budget is still visible rather than vanishing on arrival. Set ui.capture_responses: false to keep request bodies but not response text, or ui.enabled: false to record nothing at all and 404 the page.

Deduplication

Clients replay the entire conversation on every turn, so the same system prompt and the same earlier messages arrive again with each request — and parsing produces a fresh copy each time. The recorder interns captured strings in a refcounted pool: identical text is stored once, and released the moment the last request holding it is evicted. A 20-turn conversation holds one copy of its system prompt, not 40. The header shows what is retained and what dedup saved.

Interning also removes a class of false signal: an unrewritten system prompt yields the same string for both the before and after side, so it cannot read as changed.

Two properties are worth knowing:

• Thinking content is never captured. Thinking and redacted_thinking blocks are recorded as present and nothing more — no text, no signature. The viewer shows [thinking block — never captured, never rewritten].
• The sidebar payload carries no prompt text. Bodies are fetched only when you select a request, so an open viewer does not stream your prompts continuously.

Exposure

The viewer serves prompt contents to anyone who can reach the port. The default bind is 127.0.0.1, which keeps it local; in Docker the bind is necessarily 0.0.0.0, so the port publish is what limits reach — -p 127.0.0.1:8787:8787.

Binding to a wildcard address with the UI enabled logs a warning at startup. Set ui.enabled: false if the proxy must listen on a public interface, or ui.allow_remote: true to accept the exposure and silence the warning — it changes nothing else.

Extended thinking

Two properties of this proxy exist specifically so extended thinking keeps working.

Thinking blocks are never modified. thinking.thinking, thinking.signature, and redacted_thinking.data carry a signature the API verifies whenever the block is replayed in a later turn; changing one byte invalidates it. No target can reach them — the traversal is an allowlist of the fields in the table above, so thinking blocks (and any block type added to the API later) are unreachable by construction rather than by convention. The same holds mid-stream: thinking_delta and signature_delta frames are forwarded verbatim.

Rewriting is deterministic, so replayed history stays byte-stable. Clients resend the whole conversation on every turn. If turn 1's user text is rewritten foo → bar, turn 2 resends the original foo and must produce bar again, so the API sees an unchanged history. Rules are pure string substitutions with no clock, randomness, or request counter, and key order is preserved, so this holds.

Two things break that guarantee, both deliberate and both opt-in:

• only_last_user_message moves with the conversation by definition — the turn it rewrites changes every request, which edits replayed history.
• Editing the config mid-conversation (including --watch reloads) changes how earlier turns are rewritten.

Either one busts the prompt cache, and on models with preserved thinking it can invalidate thinking blocks in the replayed history. Change rules between conversations, not during one.

Rewriting responses

Response rewriting is off unless a rule lists response_text. When no rule does, the response body is piped straight through without being parsed.

For streaming, the SSE stream is parsed into frames and only text_delta deltas on a block that announced type: "text" are eligible. Because a pattern can straddle two frames ("Project " then "Nimbus ships"), the proxy holds back the trailing streaming.hold_bytes characters of each text block and flushes them when the block closes. A rule that can match a string longer than hold_bytes may miss at a frame boundary — raise the setting if your patterns are long.

One round-trip caveat: a client stores the rewritten assistant text and replays it on the next turn, where the rule runs again. Make sure replace cannot re-match find, or the substitution compounds.

Things the proxy will not touch

Regardless of configuration:

• thinking and redacted_thinking blocks, and their signatures
• tool_use.id and tool_result.tool_use_id — they pair a call with its result
• tool name — renaming it breaks that same pairing
• base64 source.data on image and document blocks
• model, citations, and every field of a content-block type it does not recognise

tool_use_input is the one deliberately sharp edge. It rewrites the arguments the model chose for a tool call, which can easily break the tool being called. It is off unless you list it.

Operational notes

• Requests are sent upstream with accept-encoding: identity, since a rewritten body cannot be re-compressed. content-encoding and content-length are dropped from the response.
• Hop-by-hop headers are stripped in both directions; everything else, including anthropic-version and anthropic-beta, is forwarded.
• Upstream errors reach the client verbatim with their status. Failures inside the proxy return Anthropic's error envelope (502, or 504 on timeout) so SDK clients raise a typed error rather than a parse failure.
• log_replacements logs rule names and hit counts only — never the text that matched.
• Rules are operator-authored and run against every request, so avoid catastrophically backtracking regexes.

Development

bun test           # 145 tests
bun run typecheck  # tsc --noEmit
bun run dev        # reload on source change

The suite covers the rule engine, the rewrite allowlist (a fixture carrying the same sentinel in every block type, asserting only the targeted fields change), SSE frame handling including patterns split across frames and arriving one character at a time, history stability across turns, config validation, an end-to-end pass driving the proxy with the real @anthropic-ai/sdk against a mock upstream, and the recorder's eviction, capture budget, and the guarantee that no thinking content reaches it.

Layout

index.ts        CLI entry
src/config.ts   YAML load, validation, env expansion
src/rules.ts    rule compilation and string substitution
src/rewrite.ts  request/response traversal — the allowlist lives here
src/sse.ts      SSE parsing and the streaming transform
src/proxy.ts    routing, headers, upstream fetch, control endpoints
src/recorder.ts bounded in-memory request log and its subscription feed
src/ui.ts       the viewer, as one self-contained HTML document
Dockerfile      two-stage build on oven/bun:1.3-alpine, runs as non-root