TransitTruth
A local CLI that checks whether an OpenAI-compatible endpoint actually serves the model it claims to.
MIT license
v0.5.9
374 passed / 7 skipped
Python CLI + local web UI
You pay for gpt-4o-mini at a relay. Are you actually getting it? TransitTruth probes
tokenizer counts and behavioral baselines against a bundled reference fingerprint and returns one of
pass / suspicious / insufficient_evidence as a single JSON verdict.
It is probabilistic consistency evidence, not a cryptographic proof.
Problem
Third-party LLM relays and gateways commonly forward traffic under a claimed model name, and existing
gateways (one-api, new-api, LiteLLM) trust that field. A cheaper model can be substituted while billing
as the advertised one, and the buyer has no independent check from their own machine.
What it does
- Stateless fingerprint battery: tokenizer-count consistency, chi-square / KS distribution distance,
Bayesian log-odds self-consistency, and an Okapi-style resubstitution check.
- Three-state verdict with confidence; fail-closed to
insufficient_evidence when no
reference matches, rather than guessing.
- The API key stays in session memory: it never touches disk, logs, or the report; the tool probes
only and does not forward traffic.
- A guided offline web demo walks through the check with bundled seed data; the real audit runs locally
with
transit-truth serve.
Honest limits
- Probabilistic, not cryptographic. The verdict is auxiliary consistency evidence;
it cannot establish official-provider identity, and a relayed upstream can still look consistent.
- Local fingerprint coverage is limited. The bundle ships one open Qwen2.5 tokenizer
fingerprint; the gpt-4o / gpt-4o-mini baselines were collected through a single third-party relay,
not official endpoints.
- Temperature=0 replay is point-mass. A deterministic upstream can match on
repeated prompts; behavioral probes need more samples to separate a real match from a replay.
- Not published to PyPI yet. Install from source; the legacy billing/monitoring
web platform is a separate self-hosted app and is not part of this bundle.
Measured evidence (with qualifications)
- 91.3% in-distribution resubstitution posterior (training-set fit, not held-out generalization).
- 0.90 total-variation distance between claimed and observed response distributions (a distribution
gap, not an accuracy score).
- 88% Okapi resubstitution rate, Wilson interval [0.80, 0.93], same in-distribution caveat.
All numbers are reproduced in the committed reference-data transparency section and the
extended technical report; see the links below.