Token Lens

Token consumption analytics and reduction suggestions, built as a drop-in dashboard plugin for Hermes Agent.

Hermes' core Analytics page tells you how much you spend. Token Lens tells you where the tokens go and what to change — every category bucket calibrated so it sums exactly to your billed totals, and every suggestion shipped with evidence, expected savings, a capability-risk note, and a copy-paste plan you can run in a Hermes chat.

What it does

• Where tokens go. A hook recorder captures exact billed usage per API call and decomposes it into stable categories: system prompt, tool schemas (built-in and per-MCP-server), skill loading, memory, history, tool results, output, reasoning.
• Suggestions. Deterministic detectors (unused MCP servers, low cache hit rate, oversized system prompt, runaway turns, heavy tool results) fire after 3 recorded sessions. AI-generated suggestions unlock at 10, gated by a rubric-scored evaluator.
• Acted-on tracking. Mark a suggestion done and Token Lens shows predicted vs. observed change, normalized per session so a quiet week doesn't masquerade as savings.
• Honest data. Exact vs. estimated precision is badged everywhere, backfilled history renders hatched, and calibration drift is monitored in /health.

The entry point is an insight strip on the Sessions page that leads with the finding ("Token Lens found 4% avoidable weekly token waste"), linking into a dedicated tab whose work surface puts the top suggestion and a waste map side by side.

Architecture

Two processes share one SQLite database (~/.hermes/token_lens.db, WAL mode). The recorder runs agent-side via plugin hooks (pre/post_api_request, on_session_finalize), buffers writes, and never blocks the conversation loop. The dashboard API is a FastAPI router running in the dashboard process. Shared logic lives in a single token_lens_core.py. The UI is a no-build IIFE bundle on the Hermes plugin SDK — no bundled React, hand-rolled SVG charts, themed entirely through host CSS tokens.

Implementation Process

Fable High via Claude Code was used along with gstack. For those not familiar with gstack read What is Gstack?

The feature took two main sessions, back to back.

Session one: planning. It started as a /office-hours conversation in gstack — product framing before any architecture. The output was a design plan that then ran the review gauntlet: /plan-eng-review twice, /plan-design-review once, and /codex review twice as a cross-model outside voice (GPT-5 Codex reading the Hermes source cold, no shared context with the plan's author).

• The design review scored the first UI proposal 6/10, and Codex hard-rejected the card-stack layout outright — which forced the composed work surface and the insight-strip entry card that actually shipped.
• The eng reviews caught that token estimators drift 10–15% from billed numbers (→ per-call calibration) and that session-finalize hooks can't be trusted to fire (→ idempotent rollups plus a catch-up sweep).
• For the visual direction, /design-shotgun generated three mockup variants; the dense, terminal-inspired, mono-forward variant C won.
  The plan ended with ~25 tagged decisions and 22 ordered tasks, all reviews CLEAR.

Session two: implementation. Fable 5 built it from the approved plan — recorder, calibration, detectors, suggestion engine, evaluator, dashboard UI — through three milestones in one long session, writing 95 tests, two e2e flows, and LLM evals along the way. QA ran live: gstack's headless /browse drove the real dashboard against real data, and that live pass caught two bugs no unit test would have — the SDK's type definitions documenting registerSlot's argument order wrong (an upstream doc bug, patch drafted), and fast one-shot hermes -z runs exiting before the write buffer flushed, silently dropping data.

The token bill. Planning consumed 225k output tokens; implementation, 374k — about 600k output tokens total, 740k counting fresh input.

Decision points

The design plan went through five reviews (eng ×2, design ×1, Codex ×2) and accumulated ~25 tagged decisions. The ones that shaped the system most:

• Calibration over estimation. Token estimators drift 10–15% from billed numbers, so estimated category buckets are rescaled per API call to sum exactly to billed prompt tokens. The scale factor is stored as a health metric — when the chars/4 estimator overestimated real traffic by ~2×, calibration absorbed it exactly as designed, and a self-correcting per-model ratio later tightened the estimator itself.
• Frozen category IDs, evolving mapping rules. The spec wanted categories that "self-improve" but stay consistent enough for trend analysis. The resolution: category IDs (and their chart palette slots) are frozen forever; only the rules that map traffic into them evolve, as versioned rows with every change logged to a human-readable EVOLUTION.md. Rollups recompute lazily under new rule versions so historical trend lines stay comparable.
• Fail-open recorder. Instrumentation must never break the agent. Recorder hooks catch everything, log once, and no-op; five consecutive errors trip a circuit breaker that disables recording for the session and surfaces the degradation in the dashboard — visible gaps, never silent ones, never agent breakage.
• Split suggestion gates. Deterministic findings appear after 3 sessions; LLM suggestions wait for 10. Backfilled history never counts toward either gate, because estimated attribution isn't strong enough evidence to recommend config changes.
• Suggestion lifecycle by fingerprint. Suggestions carry a canonical target-derived fingerprint, so a dismissal or "done" cascades across refreshes and across detector/LLM duplicates of the same finding. A dismissed suggestion only resurrects if its estimated savings at least double. The inheritance is a single SQL statement reading only user-actioned rows — a 100-round two-thread race test pins the semantics.
• Copy-paste plans, not auto-mutation. The plugin never touches Hermes config. Remediation is a plan you paste into a chat session, which keeps the trust model simple (localhost dashboard, no privileged writes) and keeps the human in the loop on capability trade-offs.
• Bounded meta-cost. The tokens Token Lens spends analyzing tokens are ledgered under audit purpose tags, displayed in a quiet footer, and hard-capped per suggestion refresh (50k default).
• Two-process SQLite discipline. Every multi-statement write opens with BEGIN IMMEDIATE — Python's default deferred transactions upgrade read→write mid-flight and fail SQLITE_BUSY while ignoring busy_timeout. WAL + busy_timeout + short immediate transactions, all three, always.

Verification

95 unit tests, two e2e shell flows (install, suggestion unlock), and LLM evals for both the generator (17/17, including a planted dead-server waste case) and the evaluator (strong/vague/dishonest suggestion fixtures). Live QA against real traffic verified the core invariant — category buckets summing exactly to billed prompt tokens — and the first real detector finding: an enabled-but-unused MCP server costing 3.3k schema tokens on every call.

Next

• Submit the upstream patches to hermes-agent (an SDK type-definition fix found live, plus a full-fidelity tools passthrough for attribution-grade schema costing).
• Replay lab (designed, deferred): a counterfactual simulator that replays your own session history under hypothetical configs — "what if MCP X was disabled?" — to attach evidence-backed projected savings to every suggestion. Its per-call data prerequisite has been accumulating since install.