Hermes OTEL Usage

Content-free, metrics-only OpenTelemetry plugin for Hermes Agent. Centrally instrumented runtimes report aggregate token counters through record_usage(...). Session attribution is an explicit opt-in; the default does not export session identifiers.

Codex usage is reported centrally by Hermes' Codex runtime, including the opaque credential_id selected for each call. The plugin intentionally has no generic API lifecycle hook, so provider ownership is explicit and cannot double count the same request through two layers. Other centrally instrumented runtimes may call hermes_plugins.otel_usage.record_usage(usage, provider=..., model=..., platform=..., request_id=...). Reusing the same request ID is counted once per route; missing/unavailable telemetry fails open.

Metric contract

• OTLP/HTTP metrics endpoint: configured per installation
• metric: hermes.tokens.total
• temporality: cumulative, monotonic counter
• attributes:
  • provider
  • model (canonical model ID)
  • platform
  • type: exactly cache_read, input, output, or reasoning
  • session.id: optional owning Hermes session identifier (disabled by default)

It does not export prompts, responses, conversation history, tool arguments/results, traces, logs, or pricing.

Architecture

The plugin uses the official OpenTelemetry SDK in the Hermes process, but does not install it into Hermes' managed virtual environment. An explicit bootstrap installs exact hash-locked packages into:

$HERMES_HOME/plugin-envs/otel_usage/<python-cache-tag>/site-packages

At runtime that target is appended directly to sys.path without processing .pth files. Hermes' own packages retain precedence, so the plugin target cannot downgrade or shadow a core dependency. The target is outside the Hermes checkout and venv, so a Hermes update or venv recreation does not remove it. Named profiles under $HERMES_HOME/profiles/<name> resolve this dependency target and the privacy-safe plugin state against the shared Hermes root, allowing standalone profile/CLI workers to use the same installation and OTEL configuration. No sidecar, subprocess, IPC, runtime installer, or local Hermes source/lockfile modification is used.

Codex compatibility

Credential-aware Codex accounting requires Hermes Core patch 0120-codex-otel-usage-accounting.patch from hermes-local-patches (or equivalent upstream runtime support) together with this plugin at version 2.5.0 or newer. The Core runtime owns capture for interactive CLI, hermes chat -q one-shots, named-profile processes, gateway turns, cron/Kanban workers, auxiliary calls, memory flush, compression, and Codex App Server calls because those surfaces converge on agent.codex_runtime.

Upgrade the plugin to 2.5.0 before or together with the credential-aware Core patch. Version 2.5.1 also isolates each credential route's cumulative OTEL series with an unlinkable random route-specific service.instance.id; this is required when multiple routes share one receiver-normalized service name. Existing single-account installations need no configuration change while codex_credential_routes is absent. Once that key is present—even as an empty map—Codex route mode is fail-closed for attribution. Do not add a second Codex credential until its complete route map is configured.

Do not combine Core patch 0120 with a pre-2.4 plugin that still registers the generic post_api_request hook; that legacy combination can double count Codex requests. A custom program that bypasses agent.codex_runtime and calls the provider SDK directly remains outside this contract.

Install

hermes plugins install https://forgejo.horotw.dev/HoroTW/hermes-otel-usage.git --enable
~/.hermes/hermes-agent/venv/bin/python ~/.hermes/plugins/otel_usage/bootstrap.py
~/.hermes/hermes-agent/venv/bin/python ~/.hermes/plugins/otel_usage/bootstrap.py --check

Run bootstrap.py with the same Python interpreter used by Hermes. The package target is scoped to that interpreter's ABI. Bootstrap builds a fresh private target from hash-locked wheels, verifies the complete locked distribution/version set and real SDK imports in an isolated interpreter, computes an exact SHA-256 file manifest, and only then swaps it into place. Because the runtime appends the target after Hermes' own sys.path, any OpenTelemetry package already in the host environment wins; bootstrap and --check therefore also import the plugin's modules host-first (host site-packages, then the target) and fail if a host package is incompatible with the locked SDK, for example a newer host opentelemetry-api that no longer provides a private API the locked SDK imports. If Hermes runs from a managed install environment rather than ~/.hermes/hermes-agent/venv, use that environment's interpreter: the shown path is only the default checkout layout. The canonical target is sealed read-only so normal imports cannot create unmanaged bytecode files.

--check independently rebuilds the hash-locked reference and compares every regular file in the installed target against it. Modified, missing, unmanaged, linked, special, source, bytecode, metadata, and injected RECORD artifacts therefore fail verification. Because this deliberately reconstructs the immutable reference, uv and either its package cache or network access are required for --check.

The swap is transactional: the canonical path is checked again after replacement. A failed candidate is moved out of the canonical path, and the previous target is restored only if it matched the fresh reference before the swap. Re-run bootstrap when the Python ABI or committed dependency lock changes; running it again also replaces stale or altered target contents.

Configuration follows the current deployment contract. Standard OTEL environment variables take precedence:

export OTEL_SERVICE_NAME=hermes-agent
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://collector.example/v1/metrics
export HERMES_OTEL_EXPORT_INTERVAL_MS=60000
export HERMES_OTEL_FORCE_FLUSH_ON_EVENT=true

OTEL_EXPORTER_OTLP_ENDPOINT is also supported and receives the standard /v1/metrics suffix. For backward compatibility, an existing plugins.entries.otel_usage block in ~/.hermes/config.yaml remains a fallback:

plugins:
  entries:
    otel_usage:
      enabled: true
      service_name: hermes-agent
      metrics_endpoint: https://collector.example/v1/metrics
      export_interval_ms: 60000
      force_flush_on_event: true
      include_session_id: false

Provide OTLP authorization through standard OTEL_EXPORTER_OTLP*_HEADERS environment variables. Keep credentials outside the repository and config.yaml.

Optional session attribution (2.7.0+)

Set the YAML boolean plugins.entries.otel_usage.include_session_id: true to add session.id to token datapoints. It applies to both the legacy route and configured Codex credential routes. Omitted, false, strings such as "true", and invalid values do not opt in. There is no environment-variable override for this setting.

The companion Core must pass session_id= to record_usage(...). Session-aware Core captures the owning identity before main Responses and auxiliary requests, passes it explicitly through the exporter thread, and also attributes App Server usage. It never recovers a missing session from process-global environment variables. A missing/empty identity produces an ordinary datapoint without session.id, not a guessed or synthetic session. Other callers may use the optional keyword directly.

With opt-in enabled, every session has its own cumulative series, zero baseline, and int64 overflow guard. Request-ID deduplication and account routing are unchanged; no duplicate aggregate series is emitted. Disabling the option restores the original four-attribute metric shape. Session IDs are not added to the operational state files.

Privacy and cardinality: identifiers let backend readers correlate calls belonging to one session, even though no prompts or content are exported. Each session increases SDK memory and backend time-series cardinality; retention and SDK/receiver cardinality limits still apply. This is observability, not a durable per-session accounting ledger. service.instance.id remains a process/route identity and is not replaced by session.id.

Configuration is read at initialization. Install both companion versions and start a fresh process (or perform an authorized gateway restart) to activate changes. A new plugin with an older Core continues exporting without session IDs; a new Core detects older plugin signatures and keeps token export working without the optional keyword.

No OTLP protocol or receiver schema change is required for an additional metric attribute when the receiver already retains arbitrary attributes. A backend rollup or dashboard that discards the session dimension may still need a query/rollup change to display it; raw-data support alone does not prove that existing aggregates preserve it.

Account-aware Codex routes

plugins.entries.otel_usage.codex_credential_routes is an optional mapping from the exact opaque Codex credential ID reported by Hermes Core to one non-secret route object:

plugins:
  entries:
    otel_usage:
      enabled: true
      service_name: hermes-agent                 # legacy/non-Codex route
      metrics_endpoint: https://collector.example/v1/metrics
      codex_credential_routes:
        "<opaque-credential-id-a>":
          service_name: hermes-codex-account-a
          metrics_endpoint: https://collector-a.example/v1/metrics
          headers_env: HERMES_CODEX_A_OTLP_HEADERS
        "<opaque-credential-id-b>":
          service_name: hermes-codex-account-b
          metrics_endpoint: https://collector-b.example/v1/metrics
          headers_env: HERMES_CODEX_B_OTLP_HEADERS
          usage_factor: 0.1

Set each named environment variable to OTLP header syntax; percent-encoding is accepted:

export HERMES_CODEX_A_OTLP_HEADERS='Authorization=Bearer%20<account-a-token>'
export HERMES_CODEX_B_OTLP_HEADERS='Authorization=Bearer%20<account-b-token>'

The route contract accepts service_name, metrics_endpoint, headers_env, optional dataset, and optional usage_factor (default 1). usage_factor scales only token buckets emitted through that exact credential route; use it for a real account-specific accounting factor, not for load testing. The effective factor is also emitted as the non-secret resource attribute telemetry.usage.factor, so downstream normalization can distinguish already-scaled series from legacy history. headers_env is the name of an environment variable, never a credential value. Route secrets are read only from that environment variable and are never metric attributes, state-file fields, or log values. Endpoint, headers, and service name are exporter/resource state and are not metric labels.

When codex_credential_routes is absent, all events retain the legacy process-global route behavior. When the key is present—even as an empty map or with only malformed entries—Codex route mode is active: every openai-codex event must carry a credential ID with a complete, initialized mapping. Missing, unknown, incomplete, malformed, or uninitializable routes are dropped for telemetry (record_usage(...) returns False) without interrupting the model call and never fall back to the legacy account. Non-Codex events continue to use the legacy global route. To keep legacy single-account Codex accounting, omit the key entirely. Every configured Codex route owns an independent meter provider, exporter, counter, cumulative-series guard, totals, dirty state, and request-ID dedupe window.

A privacy-safe operational state file is written atomically with mode 0600 under $HERMES_HOME/state/otel_usage.json. Codex routes use separate otel_usage-codex-<credential-id-hash>.json files. They never contain the raw credential ID, OTLP endpoint, or credentials.

Debug-only controls are documented separately in DEBUG.md.

Avoid duplicate accounting

Run only one Hermes token exporter for a given call stream. Disable hermes_otel or any exporter producing hermes_llm_*_tokens unless the receiving rollup has explicit deduplication semantics. Repeated callbacks carrying the same non-empty request_id are deduplicated in memory within the running Hermes process.

Development

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt pytest
pytest -q

The integration tests create a sterile host venv, bootstrap the external target, load the plugin using Hermes' package-loader shape, delete/recreate the host venv, and decode a real OTLP protobuf request received by a local HTTP collector.

tests/test_session_attributes.py verifies default-off behavior, strict boolean opt-in, independent session baselines/counters, absent-session handling, and legacy/account routes. To additionally exercise the companion Core's actual asynchronous exporter through this plugin into the local HTTP collector, run:

PYTHONPATH=/path/to/session-aware/hermes-agent \
  HERMES_TEST_CORE_PATH=/path/to/session-aware/hermes-agent \
  pytest -q tests/test_session_attributes.py

Use a Hermes-capable interpreter for this optional integration run. Without that variable, only the four companion-Core cases are skipped. Run bootstrap/integrity tests with a temporary HOME/HERMES_HOME, a dedicated UV_CACHE_DIR, and UV_LINK_MODE=copy: intentional negative package-integrity tests must not mutate a shared hardlink cache or production dependency target.

License

MIT