Hermes OTEL Usage

Privacy-safe, metrics-only OpenTelemetry plugin for Hermes Agent. Centrally instrumented runtimes report aggregate token counters through record_usage(...).

Codex usage is reported centrally by Hermes' Codex runtime, including the opaque credential_id selected for each call. The plugin intentionally has no generic API lifecycle hook, so provider ownership is explicit and cannot double count the same request through two layers. Other centrally instrumented runtimes may call hermes_plugins.otel_usage.record_usage(usage, provider=..., model=..., platform=..., request_id=...). Reusing the same request ID is counted once per route; missing/unavailable telemetry fails open.

Metric contract

• OTLP/HTTP metrics endpoint: configured per installation
• metric: hermes.tokens.total
• temporality: cumulative, monotonic counter
• attributes:
  • provider
  • model (canonical model ID)
  • platform
  • type: exactly cache_read, input, output, or reasoning

It does not export prompts, responses, conversation history, tool arguments/results, traces, logs, or pricing.

Architecture

The plugin uses the official OpenTelemetry SDK in the Hermes process, but does not install it into Hermes' managed virtual environment. An explicit bootstrap installs exact hash-locked packages into:

$HERMES_HOME/plugin-envs/otel_usage/<python-cache-tag>/site-packages

At runtime that target is appended directly to sys.path without processing .pth files. Hermes' own packages retain precedence, so the plugin target cannot downgrade or shadow a core dependency. The target is outside the Hermes checkout and venv, so a Hermes update or venv recreation does not remove it. Named profiles under $HERMES_HOME/profiles/<name> resolve this dependency target and the privacy-safe plugin state against the shared Hermes root, allowing standalone profile/CLI workers to use the same installation and OTEL configuration. No sidecar, subprocess, IPC, runtime installer, or local Hermes source/lockfile modification is used.

Codex compatibility

Credential-aware Codex accounting requires Hermes Core patch 0120-codex-otel-usage-accounting.patch from hermes-local-patches (or equivalent upstream runtime support) together with this plugin at version 2.5.0 or newer. The Core runtime owns capture for interactive CLI, hermes chat -q one-shots, named-profile processes, gateway turns, cron/Kanban workers, auxiliary calls, memory flush, compression, and Codex App Server calls because those surfaces converge on agent.codex_runtime.

Upgrade the plugin to 2.5.0 before or together with the credential-aware Core patch. Version 2.5.1 also isolates each credential route's cumulative OTEL series with an unlinkable random route-specific service.instance.id; this is required when multiple routes share one receiver-normalized service name. Existing single-account installations need no configuration change while codex_credential_routes is absent. Once that key is present—even as an empty map—Codex route mode is fail-closed for attribution. Do not add a second Codex credential until its complete route map is configured.

Do not combine Core patch 0120 with a pre-2.4 plugin that still registers the generic post_api_request hook; that legacy combination can double count Codex requests. A custom program that bypasses agent.codex_runtime and calls the provider SDK directly remains outside this contract.

Install

hermes plugins install https://forgejo.horotw.dev/HoroTW/hermes-otel-usage.git --enable
~/.hermes/hermes-agent/venv/bin/python ~/.hermes/plugins/otel_usage/bootstrap.py
~/.hermes/hermes-agent/venv/bin/python ~/.hermes/plugins/otel_usage/bootstrap.py --check

Run bootstrap.py with the same Python interpreter used by Hermes. The package target is scoped to that interpreter's ABI. Bootstrap builds a fresh private target from hash-locked wheels, verifies the complete locked distribution/version set and real SDK imports in an isolated interpreter, computes an exact SHA-256 file manifest, and only then swaps it into place. The canonical target is sealed read-only so normal imports cannot create unmanaged bytecode files.

--check independently rebuilds the hash-locked reference and compares every regular file in the installed target against it. Modified, missing, unmanaged, linked, special, source, bytecode, metadata, and injected RECORD artifacts therefore fail verification. Because this deliberately reconstructs the immutable reference, uv and either its package cache or network access are required for --check.

The swap is transactional: the canonical path is checked again after replacement. A failed candidate is moved out of the canonical path, and the previous target is restored only if it matched the fresh reference before the swap. Re-run bootstrap when the Python ABI or committed dependency lock changes; running it again also replaces stale or altered target contents.

Configuration follows the current deployment contract. Standard OTEL environment variables take precedence:

export OTEL_SERVICE_NAME=hermes-agent
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://collector.example/v1/metrics
export HERMES_OTEL_EXPORT_INTERVAL_MS=60000
export HERMES_OTEL_FORCE_FLUSH_ON_EVENT=true

OTEL_EXPORTER_OTLP_ENDPOINT is also supported and receives the standard /v1/metrics suffix. For backward compatibility, an existing plugins.entries.otel_usage block in ~/.hermes/config.yaml remains a fallback:

plugins:
  entries:
    otel_usage:
      enabled: true
      service_name: hermes-agent
      metrics_endpoint: https://collector.example/v1/metrics
      export_interval_ms: 60000
      force_flush_on_event: true

Provide OTLP authorization through standard OTEL_EXPORTER_OTLP*_HEADERS environment variables. Keep credentials outside the repository and config.yaml.

Account-aware Codex routes

plugins.entries.otel_usage.codex_credential_routes is an optional mapping from the exact opaque Codex credential ID reported by Hermes Core to one non-secret route object:

plugins:
  entries:
    otel_usage:
      enabled: true
      service_name: hermes-agent                 # legacy/non-Codex route
      metrics_endpoint: https://collector.example/v1/metrics
      codex_credential_routes:
        "<opaque-credential-id-a>":
          service_name: hermes-codex-account-a
          metrics_endpoint: https://collector-a.example/v1/metrics
          headers_env: HERMES_CODEX_A_OTLP_HEADERS
        "<opaque-credential-id-b>":
          service_name: hermes-codex-account-b
          metrics_endpoint: https://collector-b.example/v1/metrics
          headers_env: HERMES_CODEX_B_OTLP_HEADERS
          usage_factor: 0.1

Set each named environment variable to OTLP header syntax; percent-encoding is accepted:

export HERMES_CODEX_A_OTLP_HEADERS='Authorization=Bearer%20<account-a-token>'
export HERMES_CODEX_B_OTLP_HEADERS='Authorization=Bearer%20<account-b-token>'

The route contract accepts service_name, metrics_endpoint, headers_env, optional dataset, and optional usage_factor (default 1). usage_factor scales only token buckets emitted through that exact credential route; use it for a real account-specific accounting factor, not for load testing. The effective factor is also emitted as the non-secret resource attribute telemetry.usage.factor, so downstream normalization can distinguish already-scaled series from legacy history. headers_env is the name of an environment variable, never a credential value. Route secrets are read only from that environment variable and are never metric attributes, state-file fields, or log values. Endpoint, headers, and service name are exporter/resource state and are not metric labels.

When codex_credential_routes is absent, all events retain the legacy process-global route behavior. When the key is present—even as an empty map or with only malformed entries—Codex route mode is active: every openai-codex event must carry a credential ID with a complete, initialized mapping. Missing, unknown, incomplete, malformed, or uninitializable routes are dropped for telemetry (record_usage(...) returns False) without interrupting the model call and never fall back to the legacy account. Non-Codex events continue to use the legacy global route. To keep legacy single-account Codex accounting, omit the key entirely. Every configured Codex route owns an independent meter provider, exporter, counter, cumulative-series guard, totals, dirty state, and request-ID dedupe window.

A privacy-safe operational state file is written atomically with mode 0600 under $HERMES_HOME/state/otel_usage.json. Codex routes use separate otel_usage-codex-<credential-id-hash>.json files. They never contain the raw credential ID, OTLP endpoint, or credentials.

Debug-only controls are documented separately in DEBUG.md.

Avoid duplicate accounting

Run only one Hermes token exporter for a given call stream. Disable hermes_otel or any exporter producing hermes_llm_*_tokens unless the receiving rollup has explicit deduplication semantics. Repeated callbacks carrying the same non-empty request_id are deduplicated in memory within the running Hermes process.

Development

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt pytest
pytest -q

The integration tests create a sterile host venv, bootstrap the external target, load the plugin using Hermes' package-loader shape, delete/recreate the host venv, and decode a real OTLP protobuf request received by a local HTTP collector.

License

MIT