Gateway call evidence
The Rust Gateway exports powercontext.gateway-evidence.v1 JSON lines on stdout by default.
Operational tracing goes to stderr. No external telemetry platform is required. Configure your existing
log collector's rotation and retention; redirecting stdout to a file does not bound that file's size.
# Keep evidence separate from operational logs; use a collector with bounded retention.
./gateway/target/release/pc-gateway | your-existing-log-collector
# Explicitly disable evidence. The startup diagnostic reports missing evidence coverage.
PC_GATEWAY_EVIDENCE=off ./gateway/target/release/pc-gatewayThe application composes JsonEvidenceSink, not the unbounded BufferedEvidenceSink retained for tests/demos.
It keeps at most 1024 queued records plus one worker-owned record. Each accepted record has bounded identifiers,
typed counts/enums and at most 4096 serialized bytes including the newline. Submission uses try_send:
when full, the new record is discarded. There is no request-side I/O, writer join or drain wait.
A stalled stdout can occupy the one exporter thread but cannot accumulate more threads or queue entries.
The rest of the Gateway still has its own request/response buffers and concurrency limits.
Every readable line has schema, operation=gateway.model_call, unit (call, attempt, delivery),
event (the same three values), outcome, duration_ms, evidence, and loss. The process-local cumulative loss.dropped,
loss.write_failed and loss.written count discarded submissions, failed/unconfirmed record writes, and completed
write-and-flush operations respectively, sampled before that line's write. Recovered output reports earlier losses;
an embedding host can also read JsonEvidenceSink.stats() during congestion. No execution IDs are metric labels.
These are best-effort records: overflow can remove a call terminal or an attempt, a failed/partial write can corrupt
one line, and process exit can lose queued records. Readers should skip malformed/blank lines; after a write failure,
the exporter restores a newline boundary before the next record. Counters reset on restart. There is no replay or
billing/audit durability guarantee, and no financial reconciliation, token savings or task-success claim.
The production host also starts one independent stderr health thread. It samples JsonEvidenceSink.stats() once
per second: the first nonzero write_failed emits gateway.evidence.write_failed at ERROR, and the first nonzero
dropped emits gateway.evidence.dropped at WARN. Each notification occurs at most once per sink. Either first
notification includes a gateway.evidence.health summary; thereafter a summary is emitted every 30 seconds even
without new requests or recovered stdout output. Healthy summaries use INFO; summaries with any cumulative loss use
WARN. Normal scheduling and writable stderr are required for timely reporting; RUST_LOG filters still apply.
These stderr events contain fixed messages and only three cumulative unsigned record counts: dropped (rejected
submissions, including invalid records/full or disconnected queue), write_failed (failed/unconfirmed serialization,
write or flush), and written (successful write-and-flush operations). They carry no execution identity, writer error,
content, URL, credential or path. Counts reset on process restart and are independent atomic snapshots, not a reconciled
ledger. The embedded JSONL loss fields and schema are unchanged. This host-owned tracing output does not add Python
operational fields or change its JSON/console allowlists.
A stalled stdout writer does not stop health reporting. A stalled stderr can occupy the one health thread; that
thread releases its sink reference before I/O, has no record queue, and cannot delay evidence submissions or sink
destruction. Dropping the sink retires the reporter on its next poll; shutdown never joins either potentially blocked
thread, so there is no guaranteed final summary or drain. Keep stderr collected separately from stdout: losing or
filtering both outputs cannot establish that there was no traffic. Embedding hosts that use JsonEvidenceSink directly
own their health reporting; the library does not configure logging or start a stderr reporter.
If either worker cannot start, the application reports a fixed diagnostic and uses Noop without changing business results.
Operational vocabulary and exporter boundary
The Server operational contract and the managed stack from
#2388 use bounded operation/unit/outcome fields.
Gateway evidence uses the same field meanings with its own operation, gateway.model_call: it counts one local
managed model execution, not a Core context_check application operation or a worker inference generation.
unit=call counts logical executions, attempt counts real transport handoffs, and delivery describes local
framework observations. Do not sum the three units or count delivery events as additional calls.
| Evidence facts | Top-level operational outcome |
|---|---|
Call release_authorized or degraded, observed upstream status < 400 | success |
| Same terminals with observed upstream status >= 400 | failure |
| Same terminals with missing upstream status | null |
Call blocked, withheld, admission_expired | blocked |
Call send_indeterminate, not_sent, gate_paused, or a confirmed stopped cause | failure |
Call abandoned, regardless of observed interruption intent/deadline | null |
Attempt delivered | success |
Attempt possibly_sent on future destruction | null |
| Other terminal attempt outcomes | failure |
| Any delivery observation | null |
Null is unavailable classification, not success or noop. In particular, destruction-time cancellation intent
is not a confirmed cancellation cause and is not projected as cancelled. Keep the nested terminal, sending,
interruption and attempt outcome fields: the common outcome does not replace them. success here does not prove
strict release authorization for degraded output, client receipt, task adoption or task success.
Top-level duration_ms copies call/attempt duration; it is null for delivery, whose elapsed_ms has a different origin.
The existing managed Collector receives Server OTLP logs and reduces them through the Server field allowlist.
It does not ingest Gateway stdout by default, and that transform would discard nested schema/evidence/loss.
Keep the complete JSONL in a collector with bounded retention; feeding it through the Server formatter or current
OTLP pipeline is not a supported evidence-preserving path. The common fields enable vocabulary-compatible queries;
this implementation adds no Prometheus metrics, OTLP exporter or duplicate monitoring stack. External Collector/Loki
storage and readback have not been verified for this stream.
Correlation and coverage
call_id identifies one execution and joins its call, numbered attempt, and delivery records.
It is generated once in request control and reused through completion and response-body lifetime.
request_id_fingerprint is a keyed process-local fingerprint of the bound request ID, or null when none exists;
target_fingerprint fingerprints the actual transport URL, never the requested model or client-claimed target.
Redirects remain disabled. Fingerprints use a random process-local key and are not portable identities or a
cryptographic anonymization promise. They never export a raw URL, request ID, cache key, conversation ID,
Scope, provider error, model, credential, prompt or response body. Records do not establish task identity or
Source projection coverage. No task association is available unless the embedding host independently supplies it.
Coverage starts when Executor::run is polled. Authentication, invalid-body, middleware rejection and other ingress
failures before execution are outside this stream. A complete model response does not imply successful release.
attempts counts actual transport handoffs, including a degraded send; an expired admission has zero attempts.
Retries keep one call_id and increment attempt numbers only after fresh admission. An uncertain attempt is never
retried. The legacy demo-only record_not_sent(0, ...) diagnostics for preparation/capture/degradation are not
exported and are not model-send facts.
Terminal and send facts
The call record contains terminal, sending, attempts, upstream_status, normalized usage, optional
interruption, and timing fields. Terminal values are release_authorized, blocked, withheld,
send_indeterminate, not_sent, gate_paused, degraded, admission_expired, and abandoned.
A known lifecycle stop is encoded as {"stopped":"deadline_exceeded"}, {"stopped":"gateway_shutdown"},
or {"stopped":"request_cancelled"}.
Lifecycle snapshots and evidence share ExecutionTerminal and ModelSendState from pc-gateway-types;
ManagedOutcome has one terminal mapping. Existing Rust import names in core and evidence remain re-exports.
sending independently reports no_send, definitely_not_sent, possibly_sent, or response_received.
A release refusal or expiry after a response preserves response_received and any reported usage.
The execution owner records confirmed deadline, shutdown, or cancellation stops. On pending-future destruction,
evidence preserves the owner's terminal and sending facts; destruction without a known stop remains abandoned.
The drop-time interruption field reports cancellation_requested if the shared cancellation signal was set,
otherwise deadline_elapsed if the host deadline has elapsed, otherwise future_dropped. This field describes
observations at destruction; the shared execution terminal identifies a confirmed stop cause. Evidence adds no
cancellation authority or timer. Cancellation during transport remains possibly_sent, with unknown usage,
never not_sent.
Each attempt contains attempt, outcome, started_ms, duration_ms, upstream_status, normalized usage,
response_first_byte_ms, and response_complete_ms. Outcomes are possibly_sent (interrupted pending send),
delivered, upstream_error (complete HTTP response with status >= 400), connect_refused, rejected_locally,
lost_after_handoff, timeout_awaiting_response, and buffer_limit_exceeded. A partial-response failure preserves
its observed status/first-byte timing but keeps completion and usage unknown. timeout_awaiting_response covers
both waiting for response headers and timing out while reading the body, including after a partial body;
lost_after_handoff covers other transport/body failures. Neither permits automatic retry. Once the transport reports delivery,
usage survives decoding/release failures. A dropped pending transport cannot provide its private partial observations.
A response-buffer overflow produces a withheld call with possibly_sent, one buffer_limit_exceeded attempt,
unknown usage/completion, and any observed status/first-byte timing. It releases no response prefix and does not retry,
including degraded passthrough. This preserves the capacity-limit decision separately from possible billing.
Usage mapping
usage_mapping is open_ai_chat or open_ai_responses, selected from the actual wire renderer, not the client model. An unrecognized renderer has null mapping and unknown usage.
Each of usage.input, usage.output, usage.cache_read, and usage.cache_write has tokens (nonnegative integer
or null) and completeness (known, estimated, unknown, reconciled). This implementation emits only
provider-reported known values or unknown; it performs no estimates or reconciliation.
| Quantity | OpenAI Chat | OpenAI Responses |
|---|---|---|
| Input | usage.prompt_tokens | usage.input_tokens |
| Output | usage.completion_tokens | usage.output_tokens |
| Cache read | usage.prompt_tokens_details.cached_tokens | usage.input_tokens_details.cached_tokens |
| Cache write | Unknown | Unknown |
Input includes cached input: do not add cache reads to input or subtract them from the reported input field.
No Anthropic cache-creation/write fields are guessed into the OpenAI mapping. Missing, null, malformed or negative
counts remain unknown; a reported zero is known zero. total_tokens alone cannot establish input/output counts.
SSE uses an explicitly present usage object from a Chat usage chunk or Responses event's response.usage.
It retains the last such object, without summing cumulative chunks. Missing/malformed SSE usage remains unknown.
The SSE media type is compared case-insensitively after separating parameters. Usage and first-content inspection
ignore exactly one leading UTF-8 BOM; first-content offsets still refer to the original outgoing bytes, including
that BOM. Response bytes and the original Content-Type are preserved.
This does not expand the Core's accepted streaming protocols.
Timing and HTTP delivery
All durations are monotonic milliseconds, truncated to integers. Null means unavailable.
| Field/boundary | Meaning |
|---|---|
Call duration_ms | Request-control creation to executor terminal/drop; includes ingress time after control creation |
Attempt started_ms | Request-control creation to transport handoff |
Attempt duration_ms | Handoff to transport result or future destruction |
response_first_byte_ms | Transport send start, before awaiting headers, to first nonempty HTTP body chunk |
response_complete_ms | Same transport origin to complete buffered response |
upstream_ttft_ms | Always null: HTTP JSON/SSE bytes do not establish a model's first token |
release_decision_ms | Request-control creation to before-release check result, including error/refusal |
release_authorized_ms | Request-control creation to validated, unexpired release after optional capture |
Delivery elapsed_ms | Request-control creation to the named framework observation |
The executor handoff and the transport's own send start are separate origins. Their exact offset is not recorded;
started_ms + response_first_byte_ms or started_ms + response_complete_ms is not an exact call-relative timestamp.
A delivery record reports framework_handoff when the released/degraded body is placed in the HTTP response,
first_body_frame when the framework first polls a nonempty data frame, first_model_content when it polls
the frame containing the first encoded byte of recognized model output, body_completed when the wrapped body
reports completion, or body_dropped if dropped before observed completion. Strict release authorization precedes
framework handoff and both first-frame/content observations. A degraded response has no release authorization timestamp.
The content boundary recognizes nonempty assistant text, refusal strings, and function-call names/arguments in OpenAI Chat/Responses JSON and their supported SSE output, including Responses replay. It skips role-only, usage, reasoning, error and other control events. An item-added event containing an actual function name or output text can establish content; the event name alone cannot. The locator inspects the exact buffered outgoing body and passes only a byte offset to the observer. The observer counts polled data bytes across arbitrary frame splits; it neither retains content nor changes response bytes. This measures the frame containing the first encoded content byte, which can be a JSON escape, rather than waiting for a complete decoded character or SSE event.
Control-only, empty, malformed and unsupported output has no first_model_content event. Neither does a body dropped
before its content frame is polled. Absence is unavailable evidence, not a zero duration; consumers must also account
for the sink's loss counters. The first body frame may contain only response.created or other SSE controls, so G8
must consume first_model_content separately. Both observations can share a timestamp when one frame contains both.
No boundary proves a socket write, client receipt, rendering or adoption. The current host cannot observe actual
socket writes. Non-streaming Responses JSON's first byte and replayed SSE output are not provider TTFT.
Deployment and compatibility
Operational diagnostics identify TCP/UDS connection mode without logging the configured Core URL or socket path.
Invalid injection settings omit the supplied value; steering-file rejection uses fixed reasons without a file path.
Operators should inspect their protected configuration for endpoints and paths.
The additive top-level outcome and duration_ms fields preserve the nested domain records. Consumers must accept
null classifications, including abandoned execution and delivery observations.
The additive delivery.boundary=first_model_content value requires consumers with exhaustive enum matching to
accept the new value; consumers must not assume a fixed number of records per call.
Collectors that previously consumed human tracing from stdout must move that input to stderr and parse evidence
stdout separately. No database migration, new HTTP endpoint, Python operational field or OpenAPI regeneration is
required. The JSON schema is separate from the Server's allowlisted Python log fields and existing exporters.
The Gateway's client-visible HTTP 503 gate_paused body normalizes detail for Core-unavailable errors to
gate_unavailable, and for malformed Core-response errors to malformed_gate_response, instead of exposing raw
diagnostic text. Clients must not depend on the former free-form detail; use the status and decision for handling.
Custom embedding sinks must keep synchronous callbacks bounded, nonblocking and free of I/O; route blocking writers
through JsonEvidenceSink and keep one sink per host. Custom writers must return I/O errors without panicking.
Rust integrations consume the nested per-field CallEvidence.usage instead of the demo record's former flat token/completeness fields. There is no production record-store migration. They must replace RawResponse.ttft_ms with the explicit first-byte/completion fields and handle
SendOutcome::IncompleteResponse as indeterminate, preserving its progress and never retrying it.

