Research9 min read
A compact grammar for production telemetry
Raw logs and traces blow through any context window. This is the event grammar Keel uses to keep a week of telemetry in a few thousand tokens.
By Noor Haddad
An hour of logs from a modest production system can run to millions of lines. Traces and metrics add more. No model can read all of it, and most of it would not help if it could. The question we keep returning to is how much of that stream an agent needs in order to reason well about what happened.
Events, not lines
Keel does not hand raw telemetry to its agents. It first rewrites the stream into a small vocabulary of typed events, each tied to a node in the System Map. A deploy, a config change, a restart, an error burst and a latency shift are all events. A log line that repeats with different request ids is one event with a count, not ten thousand lines.
t+00:14 deploy playback-api v4.18.2 pr=2148
t+00:16 config session-cache eviction=volatile-ttl
t+00:21 shift playback-api ttff.p95 410ms -> 1.9s
t+00:22 burst users-db timeouts x312How small is small enough
We measured investigation quality against the size of the context the agent received. Accuracy climbed quickly as we added typed events, flattened out around a few thousand tokens per incident and then slowly declined as we added raw logs back in. More text made the agent slower and easier to distract.
| Context given to the agent | Tokens per incident | Correct root cause |
|---|---|---|
| Raw logs, sampled | 118,000 | 52% |
| Typed events only | 3,400 | 81% |
| Typed events plus targeted log excerpts | 5,900 | 87% |
Keeping the escape hatch
Compression loses detail, and sometimes the detail is the answer. Every event keeps a pointer back to the raw records it summarizes, and agents can request them when a hypothesis depends on a specific value. In practice they do this in about one investigation in six, which is the behavior we wanted: start from the summary, go to the source when it matters.
