Teaching agents to read a flame graph

A flame graph is one of the most information-dense pictures in engineering. A person learns to read one in an afternoon: wide frames are expensive, tall stacks are deep, and the plateau at the top is usually where the time goes. An agent looking at the same data sees thousands of frames with no sense of which ones matter.

When we first gave Keel raw profiles during investigations, it did what a tired engineer does at three in the morning. It latched onto the widest frame it could find and declared victory. Most of the time that frame was a runtime function that sits under every request, wide because it is everywhere, not because it is slow.

Width is not cost

The first fix was to stop presenting frames in isolation. Keel now compares a profile with a baseline captured from the same service during a healthy window, and reasons about the difference. A frame that is wide in both profiles is background. A frame that grew by forty percent since the last deploy is a lead.

def frame_deltas(current, baseline):
    now_total = sum(current.values())
    then_total = sum(baseline.values()) or 1
    for key, samples in current.items():
        share_now = samples / now_total
        share_then = baseline.get(key, 0) / then_total
        yield key, round(share_now - share_then, 4)

Collapsing the boring parts

The second fix was compression. We fold frames that belong to the language runtime, the web framework and common libraries into labeled groups, so the agent sees framework routing instead of forty nested calls. What remains is mostly the code your team wrote, which is where fixes actually land.

• Runtime and standard library frames are grouped by subsystem.
• Frames that appear in more than 95 percent of samples are marked as ambient.
• Each remaining frame carries its owning repository and the last change that touched it.

What changed in practice

Across a month of internal investigations, the share of profile-based conclusions that a reviewing engineer accepted on first read rose from roughly half to just over four in five. More importantly, the agent stopped producing confident answers from ambiguous profiles. When the difference between baseline and current is within noise, Keel now says so and moves on to other evidence.

The most useful thing an investigator can say is sometimes that a piece of evidence does not point anywhere.

Profiles are one input among many. The next step is joining them with deploy history, so a hot frame arrives with the pull request that probably made it hot.