Ai Observability
On this page
AI: agents, models and MCP servers
The AI page shows what the agents, models and MCP servers of bx-ai are doing on this server: how many runs, how they end, how long they take, what they cost, how every agent is composed, and how the MCP servers you host are doing. It sits in the Modules group, because it is an integration with the bx-ai module.
It is a BoxLang+ page. Without a license it shows three counters and nothing else (see Without BoxLang+).
Turn it on
The integration is off until an admin turns it on, under Modules, Integrations, on the AI page itself, or with collectors.aiobs.enabled in boxlang.json. With it off Lens registers no listener. It needs bx-ai, which ships with BxLens.
How the numbers are gathered
| Source | What it gives |
|---|---|
| The events of bx-ai (every agent, no code change) | Runs, model calls, tokens, tool calls, MCP calls, searches, speech, images, embeddings, rate limits and errors. |
The Lens recorder (a passive middleware that Lens attaches to each agent as it is created, when collectors.aiobs.attach is on) | The exact error and time of a tool that fails, and the attempts of a model call. bx-ai announces nothing when a tool throws, so without it a failed run is closed by a sweep a few seconds later. |
What bx-ai keeps (agent.getConfig(), the registries, the statistics of an MCP server) | How an agent is made, what is registered, and the numbers of the MCP servers you host. |
The recorder never changes a run. Every hook catches its own errors and lets the run continue. Names and order of the middleware of an agent come from its configuration. bx-ai announces no events for middleware, so Lens reports what it sees at the edges of a run (retries, tool failures, failed runs), not what each middleware decided.
Lens keeps only values taken from these objects, never the objects, so nothing is kept alive by it.
Tabs
The range buttons (1h, 24h, 7d, 15d) and the server selector apply to the dashboards (Overview, Agents, MCP, Models and Cost). The pages about one run, the composition of an agent, the registries and the MCP servers hosted here describe this server only.
One agent
Open an agent from the dashboard. It has five sections:
- Composition is a diagram of how the agent is made: its model, instructions, memory, skills, MCP servers, tools, sub-agents and middleware, each with how it performs. Click a part to go deeper.
- Runs lists only the runs of this agent. Open one for its trace.
- Tools and parts shows each tool with its calls, time, failures and share of the time of the agent, the sub-agents it delegates to, the MCP servers it calls, its models, skills and memory.
- Middleware lists the stack in the order it runs, what Lens saw, and the flight recorder middleware of bx-ai when the agent has one: its mode, how many tapes it wrote and which run each belongs to. Lens lists the tapes and does not replay them. A tape holds raw prompts and answers, so only names, sizes and counts are read.
- Instructions shows the size and a short hash of the instructions, the text when content capture is on, and the versions: a new hash starts a new version with its own runs, failures and p95, so you can see whether a prompt edit helped.
One MCP server
Open a server you host. It shows its requests, tool calls, resource reads, failures and p95, what failed grouped by error, and what to check. A call for a tool the server does not have, for example, says what the server lists instead. A second tab lists the requests and why callers were turned away (auth, API key, IP list, size). Lens watches these servers and never pauses or changes them.
The numbers of a hosted server are read from the statistics bx-ai keeps for it, so they count requests made without HTTP too. The tool table counts since the server started.
The trace of a run
A run is a tree of steps: model calls, tool calls, sub-agents, MCP calls and searches, nested the way they ran, with the time and tokens of each. A sub-agent is a tool named delegate_to_<name> in bx-ai, so Lens links the run of the sub-agent to the run that delegated. A failed tool shows its error and time. A run whose tool threw never announces its end: Lens closes it as failed as soon as the thread does something else, or a few seconds later.
What needs attention
| Rule | Limit (setting under collectors.aiobs) |
|---|---|
| An agent, a tool or a model fails on this share of its calls | failurePct (5), judged once it has minCalls (20) calls. Critical at twice the limit. |
| An agent is slow: its p95 is this many times its median, and at least 2 s | slowFactor (3) |
| Rate limits from a provider in the last 10 minutes | rateLimits (3) |
| MCP errors of one server in the last 10 minutes | mcpErrors (4) |
| A model has tokens and no price | always, as a note |
| Runs that never finished | always |
Cost
Cost comes from a price table you enter on the Settings tab: dollars per million tokens, for the input and the output of each model. Nothing is guessed. A model with no price adds no money, and the Cost tab says how many models have none. Every amount is labelled an estimate, because tokens are counted as the provider reports them and prices change.
Prompts and answers
Nothing of the text is kept by default. An admin can set collectors.aiobs.content to preview (the first 200 characters) or full (up to 20,000). Whatever is kept is redacted first: secrets, emails and numbers that look like cards are hidden. Only an admin sees the text, whatever the setting. A viewer sees the numbers.
Without BoxLang+
Free keeps three counters for the last hour: failed runs, rate limit hits and MCP errors. No run list, no names and no text. They are counted while the integration is on. The recorder is not attached on Free.
What is stored
Memory keeps the last hour on Free and two days with the store. With BoxLang+ the numbers are saved as a rollup of each UTC day (ai-rollup-YYYY-MM-DD.json), so other servers can be read and a restart does not start from zero, and finished runs are saved to the stream ai-runs. See The store. The list of runs in memory is bounded by collectors.aiobs.maxRuns (500). Retention is the one of the store (store.retentionDays).
Limits
What the page cannot show today, and why.
- No counters for each middleware. bx-ai announces no events for middleware, so Lens cannot say how many requests a guardrail blocked or how often a retry middleware fired. The Middleware tab lists the stack in order, with its names and descriptions. What Lens does show is what it sees at the edges of a run: retries, tool failures and failed runs.
- The flight recorder is listed, not replayed. For an agent with the flight recorder middleware of bx-ai, Lens shows its mode and its tapes: names, sizes and the number of steps. It never opens the content of a tape, because a tape holds raw prompts and answers, and it does not replay one.
- Tokens are never estimated. Providers that report no token numbers show none, and a model with no price adds no money.
- bx-ai announces no event when an agent run, a tool or a model call throws. The recorder gives the tool error, and a sweep closes the run.
- An agent made with
new AiAgent()instead ofaiAgent()announces no creation event, so Lens does not attach the recorder. Its runs are still counted from the other events. - A tool run by an MCP server hosted here is counted by that server's statistics, not as a tool of an agent.