---
title: Alert runbook
order: 6
description: One entry for each of the 15 BxLens alert rules: what it means, the default thresholds, what to check first, and how to tune or silence it. BoxLang+.
icon: lucide:siren
---

# Alert runbook

BxLens has 15 alert rules (BoxLang+). They are off until an admin switches them on, one by one, in **Settings, Notifications**. This page is what to read when one fires: what it means, the default limits, what to check first and how to tune it. How alerts are sent, who gets them and how they repeat is in [Notifications](../console/notifications.md).

## How to read an alert

- An alert has a **rule**, a **server** and a **subject** (a volume, an executor, a task, a host). The same problem on the same subject is one alert. A mail says **firing** once, **still firing** (a reminder) every repeat time, and **resolved** when it stops.
- The limits below are the defaults. Every rule has its own switch, severity (warning or critical) and repeat time (60 minutes). The limits are set per rule, for the whole cluster.
- Rules that need a window (for example "above 90 percent for 5 minutes") fire only when BxLens has samples for the whole window. After a restart a window rule is quiet until it has looked long enough. BxLens looks every 30 seconds and keeps two hours of samples.
- Mails and messages for a server come from the one server that was elected to send. See [Several servers](multi-server.md#who-sends-the-alerts).

## How to tune or silence

| You want | Do this |
|---|---|
| A rule never | Switch it off. |
| A rule less sensitive | Raise its limits. A window rule: lengthen the window or raise the number. |
| A rule less loud | Raise the repeat time, or lower its severity to warning. |
| Everything quiet for a while | **Pause 1 hour** or **Pause 1 day**, or the main switch. |
| One person to hear less | Raise that person's lowest severity, or untick categories for them. |
| One channel to hear less | Choose the alerts and the lowest severity of that channel. |
| Fewer mails an hour | Lower `maxPerHour` (30 by default). Critical alerts go first. |

There is no mute for a single server or a single subject. To keep one server out of a rule, fix the thing it reports, or raise the limit for all.

## The rules

### Server down

*Critical, availability.* A server that wrote before has stopped saving its health snapshot and heartbeat. Limit: **3 minutes** (1 to 1440).

- **Means.** The process is down, the host is unreachable, or the server can no longer write to the store volume. A server that cannot write looks exactly like a dead one.
- **Check first.** Is the process running? Settings, Storage on a working server: the server's *last write*. Is the store volume full or read only for that host? A server that stopped more than a day ago is called gone and raises no alert.
- **Tune.** Raise the minutes for servers that restart slowly. A planned shutdown still looks like down: pause for the maintenance.

### Deadlock

*Critical, jvm.* The JVM reports threads that wait for each other and will never continue. No limit.

- **Means.** Requests on those threads hang until the server restarts.
- **Check first.** **Threads** (BoxLang+): the red banner lists each thread, the lock it waits for and who holds it. Take a thread dump and read the BoxLang frames. Ask Lensy about it.
- **Tune.** Nothing to tune: a deadlock is never normal. Fix the code, restart to recover.

### Heap pressure

*Critical, jvm.* The heap stays above **90 percent for 5 minutes**, or the garbage collector uses more than **30 percent** of the time (checked over the same window, when the heap is below the limit).

- **Means.** The server is close to an out of memory error, or it spends its time collecting garbage instead of serving.
- **Check first.** **System**: heap, memory pools, GC time. **Caches**, **Request history** and big sessions are common causes. A heap dump (BoxLang+, off until `console.allowHeapDump`) shows what holds the memory, but it holds every secret: read [Heap dumps](../security.md#heap-dumps).
- **Tune.** Raise the percent only if the heap is sized to run hot. The real fix is more heap or less retention.

### Disk space

*Critical, capacity.* A volume is **90 percent used** or more (50 to 100), or a folder BxLens must write to (the store, the secrets, the logs) has a problem such as a read only mount.

- **Means.** Logs, the store or a heap dump can no longer be written. The store drops records and counts them.
- **Check first.** **System**, Volumes card: the fullest volume and the folders BxLens writes to. Then the store size in Settings, Storage against `store.maxMB`, the logs directory, and temporary heap dumps.
- **Tune.** The limit is per rule. The amber and red colours on the System page are the live settings `volumes.warnPct` (80) and `volumes.critPct` (90), a separate pair.

### Error spike

*Critical, errors.* **20 errors a minute** or more, measured over **5 minutes**, or a **new error group**: one first seen in the last 10 minutes and after this server started (on by default).

- **Means.** Something broke now, or after a deploy. The new error flag catches a single new kind of failure even at a low rate.
- **Check first.** **Errors**: sort by last seen, open the newest group, read the first sample and its request. Compare with the time of the last deploy. **Reports**, Boards: Highest error rate.
- **Tune.** Raise `Errors per minute` for a noisy site. Set the new group flag to 0 if new groups are frequent and not important.

### Failed requests

*Critical, traffic.* **5 percent** of the requests failed (a 5xx or an uncaught error) over **5 minutes**, with at least **50 requests**.

- **Means.** Users see errors. The minimum number of requests keeps a quiet server from alerting on one failure.
- **Check first.** **Reports**: status codes and the URLs with the most errors. **Errors** for the cause. **Datasources** and **Outgoing HTTP** for a dependency that fell over.
- **Tune.** On a small site lower the minimum requests, or lengthen the window.

### Slow site

*Warning, traffic.* The **average** response time is **2000 ms** or more over **10 minutes**, with at least **50 requests**.

- **Means.** The whole site is slow, not one URL. It is a mean, so a few very slow requests can lift it.
- **Check first.** **Reports**: p95 and the slowest URLs. **In flight** (BoxLang+) shows what is running now. **Queries**: the heaviest statement. **Datasources**: waiting threads. **System**: CPU and GC.
- **Tune.** Set the average to what is normal for your slowest healthy site.

### Datasource pool

*Critical, database.* Requests have **waited for a database connection for 2 minutes** (1 to 120): at every look in the window, at least one thread waited for a connection.

- **Means.** The pool is used up. Requests queue, then time out.
- **Check first.** **Datasources**: active against maximum, and the waits. **In flight** for requests holding connections for long. **Queries**: a slow statement that holds connections. Check for connections that are never closed.
- **Tune.** Lengthen the minutes if short bursts are normal. Raise the pool size in the datasource configuration only after you know why it filled.

### Executor saturated

*Warning, runtime.* An executor has **100 or more tasks queued**, or its health is critical.

- **Means.** Work is waiting for threads. Scheduled tasks and async work run late.
- **Check first.** **Executors**: the pool, queue and rejected tasks of the one named in the alert. **Tasks** bound to that executor.
- **Tune.** Raise the queue limit for executors that queue by design.

### Scheduled task failing

*Warning, tasks.* A scheduled task's last run **failed** and its failure count is **2 or more**.

- **Means.** A job that should run is failing. The Tasks page marks the status *failing*.
- **Check first.** **Tasks**: open the task, read the last error and the history. Run it once with *Run now* (BoxLang+, admin, `console.actions`).
- **Tune.** Raise the number if a flaky task is retried by design. The status is the last run of the task: it is *failing* until the next run succeeds. The rule's label says "in a row", but the rule compares the total failure count the Tasks page shows.

### Outgoing HTTP

*Warning, dependencies.* A host or an endpoint the app calls needs attention on the **HTTP page**. No limit of its own: it uses the limits of the HTTP collector.

- **Means.** A host is slow (p95 above `collectors.http.slowMs`, 2000 ms), fails more than `errorPct` (5) percent once it has `minCalls` (20) calls, or a request repeats the same call `repeat` (20) times, or a response is larger than `largeMB` (5). Notes (severity info) never raise an alert.
- **Check first.** **HTTP**: the *what needs attention* list, then the Calls tab (BoxLang+) for the request that made the call. Is the host itself down?
- **Tune.** Change the `collectors.http.*` limits in Settings (live). They also change the HTTP page.

### Mail

*Warning, mail.* The Mail page has a problem: bounces, a high failure rate, or a spool that does not drain. Needs the Mail integration (`collectors.mail.enabled`).

- **Means.** Failure rate above `collectors.mail.failurePct` (5 percent) after `minMessages` (10) messages, a bounce, a server with a high failure rate, or a message that waits in the spool longer than `spoolMinutes` (10).
- **Check first.** **Mail**: Overview, then Queue (bounced messages with the error) and Servers (which SMTP server fails). An admin can send a bounced message again or discard it (BoxLang+).
- **Tune.** Change `collectors.mail.*` in Settings (live).

### AI

*Warning, ai.* An agent or model needs attention on the AI page, or the cost of the last 24 hours is above your **daily budget**. The budget is **0 (off)** by default. Needs the AI integration (`collectors.aiobs.enabled`).

- **Means.** Failed agent runs (above `failurePct`, 5 percent after `minCalls`, 20), rate limit hits (`rateLimits`, 3 in 10 minutes), MCP errors (`mcpErrors`, 4 in 10 minutes), an agent whose p95 is `slowFactor` (3) times its median, or a spend over the budget.
- **Check first.** **AI** (the observability page): the failing run, its steps and its error. Provider status and your quota. The price table (`collectors.aiobs.prices`) must be filled for the cost to show.
- **Tune.** Set the daily budget in dollars. Change the `collectors.aiobs.*` limits in Settings (live).

### Security

*Critical, security.* **5 failed console logins** (or lockouts) in **5 minutes**, or **any change of settings** in that window.

- **Means.** Someone is guessing a password, or a script is. Or somebody changed a setting: the rule counts the audit events `settings.change`, `settings.reset`, `integration.change`, `ai.mcp.change` and `bar.layout`, and `login.fail` and `login.locked`.
- **Check first.** The audit log, `bxlens-audit.log`, on the **Logs** page of the server that raised it: the `ip` and `role` of each line. Is the address inside your network? Is `console.access` as tight as it should be? A change of settings by a colleague is expected: look for the admin and the keys.
- **Tune.** Raise the failure count for a shared address (many people behind one proxy). If every settings change is noise, switch the rule off and read the audit log on your own schedule. Password changes (`password.*`) are in the audit log but do not raise this alert. The counts start at zero when the server starts.

### Errors in the logs

*Warning, logs.* **10 ERROR lines a minute** or more in the BoxLang log files over **5 minutes**.

- **Means.** The application or BoxLang itself logs many errors, even when no request fails.
- **Check first.** **Logs**: filter by level ERROR in the file that grows. BxLens counts ERROR lines in the `.log` files of the logs directory (up to 8 files), from when it started.
- **Tune.** Raise the number for an application that logs errors it handles.

## Test the setup

- **Send test mail** and **Send test** on a channel prove the path (SMTP, secrets, network).
- **Settings, Notifications, Mail previews** shows the exact mail of each rule with sample numbers, and *Send me this* sends it to one address you type.
- Pause the notifications before a drill. A real alert uses the same path, so a drill that raises Security (try a wrong password five times) really sends.