Alert runbook

Alert runbook

BxLens has 15 alert rules (BoxLang+). They are off until an admin switches them on, one by one, in Settings, Notifications. This page is what to read when one fires: what it means, the default limits, what to check first and how to tune it. How alerts are sent, who gets them and how they repeat is in Notifications.

How to read an alert

• An alert has a rule, a server and a subject (a volume, an executor, a task, a host). The same problem on the same subject is one alert. A mail says firing once, still firing (a reminder) every repeat time, and resolved when it stops.
• The limits below are the defaults. Every rule has its own switch, severity (warning or critical) and repeat time (60 minutes). The limits are set per rule, for the whole cluster.
• Rules that need a window (for example "above 90 percent for 5 minutes") fire only when BxLens has samples for the whole window. After a restart a window rule is quiet until it has looked long enough. BxLens looks every 30 seconds and keeps two hours of samples.
• Mails and messages for a server come from the one server that was elected to send. See Several servers.

How to tune or silence

There is no mute for a single server or a single subject. To keep one server out of a rule, fix the thing it reports, or raise the limit for all.

The rules

Server down

Critical, availability. A server that wrote before has stopped saving its health snapshot and heartbeat. Limit: 3 minutes (1 to 1440).

• Means. The process is down, the host is unreachable, or the server can no longer write to the store volume. A server that cannot write looks exactly like a dead one.
• Check first. Is the process running? Settings, Storage on a working server: the server's last write. Is the store volume full or read only for that host? A server that stopped more than a day ago is called gone and raises no alert.
• Tune. Raise the minutes for servers that restart slowly. A planned shutdown still looks like down: pause for the maintenance.

Deadlock

Critical, jvm. The JVM reports threads that wait for each other and will never continue. No limit.

• Means. Requests on those threads hang until the server restarts.
• Check first. Threads (BoxLang+): the red banner lists each thread, the lock it waits for and who holds it. Take a thread dump and read the BoxLang frames. Ask Lensy about it.
• Tune. Nothing to tune: a deadlock is never normal. Fix the code, restart to recover.

Heap pressure

Critical, jvm. The heap stays above 90 percent for 5 minutes, or the garbage collector uses more than 30 percent of the time (checked over the same window, when the heap is below the limit).

• Means. The server is close to an out of memory error, or it spends its time collecting garbage instead of serving.
• Check first. System: heap, memory pools, GC time. Caches, Request history and big sessions are common causes. A heap dump (BoxLang+, off until console.allowHeapDump) shows what holds the memory, but it holds every secret: read Heap dumps.
• Tune. Raise the percent only if the heap is sized to run hot. The real fix is more heap or less retention.

Disk space

Critical, capacity. A volume is 90 percent used or more (50 to 100), or a folder BxLens must write to (the store, the secrets, the logs) has a problem such as a read only mount.

• Means. Logs, the store or a heap dump can no longer be written. The store drops records and counts them.
• Check first. System, Volumes card: the fullest volume and the folders BxLens writes to. Then the store size in Settings, Storage against store.maxMB, the logs directory, and temporary heap dumps.
• Tune. The limit is per rule. The amber and red colours on the System page are the live settings volumes.warnPct (80) and volumes.critPct (90), a separate pair.

Error spike

Critical, errors. 20 errors a minute or more, measured over 5 minutes, or a new error group: one first seen in the last 10 minutes and after this server started (on by default).

• Means. Something broke now, or after a deploy. The new error flag catches a single new kind of failure even at a low rate.
• Check first. Errors: sort by last seen, open the newest group, read the first sample and its request. Compare with the time of the last deploy. Reports, Boards: Highest error rate.
• Tune. Raise Errors per minute for a noisy site. Set the new group flag to 0 if new groups are frequent and not important.

Failed requests

Critical, traffic. 5 percent of the requests failed (a 5xx or an uncaught error) over 5 minutes, with at least 50 requests.

• Means. Users see errors. The minimum number of requests keeps a quiet server from alerting on one failure.
• Check first. Reports: status codes and the URLs with the most errors. Errors for the cause. Datasources and Outgoing HTTP for a dependency that fell over.
• Tune. On a small site lower the minimum requests, or lengthen the window.

Slow site

Warning, traffic. The average response time is 2000 ms or more over 10 minutes, with at least 50 requests.

• Means. The whole site is slow, not one URL. It is a mean, so a few very slow requests can lift it.
• Check first. Reports: p95 and the slowest URLs. In flight (BoxLang+) shows what is running now. Queries: the heaviest statement. Datasources: waiting threads. System: CPU and GC.
• Tune. Set the average to what is normal for your slowest healthy site.

Datasource pool

Critical, database. Requests have waited for a database connection for 2 minutes (1 to 120): at every look in the window, at least one thread waited for a connection.

• Means. The pool is used up. Requests queue, then time out.
• Check first. Datasources: active against maximum, and the waits. In flight for requests holding connections for long. Queries: a slow statement that holds connections. Check for connections that are never closed.
• Tune. Lengthen the minutes if short bursts are normal. Raise the pool size in the datasource configuration only after you know why it filled.

Executor saturated

Warning, runtime. An executor has 100 or more tasks queued, or its health is critical.

• Means. Work is waiting for threads. Scheduled tasks and async work run late.
• Check first. Executors: the pool, queue and rejected tasks of the one named in the alert. Tasks bound to that executor.
• Tune. Raise the queue limit for executors that queue by design.

Scheduled task failing

Warning, tasks. A scheduled task's last run failed and its failure count is 2 or more.

• Means. A job that should run is failing. The Tasks page marks the status failing.
• Check first. Tasks: open the task, read the last error and the history. Run it once with Run now (BoxLang+, admin, console.actions).
• Tune. Raise the number if a flaky task is retried by design. The status is the last run of the task: it is failing until the next run succeeds. The rule's label says "in a row", but the rule compares the total failure count the Tasks page shows.

Outgoing HTTP

Warning, dependencies. A host or an endpoint the app calls needs attention on the HTTP page. No limit of its own: it uses the limits of the HTTP collector.

• Means. A host is slow (p95 above collectors.http.slowMs, 2000 ms), fails more than errorPct (5) percent once it has minCalls (20) calls, or a request repeats the same call repeat (20) times, or a response is larger than largeMB (5). Notes (severity info) never raise an alert.
• Check first. HTTP: the what needs attention list, then the Calls tab (BoxLang+) for the request that made the call. Is the host itself down?
• Tune. Change the collectors.http.* limits in Settings (live). They also change the HTTP page.

Mail

Warning, mail. The Mail page has a problem: bounces, a high failure rate, or a spool that does not drain. Needs the Mail integration (collectors.mail.enabled).

• Means. Failure rate above collectors.mail.failurePct (5 percent) after minMessages (10) messages, a bounce, a server with a high failure rate, or a message that waits in the spool longer than spoolMinutes (10).
• Check first. Mail: Overview, then Queue (bounced messages with the error) and Servers (which SMTP server fails). An admin can send a bounced message again or discard it (BoxLang+).
• Tune. Change collectors.mail.* in Settings (live).

AI

Warning, ai. An agent or model needs attention on the AI page, or the cost of the last 24 hours is above your daily budget. The budget is 0 (off) by default. Needs the AI integration (collectors.aiobs.enabled).

• Means. Failed agent runs (above failurePct, 5 percent after minCalls, 20), rate limit hits (rateLimits, 3 in 10 minutes), MCP errors (mcpErrors, 4 in 10 minutes), an agent whose p95 is slowFactor (3) times its median, or a spend over the budget.
• Check first. AI (the observability page): the failing run, its steps and its error. Provider status and your quota. The price table (collectors.aiobs.prices) must be filled for the cost to show.
• Tune. Set the daily budget in dollars. Change the collectors.aiobs.* limits in Settings (live).

Security

Critical, security. 5 failed console logins (or lockouts) in 5 minutes, or any change of settings in that window.

• Means. Someone is guessing a password, or a script is. Or somebody changed a setting: the rule counts the audit events settings.change, settings.reset, integration.change, ai.mcp.change and bar.layout, and login.fail and login.locked.
• Check first. The audit log, bxlens-audit.log, on the Logs page of the server that raised it: the ip and role of each line. Is the address inside your network? Is console.access as tight as it should be? A change of settings by a colleague is expected: look for the admin and the keys.
• Tune. Raise the failure count for a shared address (many people behind one proxy). If every settings change is noise, switch the rule off and read the audit log on your own schedule. Password changes (password.*) are in the audit log but do not raise this alert. The counts start at zero when the server starts.

Errors in the logs

Warning, logs. 10 ERROR lines a minute or more in the BoxLang log files over 5 minutes.

• Means. The application or BoxLang itself logs many errors, even when no request fails.
• Check first. Logs: filter by level ERROR in the file that grows. BxLens counts ERROR lines in the .log files of the logs directory (up to 8 files), from when it started.
• Tune. Raise the number for an application that logs errors it handles.

Test the setup

• Send test mail and Send test on a channel prove the path (SMTP, secrets, network).
• Settings, Notifications, Mail previews shows the exact mail of each rule with sample numbers, and Send me this sends it to one address you type.
• Pause the notifications before a drill. A real alert uses the same path, so a drill that raises Security (try a wrong password five times) really sends.