---
title: Several servers
order: 4
description: Set up BxLens for more than one server with one shared store folder and one shared secrets folder, how the sender of alerts is elected, what happens when it fails, server names and the cautions for NFS and SMB volumes. BoxLang+.
icon: lucide:server-cog
---

# Several servers

BxLens has no agent and no central server. Servers that share one folder find each other in it. Each one writes only its own subfolder, and with BoxLang+ each can read what the others saved. This page shows how to set that up and what to expect.

!!! note "BoxLang+"
    Many servers in one console is a BoxLang+ feature. Free shows one server only. The shared store itself is the BoxLang+ disk store.

## What you need

1. BoxLang+ (or a trial) on every server, and BxLens switched on (`console.enabled`).
2. One **store folder** that all servers can write: a network volume, or a mounted disk in a cluster. This is `store.dir`.
3. One **secrets folder** that all servers can write, if the servers must read the same secrets (the SMTP password, channel addresses, console passwords set in the console). This is `secrets.dir`. It can be the same volume as the store, in another folder.
4. The **same key** for those secrets on every server.
5. A **different, stable name** for each server.

## Set it up

The same `boxlang.json` on every server, except the name:

```json title="boxlang.json on web-a"
{
	"modules": {
		"bxLens": {
			"settings": {
				"console": { "enabled": true, "password": "${env.BXLENS_PASSWORD}", "requireHttps": true, "access": [ "10.8.0.0/24" ] },
				"store": { "dir": "/mnt/shared/lens-store", "instanceFolder": "web-a" },
				"secrets": { "dir": "/mnt/shared/lens-secrets" },
				"server": { "name": "web-a" }
			}
		}
	}
}
```

On `web-b` only `instanceFolder` and `server.name` change. Make the key once and give the same value to every server from your platform's secret store, so the key does not sit on the shared volume:

```bash
openssl rand -base64 32
# set the output as LENS_SECRETS_KEY on every server
```

If you leave `LENS_SECRETS_KEY` out, the first server that starts creates `secrets.key` in `secrets.dir`, and the others use it. That is simpler, and it puts the key on the same volume as the secrets. Read [What it protects, and what it does not](../security.md#the-secret-store) before you choose.

Start the servers. Open the console on any of them (Plus):

- **Settings, Storage** lists the servers that write in the store folder, with their last write.
- The header has a **Server** selector, and the Overview has a **Servers** table. See [The server selector](../console/servers.md).
- **Settings, Notifications** says which server sends the alerts.

## Names

Four names look alike. They are different.

| Name | Setting | What it is for |
|---|---|---|
| Store folder name | `store.instanceFolder`, then `LENS_INSTANCE`, then the host name | The subfolder this server writes. It is also the **id** you see in the selector, the Servers table, the audit log and the alerts, and it decides which server sends alerts. Keep it stable and unique. Letters, digits, dot, underscore and dash, 64 at most. |
| Server name | `server.name`, then `LENS_SERVER_NAME`, then the host name | The name in the records of the server. On Free and Trial it is also the name a registration key is tied to. |
| Server address | `server.address`, then `LENS_SERVER_ADDRESS` | The primary address, for display. |
| Instance id | `server.id`, then `LENS_SERVER_ID`, then a short id made from the host name and the start time of the JVM | The id of one run in the records. It changes at a restart unless you set it. |

In a container the host name changes with every deploy. Without a stable `store.instanceFolder` or `LENS_INSTANCE`, each deploy leaves a new folder behind. In Kubernetes use a StatefulSet (stable pod names), or set `LENS_INSTANCE` yourself.

Two live servers must not share one folder name. The second one finds the first one's heartbeat, logs a warning once and **saves nothing** until the first stops. The Storage tab says: "Another live server writes to the folder ... This server saves nothing until it stops."

## Who sends the alerts

Every server works out its own alerts and lists them in the health snapshot it saves every minute. **One** server sends the mails and the channel messages for all of them:

- It is the server with the **lowest id** (the store folder name, compared as text) among the servers that wrote in the last **2 minutes 30 seconds** and have the notifications (BoxLang+). So `web-a` sends before `web-b`, and `Web-A` (capital) sorts before `web-a`.
- All servers read the same folder and get the same answer, so nobody has to ask.
- A server without BoxLang+ lists no alerts and never sends.
- The page in Settings, Notifications names the sender.

What follows from that:

- **The sender needs to reach your SMTP server** and your channels' hosts. If only some servers can, name them so that one of them has the lowest id.
- **The sender needs to open the secrets** (the SMTP password, the channel addresses and tokens). That is why `secrets.dir` and the key are shared. A server that cannot open them cannot send, and the Notifications page shows the error. The Storage tab, Secrets card, says "This server can open them" yes or no.
- The settings of the notifications (recipients, rules, channels without their secrets) are saved in the store. The newest copy wins, so a change on one server reaches the others.

## What happens at a failover

- The sender stops writing (a crash, a deploy, a network cut). After 2 minutes 30 seconds the others no longer count it as alive, and the one with the next lowest id becomes the sender.
- The new sender loads the **saved state of the alerts** (`alert-state`, kept per person and per channel) and goes on from it. It does not mail again what the old one already told people. Without the store, what is still true would be announced once more.
- The new sender sees the old one as a server that stopped writing. The **Server down** alert (if you switched it on) fires when it has been silent for its limit (3 minutes by default). So after a sender crash the mail about it comes a few minutes late, from the new sender.
- A server that stopped writing more than a day ago is called gone, not down, and no alert is raised for it. The folder is removed when the server has been silent for `store.staleAfterHours` (the retention when `0`), by any server.
- A clean shutdown releases the folder at once. A hard kill needs up to 90 seconds before the same folder name can be taken over by another host.
- If the sender cannot send to one person, only that person is tried again, after five minutes. The others are not mailed twice.

## What each server can and cannot see

| | This server | Another server (BoxLang+) |
|---|---|---|
| Requests, In flight, System, Executors, Tasks, Datasources, Applications, Watchers, Caches | Live, from memory | The health snapshot it saved, up to about a minute old, read only |
| Queries, ORM | Live | Its saved snapshot, read only |
| HTTP, Reports, Errors | Live plus saved history | Its saved rollups and error groups. **All servers** adds them up with a real merged p95 |
| Mail | Live | Its overview, servers, the counts of its queue and settings. No message: addresses, subjects and errors stay on the server that holds them |
| Threads, Logs, Environment, Configuration, Settings | Live | **Not copied.** Open the console of that server |
| Actions (run a task, clear a cache, restart, settings) | Yes | **Never.** The page is read only and the server refuses the call |
| Alerts | Its own | The sender reads every server's list |

Settings are **per server**. A change on the Settings page is saved in that server's `config/bxlens-settings.json`, not in the shared folder, so make the same change on each server, or set it in `boxlang.json`. The notification settings are the exception: they are in the store. The console passwords set in the console are shared when `secrets.dir` and the key are shared. A password in `boxlang.json` is per server.

BxLens never connects to another server. It reads files.

## Network volumes: NFS, SMB and cloud file systems

The store was built for a shared volume, and it avoids what network volumes do badly. Know these before you pick one.

- **No locks in the store.** Each server writes only its own folder, with a temporary file and an atomic rename. Liveness comes from a heartbeat file (`instance.json`, every 30 seconds) judged by what changes on the reader's own clock, not by comparing clocks. A lease that does not change for 90 seconds is dead. Clock differences between servers do not matter, apart from the very first look, which tolerates 5 minutes.
- **The secrets folder does use a file lock.** A change of a secret takes a lock (`secrets.lock`, up to 10 seconds of waiting) so two servers do not overwrite each other. Reading needs no lock. Use a volume on which file locks work, or change secrets from one server at a time. We have not tested every NFS or SMB setup.
- **Files may appear late.** A server reads what the others wrote. A volume that caches directory listings can show a file late. The pages say when the data was saved, and nothing breaks if it is a minute behind.
- **Case.** Windows and macOS compare folder names without regard to case. `Web-A` and `web-a` are one folder there. Keep names in lower case across servers.
- **No symbolic links.** BxLens never follows a link inside the store or in `secrets.dir`, and ignores a server folder that is a link. Mount the volume at the path you set. Do not point `store.dir` at a link.
- **Permissions.** The user that runs BoxLang must write to the folder. A full disk or a read only volume is counted (`failed`, `dropped` on the Storage tab) and never blocks a request.
- **Keep the volume near the servers.** A slow volume slows the writer thread, never a request. The queue is bounded (`store.queueSize`), and records are dropped and counted when it is full.
- **Who may read it.** The files hold redacted data, but they are still data from your servers (queue entries of the mail spool hold addresses and subjects). Give the folder the permissions you give to your logs.

## Check that it works

1. The Storage tab lists every server, each with a recent last write.
2. The selector shows every server as `saved` (wrote in the last 2 minutes). `behind` is up to a day. `stale` is more than a day.
3. Settings, Notifications names one sender. Press **Send test mail** on any server. The test goes through that server's SMTP settings and secrets, so it also proves the key is shared.
4. Stop the sender in a test environment, wait three minutes, and check that the next server takes over and that Server down mentions the first.

If it does not work, see [Troubleshooting](../troubleshooting.md#a-server-is-missing-from-the-selector).