Several servers

Several servers

BxLens has no agent and no central server. Servers that share one folder find each other in it. Each one writes only its own subfolder, and with BoxLang+ each can read what the others saved. This page shows how to set that up and what to expect.

What you need

1. BoxLang+ (or a trial) on every server, and BxLens switched on (console.enabled).
2. One store folder that all servers can write: a network volume, or a mounted disk in a cluster. This is store.dir.
3. One secrets folder that all servers can write, if the servers must read the same secrets (the SMTP password, channel addresses, console passwords set in the console). This is secrets.dir. It can be the same volume as the store, in another folder.
4. The same key for those secrets on every server.
5. A different, stable name for each server.

Set it up

The same boxlang.json on every server, except the name:

{
	"modules": {
		"bxLens": {
			"settings": {
				"console": { "enabled": true, "password": "${env.BXLENS_PASSWORD}", "requireHttps": true, "access": [ "10.8.0.0/24" ] },
				"store": { "dir": "/mnt/shared/lens-store", "instanceFolder": "web-a" },
				"secrets": { "dir": "/mnt/shared/lens-secrets" },
				"server": { "name": "web-a" }
			}
		}
	}
}

On web-b only instanceFolder and server.name change. Make the key once and give the same value to every server from your platform's secret store, so the key does not sit on the shared volume:

openssl rand -base64 32
# set the output as LENS_SECRETS_KEY on every server

If you leave LENS_SECRETS_KEY out, the first server that starts creates secrets.key in secrets.dir, and the others use it. That is simpler, and it puts the key on the same volume as the secrets. Read What it protects, and what it does not before you choose.

Start the servers. Open the console on any of them (Plus):

• Settings, Storage lists the servers that write in the store folder, with their last write.
• The header has a Server selector, and the Overview has a Servers table. See The server selector.
• Settings, Notifications says which server sends the alerts.

Names

Four names look alike. They are different.

In a container the host name changes with every deploy. Without a stable store.instanceFolder or LENS_INSTANCE, each deploy leaves a new folder behind. In Kubernetes use a StatefulSet (stable pod names), or set LENS_INSTANCE yourself.

Two live servers must not share one folder name. The second one finds the first one's heartbeat, logs a warning once and saves nothing until the first stops. The Storage tab says: "Another live server writes to the folder ... This server saves nothing until it stops."

Who sends the alerts

Every server works out its own alerts and lists them in the health snapshot it saves every minute. One server sends the mails and the channel messages for all of them:

• It is the server with the lowest id (the store folder name, compared as text) among the servers that wrote in the last 2 minutes 30 seconds and have the notifications (BoxLang+). So web-a sends before web-b, and Web-A (capital) sorts before web-a.
• All servers read the same folder and get the same answer, so nobody has to ask.
• A server without BoxLang+ lists no alerts and never sends.
• The page in Settings, Notifications names the sender.

What follows from that:

• The sender needs to reach your SMTP server and your channels' hosts. If only some servers can, name them so that one of them has the lowest id.
• The sender needs to open the secrets (the SMTP password, the channel addresses and tokens). That is why secrets.dir and the key are shared. A server that cannot open them cannot send, and the Notifications page shows the error. The Storage tab, Secrets card, says "This server can open them" yes or no.
• The settings of the notifications (recipients, rules, channels without their secrets) are saved in the store. The newest copy wins, so a change on one server reaches the others.

What happens at a failover

• The sender stops writing (a crash, a deploy, a network cut). After 2 minutes 30 seconds the others no longer count it as alive, and the one with the next lowest id becomes the sender.
• The new sender loads the saved state of the alerts (alert-state, kept per person and per channel) and goes on from it. It does not mail again what the old one already told people. Without the store, what is still true would be announced once more.
• The new sender sees the old one as a server that stopped writing. The Server down alert (if you switched it on) fires when it has been silent for its limit (3 minutes by default). So after a sender crash the mail about it comes a few minutes late, from the new sender.
• A server that stopped writing more than a day ago is called gone, not down, and no alert is raised for it. The folder is removed when the server has been silent for store.staleAfterHours (the retention when 0), by any server.
• A clean shutdown releases the folder at once. A hard kill needs up to 90 seconds before the same folder name can be taken over by another host.
• If the sender cannot send to one person, only that person is tried again, after five minutes. The others are not mailed twice.

What each server can and cannot see

Settings are per server. A change on the Settings page is saved in that server's config/bxlens-settings.json, not in the shared folder, so make the same change on each server, or set it in boxlang.json. The notification settings are the exception: they are in the store. The console passwords set in the console are shared when secrets.dir and the key are shared. A password in boxlang.json is per server.

BxLens never connects to another server. It reads files.

Network volumes: NFS, SMB and cloud file systems

The store was built for a shared volume, and it avoids what network volumes do badly. Know these before you pick one.

• No locks in the store. Each server writes only its own folder, with a temporary file and an atomic rename. Liveness comes from a heartbeat file (instance.json, every 30 seconds) judged by what changes on the reader's own clock, not by comparing clocks. A lease that does not change for 90 seconds is dead. Clock differences between servers do not matter, apart from the very first look, which tolerates 5 minutes.
• The secrets folder does use a file lock. A change of a secret takes a lock (secrets.lock, up to 10 seconds of waiting) so two servers do not overwrite each other. Reading needs no lock. Use a volume on which file locks work, or change secrets from one server at a time. We have not tested every NFS or SMB setup.
• Files may appear late. A server reads what the others wrote. A volume that caches directory listings can show a file late. The pages say when the data was saved, and nothing breaks if it is a minute behind.
• Case. Windows and macOS compare folder names without regard to case. Web-A and web-a are one folder there. Keep names in lower case across servers.
• No symbolic links. BxLens never follows a link inside the store or in secrets.dir, and ignores a server folder that is a link. Mount the volume at the path you set. Do not point store.dir at a link.
• Permissions. The user that runs BoxLang must write to the folder. A full disk or a read only volume is counted (failed, dropped on the Storage tab) and never blocks a request.
• Keep the volume near the servers. A slow volume slows the writer thread, never a request. The queue is bounded (store.queueSize), and records are dropped and counted when it is full.
• Who may read it. The files hold redacted data, but they are still data from your servers (queue entries of the mail spool hold addresses and subjects). Give the folder the permissions you give to your logs.

Check that it works

1. The Storage tab lists every server, each with a recent last write.
2. The selector shows every server as saved (wrote in the last 2 minutes). behind is up to a day. stale is more than a day.
3. Settings, Notifications names one sender. Press Send test mail on any server. The test goes through that server's SMTP settings and secrets, so it also proves the key is shared.
4. Stop the sender in a test environment, wait three minutes, and check that the next server takes over and that Server down mentions the first.

If it does not work, see Troubleshooting.