Several servers that stay in sync

Several servers that stay in sync

Servers that serve one application can share their console settings and their secrets, so a change made on one server reaches the others. This page is the setup. For the fleet view itself see Several servers.

What is shared

Nothing else is shared. Settings that are not live (console.access, access.*, store.*, secrets.dir, sync.*) stay in the boxlang.json of each server, so keep them the same with your deployment tooling.

Set it up

1. Mount one volume on every server, for example /shared/bxlens. It must allow a file lock. A network volume that cannot lock still works, only two saves in the same instant could lose one.

2. Make one secrets key and give the same value to every server. It is base64 of 32 random bytes:

   openssl rand -base64 32
   

   Keep it in your secret manager and set it as the environment variable LENS_SECRETS_KEY on every server. It replaces secrets.key, which keeps the key off the volume. A lost key means entering the secrets again.

3. Use one BoxLang secret seed if your boxlang.json holds bxsecret: values, so every server can open them. See Secrets.

4. Give every server its own name (server.name, or LENS_INSTANCE), so the fleet and the store can tell them apart.

5. Point every server at the shared paths in boxlang.json:

   "modules": { "bxlens": { "settings": {
   	"console": { "overridesFile": "/shared/bxlens/bxlens-settings.json" },
   	"secrets": { "dir": "/shared/bxlens/secrets" },
   	"store": { "dir": "/shared/bxlens/store" },
   	"sync": { "enabled": true, "pollSeconds": 5 }
   } } }
   

6. Restart the servers. These settings are read at startup.

Alerts as code

The notification settings (SMTP, recipients, channels, alert rules, report schedule) can be written in boxlang.json once and deployed to every server. See Seed the notification settings. The secrets of the channels come from bxsecret: values or from environment variables that start with LENS_CHANNEL_, and are put in the shared Secret Store. After that, changes made in the console are saved in the shared settings file and reach the other servers: a server reads the notification settings again whenever it needs them, so this works without the watcher.

Everything else in the console

Besides the notification settings, boxlang.json can seed the Lensy MCP servers (mcp.servers), the bar layout (bar.layout) and the ORM statistics switch (collectors.orm.statistics). Each follows the same deploy rule, and servers that share the settings file share what the console changes afterwards. See MCP servers and Bar layout.

The watcher

With sync.enabled each server compares the modified time and the size of the settings file every pollSeconds. When another server saved a change, this one reads the file and applies it: collectors, thresholds, the bar, the AI settings and the rest of the live settings, and it reads the MCP server list again. The bar layout and the notification settings are read from the file whenever they are needed. It does not wait for events of the file system, because those are not reliable on a network volume. It is off by default. Without it a server reads the file at startup only, and you restart the others after a change.

The secrets folder needs no watcher: the Secret Store reads its file again whenever it changed.

A deploy wins over a console change

boxlang.json is the seed, and the console is for changes while the server runs.

• BxLens saves a hash of each boxlang.json value as it was last applied, in the settings file.
• A restart with the same boxlang.json keeps every change saved in the console.
• A deploy that changes a value drops the saved change for that one setting, and the boxlang.json value applies. The other saved changes stay.
• The first start that has this rule records the hashes and drops nothing.
• Servers that share the file should get the same boxlang.json. During a rolling deploy the first server that restarts replaces the saved changes, and the others see the new value when they restart too.

The same rule holds for each section of notifications (general, smtp, recipients, channels, rules, reports): a deploy that changes a section replaces what was saved for it, and a section that boxlang.json does not have is never touched. A module upgrade that changes a default counts as a deploy for that setting.

Check it

1. Open Settings on server A and change a setting, for example the slow query limit.
2. Within pollSeconds open Settings on server B: the setting shows changed with the same value. The audit log of B has a settings.sync line.
3. Press Reset on B and see it return on A.

If it does not work

• B does not change: check that both servers show the same path for the settings file (Settings, Security shows console.overridesFile) and that sync.enabled is on for B.
• A setting snaps back after a restart: the boxlang.json value changed in your deploy, so it replaced the saved change. Change it in boxlang.json, or change it in the console after the deploy.
• The setup page appears on one server: that server cannot open the secrets. Check LENS_SECRETS_KEY, secrets.dir and the secret seed of that server.
• Skipped entries: a hand edited file with a value the console may not set is ignored for that entry and a warning is logged at startup.