uptop/docs/clustering.md

Clustering

uptop supports three deployment modes for different reliability and coverage needs.

Single node (default)

Out of the box, uptop runs as a standalone leader. One process, one database, runs all checks. No clustering config needed.

Leader + follower (HA failover)

A follower is a standby replica that takes over if the leader goes down.

How it works:

• The follower polls the leader's /api/health endpoint every 5 seconds
• After 3 consecutive failures (15 seconds), the follower promotes itself and starts running checks
• When the leader recovers, the follower detects it and goes back to standby
• Both nodes have their own database — they do not share state

Limitations:

• During a network partition where both nodes are healthy, both will run checks and fire alerts independently. There is no leader fencing — the follower has no way to confirm the leader is actually down vs. unreachable from its perspective. This window lasts until the partition heals, at which point the follower detects the leader and steps down.
• Expect duplicate alerts and doubled check history entries during a split-brain event. Alerts are idempotent for most providers (a second "site is down" notification is noisy but not harmful).
• Failover takeover time is ~15 seconds (3 missed polls × 5 second interval). This is not configurable.

Required env vars:

Node	Variable	Value
Both	UPTOP_CLUSTER_SECRET	Same shared secret
Follower	UPTOP_CLUSTER_MODE	follower
Follower	UPTOP_PEER_URL	Leader's HTTP URL (e.g. http://leader:8080)

See deploy/docker-compose.cluster.yml for a working example.

Leader + probes (distributed monitoring)

Probes are lightweight, stateless nodes that run checks from different locations and report results back to the leader.

How it works:

• A probe registers with the leader on startup
• Every 30 seconds, it fetches check assignments filtered by its region
• It runs the assigned checks (up to 10 concurrent) and posts results back
• The leader aggregates results from all probes and triggers alerts based on the aggregation strategy
• Probes have no database, no UI, and no configuration of their own

Required env vars:

Node	Variable	Value
Both	UPTOP_CLUSTER_SECRET	Same shared secret
Leader	UPTOP_AGG_STRATEGY	any-down, majority-down, or all-down
Probe	UPTOP_CLUSTER_MODE	probe
Probe	UPTOP_PEER_URL	Leader's HTTP URL
Probe	UPTOP_NODE_ID	Unique identifier (e.g. probe-us-east)
Probe	UPTOP_NODE_REGION	Region tag matching monitor assignments

Optional: UPTOP_NODE_NAME for a human-readable label in the TUI.

See deploy/docker-compose.probe.yml for a multi-region example.

Aggregation strategies

When multiple probes check the same monitor, the leader combines their results:

Strategy	Behavior
any-down (default)	DOWN if any probe reports down
majority-down	DOWN if most probes report down
all-down	DOWN only if all probes report down

Set via UPTOP_AGG_STRATEGY on the leader.

Follower vs probe

	Follower	Probe
Purpose	Failover / redundancy	Distributed checks from multiple regions
Database	Own database (independent)	None (stateless)
Runs checks	Only when leader is down	Always, on assigned monitors
Scales to	1 follower per leader	Many probes per leader

Security

• Set UPTOP_CLUSTER_SECRET on all nodes. Without it, cluster API endpoints are unauthenticated.
• Secrets are sent in HTTP headers (X-Uptop-Secret). Use TLS or a reverse proxy for production.
• uptop warns on startup if the cluster secret is missing or if cluster mode is active without TLS.