Version: v5

Clustering

Operations API for managing Harper's replication system. For an overview of how replication works, see Replication Overview. For sharding configuration, see Sharding.

All clustering operations require super_user role.

Add Node

Adds a new Harper instance to the cluster. If subscriptions are provided, it creates the specified replication relationships between the nodes. Without subscriptions or sendsTo / receivesFrom, a fully replicating system is created (all data in all databases). sendsTo / receivesFrom instead scope controlled-flow replication to the databases they name.

Parameters:

operation (required) — must be add_node
hostname or url (required) — the hostname or URL of the node to add
verify_tls (optional) — whether to verify the TLS certificate. Set to false temporarily on fresh installs with self-signed certificates. Defaults to true
authorization (optional) — credentials for the node being added. Either an object with username and password, or an HTTP Authorization style string
retain_authorization (optional) — if true, stores credentials and uses them on every reconnect. Generally not recommended; prefer certificate-based authentication. Defaults to false
revoked_certificates (optional) — array of revoked certificate serial numbers that will not be accepted for any connections
shard (optional) — shard number for this node. Only needed when using sharding
start_time (optional) — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database
subscriptions (optional) — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with:
- database — database name
- table — table name
- subscribe — if true, transactions on the remote table are replicated locally
- publish — if true, transactions on the local table are replicated to the remote node
sendsTo / receivesFrom (optional) Added in: v5.1.0 Changed in: v5.2.0 — database-scoped controlled-flow entries for this node (see Controlling Replication Flow). Each entry is an object { database?, excludeTables? }, or (less usefully) a bare string naming a peer, intended to authorize all databases for that peer rather than match a database name. An entry can also carry a target (on sendsTo) or source (on receivesFrom) naming the one peer it applies to; an entry with neither matches any peer. Only the peer-agnostic object form — { database?, excludeTables? }, with no target/source — is reliable today. Harper's reciprocal add_node_back registration to the added peer carries the array over without rewriting each entry's peer reference for the new direction, so a bare string or a target/source value that's correct for your side of the connection ends up wrong on the peer's copy, and the peer's send-authority check silently rejects the subscription — replication doesn't happen (harper-pro#710 - add_node_back does not rewrite peer-qualified sendsTo/receivesFrom entries). Use config routes for anything that needs to be scoped to one peer. Unlike a config route's replicates.sendsTo / replicates.receivesFrom, which describes the local node's own direction, these describe the added node's perspective: sendsTo lists what the added node sends (what this node receives from it), and receivesFrom lists what the added node receives (what this node sends to it). If both subscriptions and sendsTo/receivesFrom are provided, subscriptions takes precedence and sendsTo/receivesFrom are ignored. To replicate all databases while excluding specific tables, use a wildcard entry (excludeTables with no database).

Note: Because these entries aren't scoped to one peer, the resulting hdb_nodes record for the added node isn't restricted to the connection that created it — any other node that also holds the listed database(s) can match against it too. This also doesn't change how the local node advertises itself to the rest of the cluster: a node's own directional (non-mesh) hdb_nodes self-record is derived solely from its harper-config.yaml routes, so replicating system without collapsing to a full mesh requires directional routes in config, not just add_node/set_node scoping (see Replicating the system database with controlled flow). Nor is the scoping durable: each node rewrites its own hdb_nodes row from its harper-config.yaml routes when it restarts or reloads components, and that row replicates, so directional entries an add_node call wrote onto the added node's row are superseded — a node with no directional routes goes back to advertising the legacy full-mesh record. Use config routes for any constraint that has to survive a restart or a deploy_component reload.

Changed in v5.2: these parameters have been accepted since v5.1.0, but they only gate replication as described above from v5.2 on. Before v5.2, add_node wrote a blanket "sends everything" flag onto the added node's registry record alongside the entries, which authorized this node to receive every database from that node and left sendsTo's database scoping inert; in the other direction, the send-authority check honored a receivesFrom entry only when it named both this node and the database, so the peer-agnostic object form recommended above did not authorize sending. As of v5.2 the blanket flag is gone — replication from the added node is limited to the databases its sendsTo entries name — and an entry that omits the peer or the database is treated as a wildcard by both checks. If you already call add_node / set_node with sendsTo / receivesFrom, re-check that scoping before upgrading — it can change in either direction. Databases that were replicating only because of the blanket flag stop replicating on v5.2; conversely, a peer-agnostic receivesFrom entry that v5.1's send check ignored becomes an active wildcard on v5.2, so it newly authorizes any peer that subscribes for the database it names.

Request:

{
	"operation": "add_node",
	"hostname": "server-two",
	"verify_tls": false,
	"authorization": {
		"username": "admin",
		"password": "password"
	}
}

Request (controlled flow, scoped per database):

{
	"operation": "add_node",
	"hostname": "server-two",
	"sendsTo": [{ "database": "cardata", "excludeTables": ["raw_telemetry"] }],
	"receivesFrom": [{ "database": "config" }, { "database": "system" }]
}

Read from the added node's perspective: server-two sends cardata (except raw_telemetry) to this node, and receives config and system from it.

Response:

{
	"message": "Successfully added 'server-two' to cluster"
}

Note: set_node is an alias for add_node.

Update Node

Modifies an existing Harper instance in the cluster. Will attempt to add the node if it does not exist.

Parameters:

operation (required) — must be update_node
hostname (required) — hostname of the remote node to update
revoked_certificates (optional) — array of revoked certificate serial numbers
shard (optional) — shard number to assign to this node
subscriptions (required) — array of subscription objects (same structure as add_node)

Request:

{
	"operation": "update_node",
	"hostname": "server-two"
}

Response:

{
	"message": "Successfully updated 'server-two'"
}

Remove Node

Removes a Harper node from the cluster and stops all replication to and from that node.

Parameters:

operation (required) — must be remove_node
hostname (required) — hostname of the node to remove

Request:

{
	"operation": "remove_node",
	"hostname": "server-two"
}

Response:

{
	"message": "Successfully removed 'server-two' from cluster"
}

Cluster Status

Returns an array of status objects from the cluster, including active WebSocket connections and replication timing statistics.

Added in: v4.4.0 ; timing statistics added in v4.5.0

Parameters:

operation (required) — must be cluster_status

Request:

{
	"operation": "cluster_status"
}

Response:

{
	"type": "cluster-status",
	"connections": [
		{
			"replicateByDefault": true,
			"replicates": true,
			"url": "wss://server-2.domain.com:9933",
			"name": "server-2.domain.com",
			"subscriptions": null,
			"database_sockets": [
				{
					"database": "data",
					"connected": true,
					"latency": 0.7,
					"threadId": 1,
					"nodes": ["server-2.domain.com"],
					"lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT",
					"lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT",
					"lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT",
					"sendingMessage": "Wed, 12 Feb 2025 16:50:59 GMT"
				}
			]
		}
	],
	"node_name": "server-1.domain.com",
	"is_enabled": true
}

database_sockets shows the actual WebSocket connections between nodes — one socket per database per node. Timing fields:

Field	Description
`lastCommitConfirmed`	Last time a receipt of confirmation was received for an outgoing commit
`lastReceivedRemoteTime`	Timestamp (from the originating node) of the last received transaction
`lastReceivedLocalTime`	Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up
`sendingMessage`	Timestamp of the transaction actively being sent. Absent when waiting for the next transaction

While a database is taking its initial full copy, these timing fields hold the literal string "Copying" rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching.

Configure Cluster

Bulk creates or resets subscriptions for any number of remote nodes. Resets and replaces any existing clustering setup.

Parameters:

operation (required) — must be configure_cluster
connections (required) — array of node objects following the add_node schema

Request:

{
	"operation": "configure_cluster",
	"connections": [
		{
			"hostname": "server-two",
			"verify_tls": false,
			"authorization": {
				"username": "admin",
				"password": "password2"
			}
		},
		{
			"hostname": "server-three",
			"verify_tls": false,
			"authorization": {
				"username": "admin",
				"password": "password3"
			}
		}
	]
}

Response:

{
	"message": "Cluster successfully configured."
}

Cluster Set Routes

Adds routes to the replication.routes configuration. Behaves as a PATCH/upsert — adds new routes while leaving existing routes untouched.

Parameters:

operation (required) — must be cluster_set_routes
routes (required) — array of route strings (wss://host:port) or objects with hostname and port properties

Request:

{
	"operation": "cluster_set_routes",
	"routes": [
		"wss://server-two:9925",
		{
			"hostname": "server-three",
			"port": 9930
		}
	]
}

Response:

{
	"message": "cluster routes successfully set",
	"set": ["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }],
	"skipped": []
}

Cluster Get Routes

Returns the replication routes from the Harper config file.

Parameters:

operation (required) — must be cluster_get_routes

Request:

{
	"operation": "cluster_get_routes"
}

Response:

["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }]

Cluster Delete Routes

Removes routes from the Harper config file.

Parameters:

operation (required) — must be cluster_delete_routes
routes (required) — array of route objects to remove

Request:

{
	"operation": "cluster_delete_routes",
	"routes": [
		{
			"hostname": "server-three",
			"port": 9930
		}
	]
}

Response:

{
	"message": "cluster routes successfully deleted",
	"deleted": [{ "hostname": "server-three", "port": 9930 }],
	"skipped": []
}

Add Node​

Update Node​

Remove Node​

Cluster Status​

Configure Cluster​

Cluster Set Routes​

Cluster Get Routes​

Cluster Delete Routes​

Add Node

Update Node

Remove Node

Cluster Status

Configure Cluster

Cluster Set Routes

Cluster Get Routes

Cluster Delete Routes