Skip to main content

Operating Replication

Replication is how a Harper cluster stays available: peers exchange data directly over WebSockets, there is no primary, and a node that falls behind catches up on its own. The replication reference covers how to configure that. This guide covers the part that only matters once real users depend on it, which is how to tell whether replication is actually working.

The distinction that runs through this guide is between a connection and convergence. A connected socket proves two nodes can talk. It does not prove the data a user is about to read is current. Those are different claims, and only one of them is what your traffic admission decision depends on.

What You Will Learn​

  • How to determine what your cluster actually replicates, rather than assuming a default
  • What moves between peers automatically, what does not, and which of those will surprise you
  • How to read cluster_status as an operator, including which timing field actually indicates a node is behind
  • How to prove convergence with a sentinel rather than inferring it from connection state
  • Which application behaviors replication cannot make safe, and what to do about them instead

Prerequisites​

  • A cluster of at least three nodes, so you can interrupt one peer and still observe the others
  • A super_user credential for the Operations API
  • How Harper Runs in Production and your service boundary inventory
  • Familiarity with how your cluster was joined, either through harper-config.yaml routes or the clustering operations

Know your own scope​

By default Harper replicates all data in all databases. Scope can be narrowed two ways: per database in configuration, and per table in the schema.

replication:
  databases:
    - data
    - system
type LocalTableForNode @table(replicate: false) {
	id: ID! @primaryKey
	name: String!
}

All tables in a replicated database replicate unless the table opts out. So the scope you are operating is the product of a config list, a set of schema directives, and any directional routes someone added later. Do not reconstruct it from memory. Read it off the running cluster:

curl -s -X POST https://my-node.example.com:9925/ \
  -H 'Content-Type: application/json' \
  -u 'admin:password' \
  -d '{"operation":"cluster_status"}'

There is one socket per database per peer, so the sockets present in the response are the ground truth about what is flowing. Run this from every node, not one. Replication direction can be constrained per route, so node A's view of the cluster is not necessarily node B's view, and a one-sided picture is how directional configuration mistakes survive into production.

warning

Whether the system database is in your replication scope is the highest-stakes scoping question in a Harper cluster, because system holds hdb_user, hdb_role, and hdb_nodes. It is in scope by default, since the default replication scope is every database, so unless someone has narrowed it your users and roles already propagate and every node that receives system must be trusted with its contents, including encrypted secret rows. That is usually what you want, but it should be a decision rather than a surprise. Confirm your own answer from cluster_status and your configuration, and read replicating the system database with controlled flow before changing it.

What moves automatically, and what does not​

Object or changeBehaviorWhat it means for you
Data mutationsInsert, update, upsert, delete, and bulk load replicate for tables in scopeMonitor connections, latency, and convergence for the databases that serve critical journeys
TransactionsReplicated atomically, and a transaction may span multiple tablesTables that need to commit together must live in the same database
A brand new nodeRequests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replicationBudget N transfers, not one. See the note below before sizing a join
A returning nodeResynchronizes automatically to catch up on the transactions it missedThis is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately
Component deploymentdeploy_component replicates by default; pass "replicated": false to hold it to one nodeCluster-wide is the default, so single-node validation is the case you must ask for
Configurationset_configuration supports "replicated": true Added in: v5.2.0Never replicate node-local parameters. See the warning below
Users and rolesPropagate when the system database is in replication scope, which is the defaultIf you narrow scope to exclude system, identity must be provisioned on every node by your own automation
Node registry (hdb_nodes)Each node rewrites its own self-record from its harper-config.yaml routes on restart or component reloadScoping applied through add_node alone does not survive a restart. Put durable constraints in config routes
Destructive schema changesdrop_database and drop_table replicate by default; drop_attribute does notA drop is a cluster-wide event unless you pass "replicated": false. Verify the result on every node

The three rows most likely to bite you are the returning node, the new node, and the node registry.

On the returning node: it is common to see a routine restart budgeted as though it were a full resynchronization, which makes teams avoid restarts they should be comfortable with. A node whose databases have never synced does download them in full. A node that was briefly offline catches up on what it missed. Measure both on your own data volume once, and use the right number for the right situation.

On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy from it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source.

The practical consequences are worth planning around. Load lands on every existing peer at once rather than on one, so a join during peak traffic is a capacity event for the whole cluster. If you want a single designated source instead, add_node accepts isLeader: true, which tells the joining node to request its full copy from that peer alone.

On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads components, and the node quietly goes back to advertising itself more broadly than you intended.

danger

When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, node.hostname, a file path, TLS material, or replication.hostname, url, or routes overwrites every peer's own local value. To apply a cluster-wide change safely, use set_configuration with "replicated": true for the parameter, then restart_service with "replicated": true, which restarts nodes one at a time.

Read cluster_status like an operator​

A trimmed response, with the fields that matter:

{
	"type": "cluster-status",
	"node_name": "server-1.example.net",
	"is_enabled": true,
	"connections": [
		{
			"url": "wss://server-2.example.net:9933",
			"name": "server-2.example.net",
			"database_sockets": [
				{
					"database": "data",
					"connected": true,
					"latency": 0.7,
					"lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT",
					"lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT",
					"lastReceivedLocalTime": "Wed, 12 Feb 2025 19:09:31 GMT"
				}
			]
		}
	]
}

What each field is telling you:

  • connected is the liveness of this one database's socket to this one peer. A missing peer, or a peer present with connected: false, is actionable before users notice.
  • latency is the round trip to that peer in milliseconds. Alert on sustained growth rather than on a single sample.
  • lastCommitConfirmed is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer.
  • lastReceivedRemoteTime is the source node's timestamp on the newest transaction you have received.
  • lastReceivedLocalTime is your own clock when you received it.

The last two are the pair that matters. A widening gap between lastReceivedRemoteTime and lastReceivedLocalTime means this node is behind and working through a backlog. That is the signal to alert on for convergence, and it is the one that tells you a returning node is not ready for traffic yet. sendingMessage appears while a transaction is actively being sent and is absent when the socket is idle, so its absence is not a fault.

Two things will mislead you if you automate on that pair without knowing them. The gap only means "behind" while transactions are actually arriving: on an idle source both stamps freeze, and the gap sits at whatever constant it last reached rather than signalling lag. And because the two stamps come from two different clocks, skew between peers is added to the gap directly, so calibrate your threshold against a healthy baseline rather than treating the raw number as replication delay.

tip

While a database is taking a full copy, these timing fields render as the literal string "Copying" rather than a date. Anything that parses them as timestamps will fail on a joining node, which is exactly the node you most want to be watching. Handle that value explicitly.

Inventory system.hdb_nodes alongside this and compare it to your intended topology. Configuration intent and live peer state should agree, and the node's own row is in there too, not just its peers.

What to alert on​

  • A peer missing entirely from connections, or connected: false on a database that serves a critical journey
  • Sustained growth in latency, judged against your own baseline
  • lastCommitConfirmed not advancing on a peer while writes are occurring
  • A lastReceivedRemoteTime to lastReceivedLocalTime gap exceeding your admission budget, evaluated only while writes are flowing
  • Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy
  • Version or configuration drift between peers

Prove convergence, not connection​

Socket state cannot tell you that a specific business record is current. For anything where the answer matters, write a sentinel and read it back from the peer.

// On the source node: write a sentinel with a known value
const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' };
await tables.OpsProbe.put(marker.id, marker);
# On the target node: read it back and compare writtenAt
curl -s https://server-2.example.net:9926/OpsProbe/convergence-probe \
  -u 'admin:password'

The interval between the write and the moment the peer returns the new value is your measured convergence time for that database, under whatever load the cluster is carrying at the time. Run it under load, not on an idle cluster, and record the result. That number is what your node admission gate should be compared against, and it is an input to the RPO work in Engineering RPO, RTO, and Uptime.

The replication-latency metric from analytics gives you the continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard.

Note that replication-latency is not recorded for the system database. Since system is in replication scope by default and carries your users and roles, a dashboard built only on this metric will show nothing for the database that propagates identity. Use the sentinel, or the cluster_status receive-time pair, if you need to watch system convergence.

Consistency belongs in the application design​

Replication makes data available on every peer. It does not make every peer agree at every instant, and a distributed decision made on a node that has not yet converged can observe stale state and act on it.

This is not a Harper limitation to work around, it is the property that lets any node serve any request without a coordinator. But it means a specific class of operation is unsafe if you write it as a plain read followed by a write:

  • Claim-once actions: redeeming a code, assigning a unique handle, awarding a one-per-customer offer
  • Hard floors: inventory that must not go negative, a balance that must not overdraw
  • Global limits: a rate limit or quota enforced across the whole cluster
  • State machine transitions where two nodes could both believe they are making the same transition

For each of these, choose one of three designs: route all decisions for a given key to a single owner, use a serialization mechanism so the conflict is resolved in one place, or delegate to an external coordinator. Then test your read-after-write expectations through the actual public route rather than against a single node, because a single node always looks consistent to itself.

Prove it​

On a non-production cluster under write load:

  1. Record baseline cluster_status timing fields on every node.
  2. Interrupt replication to one peer, by stopping the node or blocking the replication port, while writes continue elsewhere.
  3. Watch what your monitoring reports, and how long it takes to say anything. Note whether connected flipped, whether latency alerted, and how long until a human would have known.
  4. Verify your application's actual behavior on the isolated node. Does the critical journey fail, serve stale data, or serve correctly? All three are possible and you should know which.
  5. Restore the peer, then measure convergence with the sentinel until it is current.
  6. Compare the convergence time against the admission gate in Health Checks and Traffic Admission. If your gate would have admitted the node before step 5 finished, the gate is wrong.

Operational notes​

  • mTLS is required on the secure replication port and cannot be disabled. Certificate expiry is therefore a replication outage, so treat certificate lifetime as an operational deadline with its own alert. See certificate management.
  • Gossip discovery means one route can join a whole cluster. A node that connects to one peer discovers the rest. That is convenient and it means an accidental route can widen your topology more than you intended.
  • Sharding is a separate scope control. If you use sharding, not every node holds every record, so "this node is converged" and "this node can answer this query locally" become different questions.
  • Analytics data has its own replication setting. See analytics.replicate in the analytics configuration before you build dashboards that assume metrics are or are not cluster-wide.
  • Keep node-local operational state out of scope. Availability flags, maintenance markers, and anything else that describes one node rather than the cluster should not replicate, or a routine drain propagates to peers.

Readiness checklist​

  • Replication scope read off the running cluster, from every node, not reconstructed from memory
  • Whether the system database is in scope is a documented, deliberate decision
  • Identity provisioning procedure exists and matches that decision
  • Tables requiring one transaction boundary confirmed to be in one database
  • Durable topology constraints live in harper-config.yaml routes, not only in add_node calls
  • Measured convergence time recorded under load, per critical database
  • Alerts configured on missing peers, sustained latency growth, stalled lastCommitConfirmed, and the remote-to-local receive gap
  • Non-commutative operations identified and given an owner, a serialization point, or an external coordinator
  • Replication certificate expiry dates tracked with an alert
  • Replication interruption and recovery exercise completed and dated

Additional Resources​