Skip to main content

How Harper Runs in Production

You have built and deployed an application, and now real users are going to depend on it. Operating Harper is not the same as operating separate application and database tiers. In Harper, one process holds your component code, the HTTP stack, the local database, and the replication client. That "collapsed stack" architecture reduces several problems you may be used to solving.

This guide is the map for the rest of the Administration track. It covers what a single node actually contains, what that means for failure and scale, and how to take an inventory of your own deployment before you design anything around it.

What You Will Learn​

  • What a Harper node contains, and which port carries which kind of traffic
  • Why the node is your unit of failure and your unit of scale, and what that removes from your operating burden as well as what it adds
  • The three ways a Harper runbook differs from a two-tier runbook
  • How to inventory your service boundary with the Operations API, so later guides have something concrete to work from

Prerequisites​

What one node contains​

A Harper node is a single process running your components, an HTTP server, a local storage engine, and peer replication. There is no network hop between your application code and the data it reads, no separate cache tier to keep coherent, and no connection pool to tune between tiers.

Three ports carry the traffic you will operate around:

PortServesWho should reach it
9926Application traffic: REST, WebSocket, MQTT-over-WebSocket, component routesYour traffic layer and your users
9925The Operations APIOperators and your pipeline only
9933Secure peer replicationOther nodes in the cluster only

These are documented defaults, not guarantees about your cluster. Confirm the live values rather than assuming them, because a replication port in particular can be inherited from other configuration:

{
	"operation": "get_configuration"
}

Read back http.port, operationsApi.network.port, and the replication block. Record what you find. Later guides in this track assume you know these numbers for your own deployment.

tip

Keep 9925 off any public route. The Operations API can deploy components, read logs, and read configuration, so it is an administrative surface, not an application one. See security overview.

The node is your unit of failure and your unit of scale​

Because one process holds the runtime and the local data together, there is no internal application-to-database seam that can fail over independently. The node is the practical unit of service failure.

Start with what that removes, because it is the larger half of the trade:

  • No cross-tier network latency on data access, and no tail latency from a saturated connection pool between tiers
  • No cache invalidation problem between an application cache and a database of record, because they are the same thing
  • No partial-outage state where the application tier is healthy and the data tier is not, which is the failure mode that produces the most confusing incidents
  • One capacity number to measure and one thing to size

Then the consequence: when you lose a node, you lose a whole slice of your service, not one layer of it. This is why the first real design decision in Sizing a Harper Cluster is not your peak throughput. It is how many nodes you are willing to lose at once, and whether the survivors can carry the load.

Scale works on the same unit. Adding capacity means adding a node that carries both request handling and data, so a scaling event is also a data movement event. That is not a problem, but it is a thing with a duration, and traffic should not arrive until it finishes.

Three differences that change your runbook​

A process that is up is not a node that should take traffic​

A Harper process will answer a TCP connection and return an HTTP response before it is a good place to send a user request. A new or replacement node has to synchronize the databases it serves before its answers are correct. A returning node has to catch up on transactions it missed.

So liveness and admission are two different decisions, and a load balancer health check that only proves liveness will route users to a node that is technically running and functionally wrong. This is the whole subject of Health Checks and Traffic Admission, and it is the single most common gap in a first production deployment.

Replication is peer-to-peer and scoped, so verify your own scope​

Harper peers exchange data over WebSockets with mTLS on the secure replication port and discover each other through configured routes. There is no primary. Data mutations and transactions replicate; some things do not, and the scope is configurable per database and per table.

The important operating habit is not memorizing a default. It is checking what your cluster actually replicates, because the answer depends on your configuration, your version, and whether anyone has scoped it since:

{
	"operation": "cluster_status"
}

The response lists each peer connection and, within it, one socket per database per peer. That tells you which databases are actually flowing, which is the question that matters during an incident. What is in scope, what is deliberately out of it, and how to prove convergence rather than just connection are covered in Operating Replication.

Reversal is a redeploy, not an infrastructure event​

Your application ships as a component, deployed with deploy_component from an immutable reference. Rolling back means deploying the previous immutable reference. There is no image to rebuild, no instance to replace, and no cluster to rebuild to undo a bad release.

That makes reversal fast enough to be a real option under pressure, which in turn makes it worth designing for deliberately rather than improvising. It also means code rollback, configuration rollback, and data recovery are three separate actions with three different blast radii, and conflating them during an incident is how a bad release becomes a data loss event. See Safe Deployments and Rollback.

Inventory your service boundary​

Everything else in this track builds on knowing what you have. Run these four operations against each node and write down the answers.

curl -s -X POST https://my-node.example.com:9925/ \
  -H 'Content-Type: application/json' \
  -u 'admin:password' \
  -d '{"operation":"system_information","attributes":["system","cpu","memory","disk","threads"]}'
warning

The attributes array silently drops names it does not recognize, so a typo returns a smaller response rather than an error. The valid values are system, time, cpu, memory, disk, network, harperdb_processes, table_size, metrics, and threads. If a response is missing a section you asked for, check the spelling before you check the node.

Then:

  • get_components for what is deployed, including which extensions are present
  • list_deployments for what changed and when
  • registration_info for the Harper version, which system_information does not report (it returns the node and npm versions, not Harper's)
  • cluster_status for the peers and databases actually connected

Fill in a boundary record you can keep:

ItemValue for your deployment
Harper versionFrom registration_info
Node count and rolesFrom cluster_status and your topology intent
Ports in useFrom get_configuration
DatabasesWhich exist, and which replicate
Storage enginePer database, since it constrains your backup options
Components deployedFrom get_components, with the version of each
Critical journeysThe user-facing paths that must work, named
Downstream dependenciesAnything Harper calls that can fail independently

The last two rows are the ones people skip and the ones that matter most. A healthy Harper process cannot compensate for a failed downstream dependency or for application logic returning wrong answers, so an operating model that only watches Harper will miss the incidents your users actually notice.

Prove it​

Before moving on, confirm the picture is real rather than assumed. On a non-production cluster, stop one node and watch what happens to the others: whether peers keep serving, how long the remaining nodes take to show the change in cluster_status, and what your traffic layer does about it. You are not measuring anything precisely yet. You are checking that the boundary you wrote down matches the system you have.

Operational notes​

  • Fabric and self-managed differ in what you own, not in how Harper behaves. On Fabric, cluster creation, certificates, and the metrics pipeline are managed for you. The failure unit, the replication model, and the admission problem are identical.
  • Version parity across nodes is an operating requirement, not a nicety. Mixed versions in a cluster change replication and deployment behavior. Record the version per node in your boundary inventory and alert on drift.
  • Configuration changes made through the API take effect on restart. A set_configuration call that has not been followed by a restart or restart_service leaves a node running something other than its stated configuration. Track pending changes in your change record.
  • get_status reports a restartRequired flag, but it tracks component and code restarts rather than configuration changes. Do not rely on it to tell you a configuration change is still pending.

Readiness checklist​

  • Ports confirmed from get_configuration, not assumed from documentation
  • Operations API on 9925 is not reachable from the public internet
  • Harper version recorded per node, with an alert on drift
  • Databases listed, with replication scope confirmed via cluster_status
  • Storage engine recorded per database
  • Components and versions recorded from get_components
  • Critical user journeys named and written down
  • Downstream dependencies named, with their own failure behavior understood

Additional Resources​