Skip to main content

5.3 Release Notes

Harper 5.3.0 is the first stable release of the 5.3 line. It adds native full-text search, a native memory-mapped HNSW vector index, branched databases, cluster-wide record locks, and a models.decide primitive for structured decisions. A deploy can now go back to a previous release by deployment_id. Role allowlists, SSH deploy keys, and node identity are validated more strictly, and replication, transaction, and storage correctness received a long list of fixes.

These notes summarize what changed relative to 5.2. The Harper Pro v5.3.0 release on GitHub is the complete list of changes: it includes the Harper core notes, the changes that were already backported to 5.2.x, and notes for anyone who ran a 5.3 pre-release.

Patch Releases​

All patch release notes for 5.3.x are available on the releases page.

Upgrade Notes​

Read these before upgrading a production node or cluster.

  • The RocksDB storage format is one-way. A RocksDB table that is dropped and recreated now uses a generation-suffixed column-family name. A release older than 5.3 opens the bare name and sees that table as empty, so do not downgrade a RocksDB node below 5.3 after upgrading (harper#2604).
  • Upgrade every node in a Harper Pro cluster before relying on the SSH key fixes or running the delete-echo repair tool. The SSH key name validation protects a cluster only once every node runs it. repairDeleteEchoRuns must run only after every node is on 5.3 (harper-pro#910, harper-pro#895).
  • A node.hostname or replication.hostname that is not a bare host stops the node from starting. See Node Identity.
  • Roles with a permission.operations allowlist can lose or gain access. Table DDL and SQL are now checked against the list, twenty operations the list could not previously grant are granted as listed, and the legacy catchup operation can no longer be granted. See Operation Allowlist Enforcement.
  • add_ssh_key and update_ssh_key refuse keys ssh cannot use, with a 400. A script that stored such a key now fails at the API instead of at deploy time. See SSH Deploy Keys Are Checked When Added.
  • Each node rewrites <rootPath>/ssh/config once at startup to add # BEGIN/END harper ssh key <name> lines around each key's block. An ssh config damaged by the pre-5.3 block-matching bug is not repaired automatically: run get_ssh_key, then delete_ssh_key, then add_ssh_key with the returned sealed key and the correct host/hostname. See SSH Key Blocks Are Marked in the ssh Config.
  • npm installs that use S3 export or import must add @aws-sdk/client-s3 and @aws-sdk/lib-storage. The AWS SDK is now an optional peer dependency. Without it, export_to_s3 and import_from_s3 return 501 with the install command. The official Docker image still includes it (harper#2609). See export_to_s3.
  • Restoring a backup taken with exclude_blobs requires allow_engine_only. Such a restore previously succeeded with only a log warning and could leave records pointing at the wrong blobs. A scripted engine-only restore, including one into a new target_database, fails until the flag is added (harper#2646).
  • Subscription catch-up delivers only the current version of each record by default. Pass includeSuperseded: true to Table.subscribe to also receive the retained history. rawEvents: true still includes every event, and durable MQTT subscriptions with QoS 1 or 2 still receive every retained publication (harper#2767). See Subscription Catch-up.
  • searchByIndex and a custom index's search() take an options object. A positional fifth argument to searchByIndex now throws a TypeError instead of silently permitting the scan it was meant to forbid (harper#2187).
  • Startup waits at most deployment.startupInstallTimeout for component installs before opening listeners. The default is 10 minutes, and 0 restores the previous unbounded wait. An install that runs past the limit finishes in the background, and when it succeeds Harper requests a restart to load it (harper#2784).
  • Some requests Harper used to accept are now refused: set_configuration with unrecognized parameters (harper#2272); a deploy or drop_component whose root config change HARPER_CONFIG or HARPER_SET_CONFIG would undo at the next start, with a 409 (harper#2801); and Table.clear() on a table with an active full-text declaration, with a 501 (harper#2616).
  • Harper Pro: a persisted isLeader: false now overrides a configured leader (HDB_LEADER_URL, the CLI, or routes[0]). 5.2 ignored a persisted false (harper-pro#800).
  • With mTLS client verification enabled, every client certificate is verified once more after the upgrade, because the verdict cache key changed (harper#2835).
  • Aliased databases with file-backed blobs now resolve blob paths to the first-assigned identity. A deployment whose blobs were stored under an alias that happened to load last may see them resolve differently. No migration ships with this change (harper#2683).

Full-Text Search​

Tables can declare @fullText fields and query them with BM25-ranked search through the existing Table.search, REST, and Operations API paths, with the same authorization as other queries (harper#2855). Declared fields are projected into local Tantivy indexes provided by the optional @harperfast/fulltext package. RocksDB and the audit log remain the source of truth: the index files can be rebuilt, are reused across restarts when compatible, and are caught up from the audit log before the index reports ready (harper#2569, harper#2615, harper#2616). Dropping a database or table releases every worker's index handles, and a drop that is interrupted is retried.

Vector Search: Native HNSW Index​

A newly declared HNSW index now uses a native, memory-mapped graph file (the optional @harperfast/hnsw package) instead of a graph stored in RocksDB, when it is eligible: the table is on RocksDB and has audit enabled (already recorded for the table, or set with audit: true in the same declaration), and the index uses the default int8-quantized cosine distance and default graph-structure options (M, efConstruction, mL, optimizeRouting). An ineligible index that leaves nativePlane unset, or any index that sets nativePlane: false, uses the previous graph. Setting nativePlane: true on an ineligible index is refused. Existing indexes keep the graph they were built with. Queries rerank only the rows they return, and index lag behind writes is bounded, with a way for a caller to wait until its prior writes are searchable (harper#2430, harper#2657, harper#2658).

The new per-index nativePlaneLayer0Cap defaults to 64 neighbors instead of a fixed 128, which roughly halves layer-0 memory with recall@10 within about 0.5 points at ef of 128 or more (harper#2701). Filtered vector search now answers indexed conditions from secondary indexes instead of decoding a record at every visited node; on a 20k-record test, 10%-selective filtered top-50 queries went from 7.0 ms to 4.3 ms with recall unchanged (harper#2691). See Vector Indexing.

Branched Databases​

An application can run against a private, durable fork of a database. Declare branchedDatabases: [data] in the application's entry in the root config, alongside host and urlPath, and the application reaches its fork through the same databases import it already uses, so several variants of an application can run against isolated copies of the same data. Declaring it in the application's own config.yaml is refused. Branching requires the RocksDB storage engine and the default module loader. An application that declares it under LMDB or the native loader fails to load rather than silently sharing the base database. The fork lives at a deterministic path, so it is picked up again on restart and resolves to the same place on every node (harper#2352).

A branch can declare its own tables through @table, ensureTable, and defineTable (harper#2523), copies the blob store by hard link (harper#2426), and is removed by drop_component (harper#2517).

Record Locks​

table.lock(id) gives an application an exclusive lock on one record, serialized across worker threads (harper#2462) and, in a Harper Pro cluster, across nodes (harper-pro#822). Ownership of a record's lock is held per record across calls, so repeatedly locking a busy record does not cost a network round trip each time (harper#2498). When a wait for a lock runs out, the response reports what the lock's owning node answered: 423 means the lock is held and the request can be retried, and 503 means coordination between nodes failed (harper#2685).

Models: models.decide​

Beside embed and generate, models.decide takes program state and a small closed schema (an enum, a boolean, a bounded integer, or one level of named fields) and returns the chosen value together with a probability distribution over the allowed values. An application can act automatically above a confidence threshold and route the rest to a person. It scores from token log-likelihoods where the backend supports it, supports an @decide schema directive and durable decisions with recorded outcomes, and is served by a new decision backend kind configured under models.decision (harper#2836, harper#2848). The models config block now also reloads without a restart (harper#2377).

Replication​

Replication and clustering are part of Harper Pro.

Transaction Correctness​

A replicated transaction is now applied whole, on one connection and for one origin node. One peer's transaction can no longer be split by or absorb another peer's, a frame that spans several origins is applied as one transaction per origin, a frame that ends without its end marker is aborted instead of partly committed, and a failed transaction is held and replayed a bounded number of times instead of skipped (harper#2800, harper-pro#907). A reduced form of the one-origin-per-transaction fix was backported to 5.2.x (harper-pro#908). A node that joins while a transaction is in flight no longer misses it (harper-pro#878), and records keep the version assigned on their origin node instead of taking the receiving node's clock (harper-pro#812). An out-of-order write older than the audit retention window no longer stalls replication (harper#2684).

Re-delivered deletes no longer echo across a mesh (a fix also backported to 5.2.x), but transaction logs written before that fix can still hold runs of echoed deletes. node dist/bin/repairDeleteEchoRuns.js <harper-root> reports on them on a stopped node, --apply repairs them and keeps backups, and --restore <backup-dir> undoes a repair. Run it only after every node is on 5.3 (harper-pro#895).

Retry Pacing​

Every replication retry now follows one jittered backoff schedule. Subscription setup holds at most one pending attempt per peer and database (200 ms minimum, jittered up to 30 seconds, reset on connect), so a transient DNS failure at boot can no longer drive unbounded setup retries and log volume until the process runs out of memory. Reconnects use full jitter within the existing 500 ms to 30 second bounds (harper-pro#800). A replicated operation that a peer does not answer now fails when the connection closes or at a caller-supplied deadline instead of waiting indefinitely (harper-pro#915).

Nodes and Clones​

update_node now works; the documented operation was never registered. A request that changes only revoked_certificates or shard updates the node record locally without resetting replication (harper-pro#916). See Update Node. A clone's wait for the initial sync is now bounded by the data size the leader reports and resumes across restarts, so a large clone no longer times out on a fixed budget or starts over (harper-pro#661).

Subscription Catch-up​

Table subscriptions now skip superseded record updates during catch-up with startTime or previousCount, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. The comparison uses the stored primary metadata version: a later publication can suppress a preceding mutation, and removing a record or tombstone makes its mutations ineligible. JavaScript consumers that need retained mutation history, including these cases, must set includeSuperseded: true. Raw-event subscriptions retain their existing default; rawEvents: true includes superseded events unless includeSuperseded: false is explicit.

MQTT durable subscriptions with QoS 1 or 2 explicitly include superseded versions in live delivery and reconnect catch-up, preserving intermediate retained publications. Non-retained published messages remain independent and are not suppressed by the table version check. See Subscription options and migration and MQTT durable sessions.

Node Identity​

node.hostname and replication.hostname must now be a bare hostname or IP literal. This value is the node's identity — it becomes the node's TLS certificate common name and the host that replication advertises to peers and dials to reach the node — so a URL or host:port value corrupted certificate matching and replication without surfacing an error: a node configured as http://host:9926 advertised and dialed a host literally named http (harper#2218).

Configuration validation now rejects these values at startup, so an existing install whose node.hostname or replication.hostname is a URL, host:port, or a non-string value will fail to start until it is corrected to a bare host. The startup error names the offending value and the reason it was rejected. A bare hostname (server-one), an IPv4 literal (10.0.0.5), and an unbracketed IPv6 literal (::1) are valid; a scheme, port, path, credentials, query string, fragment, bracketed IPv6 literal ([::1]), or non-string value is rejected. Route entries under replication.routes are unaffected and may still be URLs.

When node.hostname is unset, Harper resolves the identity to the first valid bare host among replication.hostname, the host in replication.url, the TLS certificate common name, and the Operations API host, falling back to 127.0.0.1. A derived source that is empty or unusable is skipped rather than failing startup.

Replication URLs now also bracket a bare IPv6 literal correctly (::1 becomes ws://[::1]:9933), which the URL parser previously rejected.

See Configuration Options.

Component Deploys​

Staged Build Retention​

deploy_component builds each release under <componentsRoot>/.deploy-staging/<deploymentId> and swaps it live only once it has built and installed. A build that completes but is never activated now survives startup, bounded per component by the new deployment.stagingRetention.maxCount setting (default 5; 0 keeps none). Older builds are removed at the start of the component's next deploy and at startup; anything belonging to an in-flight or unsettled deploy is never touched. Groundwork for deploying from an already-staged build (harper#2315). See Configuration Options.

Going Back to a Previous Release​

When a deploy replaces the release that is live, Harper now keeps the replaced one under the deployment_id that deployed it, and deploy_component with that deployment_id makes it live again. Nothing is rebuilt, resolved, or installed, and the root config entry published by that deployment comes back with it. Kept releases share deployment.stagingRetention.maxCount with staged builds (default 5). Each is a full installed copy of the component, so a component can now use up to that many extra copies' worth of disk; set it to 0 to keep none. Activating the id that is already live succeeds without a swap, so an activation that failed partway can be retried with the same id. Nodes that already switched answer success, and the rest switch (harper#2315). See Going back to a previous release.

Root Config Follows the Live Release​

A component's entry in the root harper-config.yaml now changes only once its new release is live. A package deploy used to write the entry before it built, so a deploy that failed to install or validate still left config naming the failed release, and the next restart installed it. The entry is now recorded with the deploy's activation and published after the swap, and startup settles a deploy that was interrupted partway through, so config and the live component agree after a crash, including whether the component runs isolated (harper#2315).

A payload deploy now removes package, install, and credentials from the component's entry, keeping its other keys such as host, urlPath, and isolated. Previously they stayed, so a node installing the component from its config entry installed the old package instead of the payload release that was live. See How a deploy updates the root config.

SSH Deploy Keys Are Checked When Added​

add_ssh_key and update_ssh_key now refuse a private key that ssh couldn't use, with a 400 that names the problem. Such a key used to be stored as sent, and the next deploy from a private repository failed with a generic authentication error. Refused keys include a public key sent in place of the private one, PuTTY and passphrase-protected keys, DSA and FIDO security keys, SSH certificates, unsupported algorithms, and damaged or mismatched keys. An accepted key is stored normalized (lines trimmed, blank lines dropped, a final newline), so an indented or CRLF paste now works. A host or hostname that would break the ssh config every key on the node shares, such as one containing a space, a quote or an =, or a pattern in host, is refused too.

Keys already stored are not re-checked, and a value already sealed as enc:v1: is checked only as an envelope for this cluster. When a node is cloned, a key on the leader that the new node refuses is skipped and logged by name, and the other keys are still cloned. If the leader keeps failing to return the keys, or the node can't store one, the clone stops Unavailable and unfinished, and a later start retries it, instead of completing without them. See What key, host and hostname must be.

SSH Key Blocks Are Marked in the ssh Config​

Each SSH key's block in <rootPath>/ssh/config now runs from its #<name> line through a # END harper ssh key <name> line, with a # BEGIN harper ssh key <name> line directly under the first. get_ssh_key, list_ssh_keys and delete_ssh_key read those lines instead of guessing where a block ends, so a line you add outside a key's lines is never removed with the key or read as its setting. Each node adds the lines to its existing config when it starts, without changing what ssh resolves, and a node rolled back to an earlier version still reads and deletes the blocks as it did before. See The key's block in the ssh config.

Backups​

Subscriptions End When Their Database Is Restored (5.3.1)​

An online restore_backup now ends every subscription to the database that was opened before the restore, once Harper reloads the database. Each one's last message is a DatabaseGenerationChangedError (status code 409, code DATABASE_GENERATION_CHANGED). Resubscribe to resynchronize against the restored state. A restore that fails before it changes anything reloads the database as it was, and its subscriptions end with a retryable DatabaseClosingError (status code 503, code DATABASE_CLOSING) instead. Previously such a subscription could stall silently, or start receiving the restored database's writes once another subscriber attached. See restore_backup.

Security​

Route-Owned Authentication​

Harper now defers rejection of an unrecognized Basic or Bearer credential on the application HTTP port until a handler claims the request. Harper-owned routes still return the normal unauthorized response. Unowned routes can fall through to application middleware with the original Authorization header unchanged. This lets catch-all applications and reverse proxies apply their own authentication schemes without URL exemptions or header rewriting, and the unauthorized response such an application returns — including its own WWW-Authenticate challenge — is passed back to the client unchanged. Internal authentication failures continue to fail closed. See Authentication and route ownership.

Operation Allowlist Enforcement​

A role's permission.operations allowlist is now checked before every other privilege check on the role, so an operation the list omits is denied whatever else the role carries. Table DDL and SQL previously went around it.

A role that relied on either will lose access on upgrade. Two cases to audit, both fixed by adding the operation to the list:

  • operations combined with structure_user — create_table, create_attribute, drop_table, and drop_attribute now have to be listed. structure_user still scopes them to its databases.
  • operations omitting sql — the role can no longer run SQL.

Roles with no operations field are unaffected, as are roles that already list everything they call. See Operation Permissions.

The reverse also changes: twenty operations that a list could name but never grant are now granted as listed — deploy_component, add_component, drop_component, package_component, set_component_file, set_custom_function, drop_custom_function, drop_custom_function_project, restart_service, set_configuration, get_status, set_status, clear_status, install_node_modules, delete_files_before, delete_audit_logs_before, delete_transaction_logs_before, cleanup_orphan_blobs, search_jobs_by_start_date, and registration_info. A role that already lists one gains it on upgrade, with no edit. A least-privilege CI user can therefore deploy through OIDC Trusted Publishing without being super_user. get_backup and read_transaction_log still cannot be granted by listing, and neither can the legacy catchup operation any more — it applies writes to any table with no table permission check, so a role that lists it loses it on upgrade.

OIDC Trusted Publishing​

A CI runner can now authenticate to Harper with no stored credential. It presents an identity token minted by its own provider, and if that token verifies against a trust policy configured on the instance, Harper returns a one-hour operation token for the user the policy names — the same exchange npm, PyPI, and AWS STS AssumeRoleWithWebIdentity use. This replaces a 30-day refresh-token secret with a rule you configure once and revoke with drop_oidc_trust.

Policies are managed with add_oidc_trust, list_oidc_trust, and drop_oidc_trust (all super_user), and live in the replicated system.hdb_oidc_trust table. A policy pins the issuer, an instance-specific audience, and a set of claim constraints. For GitHub Actions those constraints must pin the repository, pin the workflow, and gate the ref — a policy naming only a repository and workflow is refused, because anyone who can push a branch could otherwise add the trusted workflow to it and mint a token. Issuers with no registered profile must pin sub, which makes Kubernetes service accounts, GCP service accounts, and SPIFFE identities work with no provider-specific configuration.

A policy can also carry an operations list, narrowing the minted token to a subset of its user's role on the Operations API and SQL paths — narrowing only, and not covering an application's REST/GraphQL data path, so the policy's user should still be a least-privilege role for the data it can reach.

Identity tokens are single-use, though the replay record replicates asynchronously, so two simultaneous replays against different nodes can both succeed. Exchanges are recorded in the authentication audit stream when logging.auditAuthEvents.logSuccessful / logFailed is enabled — they are off by default, so turn them on before you need the trail. Every rejection returns the same message, with the specific reason written to the oidc-trust log. In the CLI, the exchange ranks below every configured credential, so enabling it does not silently re-point a pipeline that still has a token secret set. See OIDC Trusted Publishing and Workload identity.

SSH Key Operations Are Confined to Their Own Key​

get_ssh_key, update_ssh_key, and delete_ssh_key did not validate the key name. A crafted name could read, overwrite, or delete files outside the ssh directory, including the JWT signing key, and an update or delete replicated that change to every peer. All SSH key operations now apply one name rule, checked before anything replicates. A cluster is protected only once every node runs 5.3 (harper-pro#910). SSH private keys sent to add_ssh_key and update_ssh_key are also no longer written to the operations and MCP audit logs (harper#2741).

REST and HTTP​

Total-Count Pagination​

A REST collection request can ask for the total number of matching records with Prefer: count=estimated or Prefer: count=exact, and the response reports it in Content-Range. An exact count scans the full matched set, so it is honored only when the application's REST configuration sets exactCount: true. Otherwise a count=exact request is answered with an estimate and Preference-Applied: count=estimated (harper#2147). See Pagination and Total Count.

Faster Compressed Responses​

Buffered responses are now compressed with brotli quality 2, as streamed responses already were, instead of 11. A 242 KB JSON response on a 2-thread node went from between 0.9 and 3.8 seconds to about 0.7 seconds (harper#2899). See Compression.

Operations​

Interrupted Jobs Are Settled at Boot​

A job left running by a Harper process that has since exited is now marked ERROR at startup, instead of reporting as running indefinitely (harper#2645).