Analytics

Added in: v4.5.0 (resource and storage analytics)

Harper collects real-time telemetry and statistics across all operations, URL endpoints, and messaging topics. This data can be used to monitor server health, understand traffic and usage patterns, identify resource-intensive queries, and inform scaling decisions.

Storage Tables

Analytics data is stored in two system tables in the system database:

Table	Description
hdb_raw_analytics	Per-second raw entries recorded by each thread. One record per second per active thread.
hdb_analytics	Aggregate entries recorded once per minute, summarizing all per-second data across threads.

Both tables require superuser permission to query.

Raw Analytics (hdb_raw_analytics)

Raw entries are recorded once per second (when there is activity) by each thread. Each record captures all activity in the last second along with system resource information. Records use the timestamp in milliseconds since epoch as the primary key.

Query raw analytics using search_by_conditions on the hdb_raw_analytics table. The example below fetches 10 seconds of raw entries:

POST http://localhost:9925
Content-Type: application/json

{
    "operation": "search_by_conditions",
    "schema": "system",
    "table": "hdb_raw_analytics",
    "conditions": [{
        "search_attribute": "id",
        "search_type": "between",
        "search_value": [1688594000000, 1688594010000]
    }]
}

Example raw entry:

{
	"time": 1688594390708,
	"period": 1000.8336279988289,
	"metrics": [
		{
			"metric": "bytes-sent",
			"path": "search_by_conditions",
			"type": "operation",
			"median": 202,
			"mean": 202,
			"p95": 202,
			"p90": 202,
			"count": 1
		},
		{
			"metric": "memory",
			"threadId": 2,
			"rss": 1492664320,
			"heapTotal": 124596224,
			"heapUsed": 119563120,
			"external": 3469790,
			"arrayBuffers": 798721
		},
		{
			"metric": "utilization",
			"idle": 138227.52767700003,
			"active": 70.5066209952347,
			"utilization": 0.0005098165086230495
		}
	],
	"threadId": 2,
	"totalBytesProcessed": 12182820,
	"id": 1688594390708.6853
}

Aggregate Analytics (hdb_analytics)

Aggregate entries are recorded once per minute, combining per-second raw entries from all threads into a single summary record. Use search_by_conditions on the hdb_analytics table with a broader time range:

POST http://localhost:9925
Content-Type: application/json

{
    "operation": "search_by_conditions",
    "schema": "system",
    "table": "hdb_analytics",
    "conditions": [{
        "search_attribute": "id",
        "search_type": "between",
        "search_value": [1688194100000, 1688594990000]
    }]
}

Example aggregate entry:

{
	"period": 60000,
	"metric": "bytes-sent",
	"method": "connack",
	"type": "mqtt",
	"median": 4,
	"mean": 4,
	"p95": 4,
	"p90": 4,
	"count": 1,
	"id": 1688589569646,
	"time": 1688589569646
}

Standard Metrics

Harper automatically tracks the following metrics for all services. Applications can also define custom metrics via server.recordAnalytics().

HTTP Metrics

metric	path	method	type	Unit	Description
duration	resource path	request method	cache-hit or cache-miss	ms	Duration of request handler
duration	route path	request method	fastify-route	ms	Duration of Fastify route handler
duration	operation		operation	ms	Duration of Operations API operation
success	resource path	request method		%	Percentage of successful requests
success	route path	request method	fastify-route	%
success	operation		operation	%
bytes-sent	resource path	request method		bytes	Response bytes sent
bytes-sent	route path	request method	fastify-route	bytes
bytes-sent	operation		operation	bytes
transfer	resource path	request method	operation	ms	Duration of response transfer
transfer	route path	request method	fastify-route	ms
transfer	operation		operation	ms
socket-routed				%	Percentage of sockets immediately routed
tls-handshake				ms	TLS handshake duration
tls-reused				%	Percentage of TLS sessions reused
cache-hit	table name			%	Percentage of cache hits
cache-resolution	table name			ms	Duration of resolving uncached entries

MQTT / WebSocket Metrics

metric	path	method	type	Unit	Description
mqtt-connections				count	Number of open direct MQTT connections
ws-connections				count	Number of open WebSocket connections
connection	mqtt	connect		%	Percentage of successful direct MQTT connections
connection	mqtt	disconnect		%	Percentage of explicit direct MQTT disconnects
connection	ws	connect		%	Percentage of successful WebSocket connections
connection	ws	disconnect		%	Percentage of explicit WebSocket disconnects
bytes-sent	topic	mqtt command	mqtt	bytes	Bytes sent for a given MQTT command and topic

Replication Metrics

metric	path	method	type	Unit	Description
bytes-sent	node.database	replication	egress	bytes	Bytes sent for replication
bytes-sent	node.database	replication	blob	bytes	Bytes sent for blob replication
bytes-received	node.database	replication	ingress	bytes	Bytes received for replication
bytes-received	node.database	replication	blob	bytes	Bytes received for blob replication
replication-latency	node.database.table		ingest	ms	Time difference from source commit timestamp to local time

Resource Usage Metrics

metric	Key attributes	Other	Unit	Description
database-size	size, used, free, audit	database	bytes	Database file size breakdown
main-thread-utilization	idle, active, taskQueueLatency, rss, heapTotal, heapUsed, external, arrayBuffers	time	various	Main thread resource usage: idle/active time, queue latency, and memory breakdown
read-transaction-queue-depth	depth, maxDepth		count	Open tracked transactions holding a read handle (see transaction queue depth)
resource-usage	(see below)		various	Node.js process resource usage (see resource-usage)
storage-volume	available, free, size	database	bytes	Storage volume size breakdown
table-size	size	database, table	bytes	Table file size
transaction-commit-time	mean, median, p90, p95, p99, p999		ms	Duration from write commit submission to settlement (see transaction queue depth)
utilization			%	Percentage of time the worker thread was processing requests
write-transaction-queue-depth	depth, maxDepth		count	In-flight write transaction commits (see transaction queue depth)

Transaction Queue Depth Metrics

Added in: v5.2.0

write-transaction-queue-depth and read-transaction-queue-depth expose how many transactions are in flight against the storage engine per worker thread — a concurrency and throughput signal, not a reliable predictor on their own of the Outstanding write transactions have too long of queue, please try again later (HTTP 503) rejection: maxDepth amplitude reflects concurrent commits, not whether any single one is approaching the storage.maxTransactionQueueTime duration limit (default 45s) that actually trips the 503.

transaction-commit-time records each commit's submit-to-settle duration on that same clock, and a rising p99/p999 (in the hdb_analytics aggregate table, where percentiles are computed — they aren't present on hdb_raw_analytics) is a leading indicator of gradual slowdowns approaching that limit. It doesn't help with a single commit that hangs indefinitely, though: the metric only records once a commit settles, so a genuinely wedged commit contributes no sample at all, while write-transaction-queue-depth's depth stays elevated on that thread for as long as the commit remains outstanding. Harper also logs Rejecting writes on this thread: a commit has been outstanding for ... once per stuck commit when the 503 check itself fires, which is the authoritative signal for that specific failure.

Field	Unit	Description
depth	count	Instantaneous depth sampled at emit time
maxDepth	count	High-water mark since this thread's last emitted sample

• write-transaction-queue-depth counts write commits handed to the storage engine whose commit promises have not yet settled — how many commits this thread is juggling concurrently. This is in-flight, not durability: a settled commit promise means the storage engine accepted the write, not that it has been synced to disk.
• read-transaction-queue-depth counts concurrently open tracked transactions holding a read handle, including write transactions and transactions opened with snapshot disabled. A high count can mean either many short-lived transactions or a few long-lived ones — the count alone can't distinguish them, so use it as a concurrency signal; a duration-based metric would be needed to identify a single transaction held open long enough to hold back compaction.

Both metrics are gauges tracked only on the RocksDB write/read path, sampled per worker thread, so activity against LMDB-backed databases is not counted at all. On a mixed install they report RocksDB traffic only and silently under-count. They also carry no database dimension, so a reported depth cannot be attributed to a specific database. On an install with no RocksDB databases (storage.engine: lmdb), both always read 0 — indistinguishable from a healthy, empty queue — regardless of actual read/write load. All per-thread analytics reporting, including these gauges, piggybacks on the thread having recorded some other analytics-eligible activity in the period — a thread with no recordable activity in a given second emits no row at all rather than an explicit depth: 0. Absence of a sample is not the same as a healthy reading, particularly for read-transaction-queue-depth on an otherwise-quiet thread holding a single long-lived read.

The raw per-thread entries in hdb_raw_analytics retain each thread's true instantaneous depth and per-period maxDepth; treat those as the reliable source for spike detection. The aggregate hdb_analytics table is not a sum of per-thread peaks — each thread's maxDepth is first averaged across its raw samples for the period, then those per-thread averages are summed — so a brief single-thread spike is diluted rather than preserved. Always alert on the maxDepth field of each queue-depth entry in an hdb_raw_analytics record's metrics array (or lower the sampling/aggregation period) rather than relying on the aggregate table to catch short spikes. Tune the concrete alert threshold against a baseline for your workload, since absolute depth scales with worker-thread count and per-transaction size.

resource-usage Metric

Includes everything returned by Node.js process.resourceUsage() (with userCPUTime and systemCPUTime converted to milliseconds), plus:

Field	Unit	Description
time	ms	Unix timestamp when the metric was recorded
period	ms	Duration of the measurement period
cpuUtilization	%	CPU utilization (user + system combined)

Custom Metrics

Applications can record custom metrics using the server.recordAnalytics() API. See HTTP API for details.

Analytics Configuration

The analytics configuration section controls aggregation, replication, and storage-volume sampling. All options are optional.

analytics:
  aggregatePeriod: 60
  storageInterval: 10
  replicate: false
  logging:
    level: info

analytics.aggregatePeriod

Type: number (seconds)

Default: 60

How frequently Harper aggregates raw per-second entries into the hdb_analytics summary table. Lowering this gives higher-resolution aggregate data at the cost of more frequent aggregation work and more rows in hdb_analytics.

analytics.storageInterval

Type: number

Default: 10

Number of aggregation cycles between disk-volume measurements. With the default aggregatePeriod of 60 and storageInterval of 10, Harper records database-size, table-size, and storage-volume metrics every 10 minutes. Set to 0 to disable storage-volume sampling entirely — useful when running on systems where statfs is expensive or unavailable (e.g., some FUSE mounts).

analytics.replicate

Type: boolean

Default: false

When enabled, aggregate analytics entries are replicated across the cluster so a single peer can answer aggregate queries for the whole topology. Raw per-thread entries (hdb_raw_analytics) are always node-local. Enable when running a centralized analytics consumer; leave disabled in large clusters to avoid replication overhead for high-cardinality metrics.

analytics.logging

Type: object

Per-subsystem logging override for the analytics writer. See Logging Configuration — analytics.logging.

Related

• Analytics Operations
• HTTP API
• Logging Configuration
• Configuration Overview