Transaction Logging

Harper maintains a transaction log for every database: a record of every data change, capturing the operation type, the user who made the change, the timestamp, and both the new and original record values. There is one transaction log per database, shared by all tables.

Audit log and transaction log are the same thing

Some operations and settings still carry the older "audit log" name — read_audit_log, delete_audit_logs_before, the logging.auditLog setting, and the @table(audit:) directive — but they all act on this single transaction log. The "audit log" is not a separate mechanism; the distinct-log terminology is a historical artifact.

Available since: v4.1.0

On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log.

The transaction log is enabled by default. To disable it, set logging.auditLog to false in harper-config.yaml and restart Harper.

The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use.

Operations

read_audit_log

Queries the transaction log for a specific table. Supports filtering by timestamp, username, or primary key value.

By timestamp:

{
	"operation": "read_audit_log",
	"schema": "dev",
	"table": "dog",
	"search_type": "timestamp",
	"search_values": [1660585740558]
}

Timestamp behavior:

search_values	Result
[]	All records for the table
[timestamp]	All records after the provided timestamp
[from, to]	Records between the two timestamps

By username:

{
	"operation": "read_audit_log",
	"schema": "dev",
	"table": "dog",
	"search_type": "username",
	"search_values": ["admin"]
}

By primary key:

{
	"operation": "read_audit_log",
	"schema": "dev",
	"table": "dog",
	"search_type": "hash_value",
	"search_values": [318]
}

Response example:

{
	"operation": "update",
	"user_name": "HDB_ADMIN",
	"timestamp": 1607035559122.277,
	"hash_values": [1, 2],
	"records": [
		{
			"id": 1,
			"breed": "Muttzilla",
			"age": 6,
			"__updatedtime__": 1607035559122
		}
	],
	"original_records": [
		{
			"__createdtime__": 1607035556801,
			"__updatedtime__": 1607035556801,
			"age": 5,
			"breed": "Mutt",
			"id": 1,
			"name": "Harper"
		}
	]
}

The original_records field contains the record state before the operation was applied.

delete_audit_logs_before

Deletes transaction log entries older than the specified timestamp. Deprecated in favor of delete_transaction_logs_before.

Changed in: v4.3.0 — Audit log cleanup improved to reduce resource consumption during scheduled cleanups

Changed in: v4.5.0 — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold

Changed in: v5.2.0 — This operation is unsupported on the RocksDB storage engine (the default): it requires table, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to delete_transaction_logs_before; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB.

{
	"operation": "delete_audit_logs_before",
	"schema": "dev",
	"table": "dog",
	"timestamp": 1598290282817
}

delete_transaction_logs_before

Engines: RocksDB, LMDB

Changed in: v5.2.0 — On RocksDB, a request that includes table now fails; previously the table scope was silently ignored and the entire database's transaction log was purged. On either engine, a table that does not exist now fails with a not-found error (previously a typo'd table fell through to the database-wide purge on RocksDB, and was a silent no-op on LMDB). On LMDB, a valid table continues to scope the deletion to that table's history, unchanged.

Deletes transaction log entries older than the specified timestamp.

Database-wide and irreversible

On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit table and pass only database (or schema) and timestamp. This purges whole log files whose entries predate the timestamp, removing read_audit_log history for every table in the database — there is no per-table survivor and no undo. The only recovery route is a backup, which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position.

Parameters:

• database (or the deprecated schema alias): string (required) — a request naming neither fails validation before a job starts; a database that does not exist fails the job with a not-found error.
• timestamp: number (required) — epoch milliseconds; entries older than this are deleted.
• table: string (LMDB only) — scopes deletion to that table's history. On RocksDB the job fails (see the warning above).
• cleanup_deleted_records: boolean (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB.

On LMDB, the table-scoped deletion scans the database's full audit history (and cleanup_deleted_records: true adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history.

Request validation runs first and synchronously: a request that omits both database and schema is rejected immediately with an error and no job ID (there is nothing to poll). Once accepted, the operation runs as a background job that returns 200 with a job ID, and operation-time failures surface through get_job — the job ends with status ERROR and a message describing the failure (for example, the RocksDB table-scope rejection, or a table/database that does not exist).

RocksDB (database-wide — omit table):

{
	"operation": "delete_transaction_logs_before",
	"database": "dev",
	"timestamp": 1598290282817
}

LMDB (table is required — omitting it deletes nothing and reports entries_deleted: 0 with job status COMPLETE):

{
	"operation": "delete_transaction_logs_before",
	"database": "dev",
	"table": "dog",
	"timestamp": 1598290282817
}

Response:

{
	"message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69",
	"job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69"
}

get_job reports the outcome. A successful job's result carries entries_deleted and log_files_deleted (the count of purged log files on RocksDB; 0 on LMDB, which has no separate log files) — the record of how much was deleted. A failed job looks like:

[
	{
		"id": "2fe25039-566e-4670-8bb3-2db3d4e07e69",
		"type": "delete_transaction_logs_before",
		"status": "ERROR",
		"message": "There was an error running deleteTransactionLogsBefore job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69 - Table-level transaction log deletion is not supported for RocksDB tables because all tables in a database share one transaction log; to delete the transaction logs for the entire 'dev' database, use delete_transaction_logs_before with only 'database' and 'timestamp'"
	}
]

---

Enabling the Transaction Log Per Table

You can enable or disable the transaction log for individual tables using the @table directive's audit argument in your schema:

type Dog @table(audit: true) {
	id: Long @primaryKey
	name: String
}

This overrides the logging.auditLog global configuration for that specific table.

Related Documentation

• Logging — Application and system logging (separate from the transaction log)
• Replication — Replication and clustering, which consume the transaction log
• Logging Configuration — Global transaction log configuration (logging.auditLog)
• Operations API — Sending operations to Harper