Skip to content
PaloNexus
Request access Request

Backups & restore

Back up the tamper-evident audit hash-chain and verify its integrity after a restore. The same backup policy must also protect the registry, agent-idp state, LangGraph checkpointer, and issuer key.

What to back up

Section titled “What to back up”
DataWhereWhy it matters
Audit hash-chaincontrol-plane audit store (Loki for the shipped shipper; or the persisted audit log)Tamper-evidence + the system of record for every decision. Highest priority.
RegistryREGISTRY_DB_URL (Postgres/…)Which services/agents exist and their requireScope/allowlist/budget. Re-creatable from the declarative source, but back it up to avoid a re-seed.
agent-idp storeIDP_DB_URLAgent provisioning, delegations, revocations / StatusList. Losing revocation state could resurrect a revoked credential — back it up.
LangGraph checkpointerPALONEXUS_AGENT_DB_URLIn-flight human-in-the-loop (HITL) threads (paused approvals). Lose it and paused runs can’t resume.
Issuer keyagent-idp secretNot “data” but must survive — without it every issued Verifiable Credential (VC) fails to verify. Handle via Secrets, back up in the secret manager.

The audit chain is the crown jewel

Section titled “The audit chain is the crown jewel”

Each audit record hash-chains to its predecessor (prev_hash == previous.hash). That property is exactly what a backup must preserve: a restored chain that still verifies proves the backup was not tampered with in transit or at rest.

Terminal window
# Verify the live chain before and after any backup/restore:
curl -s localhost:8181/v1/audit/verify        # control-plane management plane
# or from the SDK:
python -c "from palonexus import PaloNexus; print(PaloNexus.from_env().audit.verify_chain())"
# -> True

In the shipped stack the chain is hash-chained JSON on the control-plane stdout, tailed by the audit-shipper DaemonSet into Loki (service.name=control-plane-audit). Back up by either snapshotting Loki’s storage, or by streaming the audit log to durable object storage (e.g. DigitalOcean Spaces) with retention. A persisted audit store, if used, is backed up like any DB.

Backing up the databases

Section titled “Backing up the databases”

With the postgres component (CloudNativePG, CNPG), use CNPG’s native backups — scheduled base backups + WAL archiving to object storage give point-in-time recovery:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata: { name: palonexus-pg-daily, namespace: palonexus }
spec:
  schedule: "0 2 * * *"
  backupOwnerReference: self
  cluster: { name: palonexus-pg }

Repeat for the agentidp-pg cluster. For a SQLite (PVC) backend, snapshot the PVC or copy the .db file while the writer is quiesced. For MongoDB, mongodump.

Restore drill (run it before it is needed)

Section titled “Restore drill (run it before it is needed)”

A backup that has never been restored is a hope, not a backup. The drill exports the audit chain plus the Postgres stores, restores them into a scratch namespace/cluster, and gates on verify_chain() — a restored chain that still verifies proves the backup is complete and untampered, and a deliberately edited row must break it. The sequence below is that drill end to end:

sequenceDiagram
  participant Src as Live cluster
  participant Bkp as Backup target - Spaces / CNPG WAL
  participant Scr as Scratch restore
  participant V as audit verify_chain
  Src->>Bkp: export audit chain + Postgres stores
  Note over Src,Bkp: registry, agent-idp, checkpointer
  Bkp->>Scr: restore base backup + WAL into scratch ns
  Scr->>V: GET /v1/audit/verify
  V-->>Scr: ok true, brokenAtSeq 0 - backup is good
  Scr->>V: re-verify after editing one row
  V-->>Scr: ok false, brokenAtSeq N - tamper detected

The restore drill: export the chain + stores, restore into scratch, then prove both that a clean chain verifies and that a single edited record is caught — the tamper-evidence guarantee, demonstrated.

Run this drill on a scratch namespace/cluster quarterly:

  1. Provision a fresh Postgres (CNPG Cluster or a throwaway instance) and restore the latest base backup + WAL (CNPG: bootstrap a new Cluster from: { backup: ... }).

  2. Restore the audit chain — point a control-plane at the restored audit store (or replay the archived audit log into Loki).

  3. Re-point REGISTRY_DB_URL / IDP_DB_URL / PALONEXUS_AGENT_DB_URL at the restored DBs and start the control plane + agent-idp.

  4. Restore the issuer key from the secret manager (so VCs still verify).

  5. Verify the chain — the drill’s pass/fail gate:

    Terminal window
    curl -s localhost:8181/v1/audit/verify        # must report the chain intact
    python -c "from palonexus import PaloNexus; assert PaloNexus.from_env().audit.verify_chain()"

  6. Spot-check — list registry services, confirm a known agent is provisioned, confirm a known revoked credential is still denied (revocation survived the restore).

Retention

Section titled “Retention”

Audit is a compliance artifact — set retention to the applicable regulatory window (Loki retention, or object-storage lifecycle rules on the archived log). The registry/agent-idp DBs only need enough history for operational recovery; the audit chain is what is kept long-term.

Section titled “Related”