CLI Reference

pact CLI is the primary interface for configuration management and admin operations. Every command is authenticated, authorized, and logged to the immutable journal.

Global Options

pact [OPTIONS] <COMMAND>

Environment Variables

Exit Codes

Authentication Commands

These commands manage OIDC authentication. login and logout are exempt from the “every command requires a valid token” rule (Auth1).

pact login

Authenticate with the pact-journal server via OIDC.

pact login                          # Interactive (Auth Code + PKCE)
pact login --server https://j:9443  # Explicit server URL
pact login --device-code            # Headless (Device Code flow)
pact login --service-account        # Machine identity (Client Credentials)

Flow selection: If no flag is given, the auth crate auto-discovers the IdP and selects the best available flow: Auth Code + PKCE → Device Code → Manual Paste.

Token cache: Tokens are stored at ~/.config/pact/auth/tokens-{server_hash}.json with mode 0600 (PAuth1: strict permission mode).

Roles: Not required (unauthenticated command).

pact logout

Clear the local token cache and revoke the session at the IdP (best-effort).

pact logout

Local cache is always cleared, even if IdP revocation fails (Auth4).

Roles: Not required (unauthenticated command).

Read Commands

These commands query state without modifying anything. Available to all roles including pact-viewer-{vcluster}.

pact status

Show node or vCluster state, drift, and capabilities.

pact status                          # All nodes in default vCluster
pact status node-042                 # Specific node
pact status --vcluster ml-training   # All nodes in a vCluster

pact log

Show configuration history from the immutable journal.

pact log                             # Last 20 entries
pact log -n 50                       # Last 50 entries
pact log --scope node:node-042       # Filter by node
pact log --scope vc:ml-training      # Filter by vCluster
pact log --scope global              # Global entries only

pact diff

Show declared vs actual state differences (drift).

pact diff                            # Current node
pact diff node-042                   # Specific node
pact diff --committed node-042       # Show committed node deltas not yet promoted

pact cap

Show node hardware capability report (CPU, GPU, memory, network).

pact cap                             # Local node
pact cap node-042                    # Remote node

pact watch

Live event stream from the journal. Streams events in real time until interrupted.

pact watch                           # Default vCluster
pact watch --vcluster ml-training    # Specific vCluster

Press Ctrl-C to stop the stream.

Write Commands

These commands modify configuration state. Requires pact-ops-{vcluster} or pact-platform-admin role. On regulated vClusters, write commands trigger the two-person approval workflow.

pact commit

Commit current drift on the node as a configuration entry in the journal.

pact commit -m "tuned hugepages for ML training"
pact commit -m "added NFS mount for datasets"

The commit is scoped to the current vCluster (from --vcluster, PACT_VCLUSTER, or config file). On regulated vClusters, this triggers approval workflow.

pact rollback

Roll back to a previous configuration state by sequence number.

pact rollback 42                     # Roll back to seq 42

Use pact log to find the sequence number you want to roll back to.

pact apply

Apply a declarative configuration spec from a TOML file.

pact apply overlay.toml              # Apply a spec file
pact apply /tmp/hugepages.toml       # Apply from absolute path

The spec file format matches the vCluster overlay format. See config/vcluster-examples/overlays.toml for the schema.

Exec Commands

These commands execute operations on remote nodes. Requires pact-ops-{vcluster} or pact-platform-admin role. All executions are logged to the journal.

pact exec

Run a whitelisted command on a remote node. The command and its output are recorded in the immutable audit log.

pact exec node-042 -- nvidia-smi
pact exec node-042 -- dmesg -T | tail -20
pact exec node-042 -- cat /proc/meminfo

Commands must be on the agent’s whitelist. Non-whitelisted commands return exit code 6.

pact shell

Open an interactive shell session on a remote node. This replaces SSH access.

pact shell node-042

Inside the shell, commands are subject to the whitelist policy configured on the agent (whitelist_mode in agent config). The session is fully logged.

pact:node-042> dmesg | tail -5
pact:node-042> cat /etc/hostname
pact:node-042> exit

pact service

Manage services on a node.

pact service status

pact service status                  # All services
pact service status chronyd          # Specific service

pact service restart

pact service restart nvidia-persistenced

Restarts are subject to the commit window. If the commit window has expired, you must commit or extend first.

pact service logs

pact service logs lattice-node-agent

Streams the last 50 log lines for the service.

Diagnostic Commands

Structured diagnostic log retrieval from nodes. Replaces ad-hoc pact exec for common log retrieval tasks with a purpose-built command that enforces server-side filtering.

pact diag

Collect diagnostic logs from one or more nodes. Logs are retrieved directly from the agent, which reads local sources (dmesg via /dev/kmsg, syslog, service logs under /run/pact/logs/). Grep filtering and line limits are enforced on the agent side, so only matching data crosses the network.

pact diag node-042                              # All sources, last 200 lines
pact diag node-042 --lines 500                  # Last 500 lines per source
pact diag node-042 --source dmesg               # Only kernel messages
pact diag node-042 --service nvidia-persistenced # Logs for a specific service
pact diag node-042 --grep "ECC"                 # Server-side grep across all sources
pact diag --vcluster ml-training                # Fleet-wide: all nodes in vCluster
pact diag --vcluster ml-training --grep "ECC"   # Fleet-wide log grep

In fleet mode (--vcluster), output lines are prefixed with [node_id]. Unreachable agents produce a warning and partial results are returned.

Roles: Requires pact-ops-{vcluster} or pact-platform-admin role (LOG1).

Design notes:

• Grep and line limit are enforced on the agent, not the CLI (LOG2, LOG3).
• Fleet fan-out: max 50 concurrent agent connections, 5s timeout per agent.

Admin Commands

These commands handle emergency operations and approval workflows.

pact emergency

Enter or exit emergency mode. Emergency mode relaxes policy constraints while maintaining the full audit trail. Use only for genuine emergencies.

pact emergency start

pact emergency start -r "GPU node unresponsive, need unrestricted diagnostics"

Emergency mode extends the commit window to 4 hours (configurable via emergency_window_seconds) and relaxes whitelist restrictions.

pact emergency end

pact emergency end                   # End your own emergency
pact emergency end --force           # Force-end another admin's emergency

pact approve

Manage the two-person approval workflow for regulated vClusters.

pact approve list

pact approve list

Lists all pending approval requests across vClusters you have access to.

pact approve accept

pact approve accept ap-7f3a

You cannot approve your own request. The approver must have pact-regulated-{vcluster} or pact-platform-admin role.

pact approve deny

pact approve deny ap-7f3a -m "change window not scheduled"

pact extend

Extend the current commit window.

pact extend                          # Extend by 15 minutes (default)
pact extend 30                       # Extend by 30 minutes

Delta Promotion

pact promote

Export committed node deltas as overlay TOML. This aggregates per-node configuration changes into a vCluster-wide overlay spec that can be reviewed, edited, and applied with pact apply.

pact promote node-042                # Export deltas as TOML to stdout
pact promote node-042 --dry-run      # Preview without generating output
pact promote node-042 > changes.toml # Export to file, then: pact apply changes.toml

If other nodes in the vCluster have local changes on the same config keys, promote detects the conflict and requires explicit acknowledgment (overwrite or keep local). See failure-modes.md FM-8.

Requires pact-ops-{vcluster} or pact-platform-admin role.

Node Enrollment Commands

These commands manage node enrollment, assignment, and inventory. Requires pact-ops-{vcluster} or pact-platform-admin role.

pact node enroll

Register a node with a hardware identity.

pact node enroll compute-001 --mac aa:bb:cc:dd:ee:01
pact node enroll compute-002 --mac aa:bb:cc:dd:ee:02 --bmc-serial SN12345

pact node import

Batch-import nodes from OpenCHAMI SMD inventory. Discovers nodes via the SMD /hsm/v2/State/Components API and enrolls them with their hardware identity (MAC addresses from /hsm/v2/Inventory/EthernetInterfaces).

Requires PACT_OPENCHAMI_SMD_URL to be configured.

pact node import                        # Import all nodes from SMD
pact node import --group Compute        # Import only nodes with role "Compute"

Environment variables:

pact node decommission

Decommission an enrolled node.

pact node decommission compute-001
pact node decommission compute-001 --force

pact node assign

Assign a node to a vCluster.

pact node assign compute-001 --vcluster ml-training

pact node unassign

Unassign a node from its vCluster.

pact node unassign compute-001

pact node move

Move a node between vClusters.

pact node move compute-001 --to-vcluster dev-sandbox

pact node list

List enrolled nodes with optional filters.

pact node list
pact node list --vcluster ml-training
pact node list --state active
pact node list --unassigned

pact node inspect

Show detailed enrollment information for a node.

pact node inspect compute-001

Node Lifecycle Commands

These commands manage node state transitions via delegation to external systems. Requires pact-ops-{vcluster} or pact-platform-admin role. All operations are logged to the journal.

pact drain

Drain workloads from a node. Delegates to lattice to gracefully migrate running workloads before taking the node out of service.

pact drain node-042

pact undrain

Cancel a drain operation, returning a draining node to Ready state.

pact undrain node-042

pact cordon

Mark a node as unschedulable. Existing workloads continue running but no new workloads will be placed on the node.

pact cordon node-042

pact uncordon

Remove a cordon from a node, making it schedulable again.

pact uncordon node-042

pact reboot

Reboot a node via BMC. Delegates to the configured node management backend (CSM CAPMC or OpenCHAMI Redfish).

pact reboot node-042

pact reimage

Re-image a node via the configured node management backend. CSM creates a BOS reboot session; OpenCHAMI triggers a Redfish power cycle (BSS serves the new image).

pact reimage node-042

Group Commands

Manage vCluster groups and their policies.

pact group list

List all vCluster groups.

pact group list
pact group list --output json

pact group show

Show details for a specific group.

pact group show ml-training

pact group set-policy

Update the policy for a group.

pact group set-policy ml-training --file policy.toml

Blacklist Commands

Manage drift detection exclusion patterns.

pact blacklist list

List current blacklist patterns for a node or vCluster.

pact blacklist list
pact blacklist list --vcluster ml-training

pact blacklist add

Add a pattern to the drift detection blacklist.

pact blacklist add "/var/cache/**"
pact blacklist add "/opt/scratch/**" --vcluster ml-training

pact blacklist remove

Remove a pattern from the drift detection blacklist.

pact blacklist remove "/var/cache/**"

Supercharged Commands (pact + lattice)

These commands combine pact and lattice data into unified views. They require PACT_LATTICE_ENDPOINT to be configured (or --lattice-endpoint flag).

Note: Lattice commands (including node lifecycle delegation) are hidden from pact --help when PACT_LATTICE_ENDPOINT is not set. They are always compiled in and can be invoked directly — they return a clear “not configured” error. Set the environment variable to see them in help output.

pact jobs list

List running job allocations across nodes.

pact jobs list                           # All jobs in default vCluster
pact jobs list --node node-042           # Jobs on a specific node
pact jobs list --vcluster ml-training    # Jobs in a vCluster

pact jobs cancel

Cancel a stuck or runaway job allocation.

pact jobs cancel alloc-7f3a

Requires pact-ops-{vcluster} or pact-platform-admin role.

pact jobs inspect

Show detailed information about a job allocation, including resource requests, node placement, and liveness probe configuration (displayed after the Resources section when probes are configured).

pact jobs inspect alloc-7f3a

pact queue

Show the scheduling queue status from lattice.

pact queue                               # Default vCluster
pact queue --vcluster ml-training        # Specific vCluster

pact cluster

Show combined Raft cluster health for both pact-journal and lattice quorums.

pact cluster

Displays leader status, term, committed index, and member health for both the pact journal Raft group and the lattice Raft group.

pact audit

Show a unified audit trail combining pact journal events and lattice audit events.

pact audit                               # pact events only (default)
pact audit --source all                  # Combined pact + lattice events
pact audit --source lattice              # Lattice events only
pact audit -n 50                         # Last 50 entries

pact accounting

Show resource usage accounting (GPU hours, CPU hours) aggregated from lattice.

pact accounting                          # Default vCluster
pact accounting --vcluster ml-training   # Specific vCluster

pact health

Combined system health check across pact and lattice components.

pact health

Reports health status for: pact-journal Raft quorum, pact-agent connectivity, lattice scheduler, lattice node-agents, OPA policy engine, telemetry pipeline, and Lattice Services (service/endpoint counts from the service registry).

pact services list

List services registered in the lattice service registry.

pact services list                          # All services
pact services list --vcluster ml-training   # Filter by vCluster

pact services lookup

Look up a specific service by name in the lattice service registry.

pact services lookup my-inference-api

Returns service details including registered endpoints, health status, and vCluster association.

pact dag

Manage DAG (directed acyclic graph) workflows in lattice.

pact dag list                            # List all DAGs
pact dag list --tenant ml-team           # Filter by tenant
pact dag list --state running            # Filter by state
pact dag inspect dag-7f3a                # Show DAG details and steps
pact dag cancel dag-7f3a                 # Cancel a running DAG

pact budget

Query resource budget and usage tracking from lattice.

pact budget tenant ml-team               # Tenant GPU/node hours
pact budget tenant ml-team --days 30     # Last 30 days
pact budget user alice                   # User usage across all tenants

pact backup

Manage lattice Raft state backups. Requires pact-platform-admin role.

pact backup create /path/to/backup.bin   # Create a backup
pact backup verify /path/to/backup.bin   # Verify backup integrity
pact backup restore /path/to/backup.bin --confirm  # Restore from backup

Note: restore requires the --confirm flag — it replaces the entire lattice scheduler state and cannot be undone.

pact nodes

Query lattice node inventory with hardware and ownership details.

pact nodes list                          # All nodes
pact nodes list --state draining         # Filter by state
pact nodes list --vcluster ml-training   # Filter by vCluster
pact nodes inspect node-042              # Full node details

Configuration File

The CLI reads its configuration from ~/.config/pact/cli.toml:

endpoint = "https://journal.example.com:9443"
default_vcluster = "ml-training"
output_format = "text"
timeout_seconds = 30
token_path = "~/.config/pact/token"

All fields are optional and have sensible defaults. See the Getting Started guide for the full precedence chain.