Kiseki

Kiseki is a distributed storage system built for HPC and AI workloads. It provides a unified data plane that serves files and objects through multiple protocol gateways (S3, NFS, FUSE) while handling encryption, replication, and caching transparently.

Key Features

• S3 and NFS gateways – access the same data through S3-compatible HTTP, NFSv3/v4.2, or a native FUSE mount. Protocol gateways translate wire protocols into operations on the shared log-structured data model.

• Client-side cache with staging – a two-tier cache (L1 in-memory, L2 local NVMe) on compute nodes eliminates repeated fabric traversals. Three modes (pinned, organic, bypass) match the dominant workload patterns: epoch-reuse training, mixed inference, and streaming ingest.

• Per-shard Raft consensus – every shard is a single-tenant Raft group. Deltas (metadata mutations) are totally ordered within a shard and replicated to a quorum before acknowledgement.

• Erasure coding and placement – chunks are stored across affinity pools with configurable EC profiles. The placement engine distributes data across device classes (fast-NVMe, bulk-NVMe) and rebuilds lost chunks from parity.

• FIPS 140-2/3 encryption – always-on, two-layer envelope encryption. System DEKs (AES-256-GCM via aws-lc-rs) encrypt chunk data; tenant KEKs wrap the DEKs for access control. Five tenant KMS backends: Kiseki-Internal, HashiCorp Vault, KMIP 2.1, AWS KMS, PKCS#11.

• GPU-direct and fabric transports – the native client selects the fastest available transport: libfabric/CXI (Slingshot), RDMA verbs, or TCP+TLS. Transport selection is automatic based on fabric discovery.

• Multi-tenant isolation – tenant hierarchy (organization / project / workload) with per-level quotas, compliance tags, and key isolation. Shards are single-tenant. Cross-tenant data access is out of scope by design.

• OIDC and mTLS authentication – Keycloak (or any OIDC provider) for identity; Cluster CA-signed mTLS certificates for data-fabric authentication. Certificates work on the SAN with no control plane access needed on the hot path.

• Workflow advisory – a bidirectional advisory channel carries workload hints (access pattern, prefetch range, priority) inbound and telemetry feedback (backpressure, locality, staleness) outbound. The advisory path is side-by-side with the data path – it never blocks or delays data operations.

Architecture at a Glance

Kiseki is a single-language Rust system organized as 18 crates in a Cargo workspace:

The data model is log-structured: mutations are recorded as deltas appended to per-shard Raft logs. Compositions describe how content-addressed, encrypted chunks assemble into files or objects. Views are materialized projections of shard state, maintained incrementally by stream processors and served by protocol gateways.

Four binaries are produced:

Target Workloads

Quick Links

• Getting Started – Docker Compose quickstart
• S3 API – supported operations, examples
• NFS Access – NFSv3/v4.2 mount instructions
• FUSE Mount – native client mount
• Python SDK – PyO3 bindings
• Client Cache & Staging – ADR-031 cache modes

Getting Started

This guide walks through running a single-node Kiseki stack with Docker Compose, verifying the deployment, and performing basic S3 operations.

Prerequisites

• Docker 24+ with Compose V2 (docker compose)
• curl (for health checks)
• aws-cli (optional, for S3 operations)

If building from source instead of Docker:

• Rust 1.78+ (stable)
• Protobuf compiler (protoc)

Quick Start with Docker Compose

The repository includes a docker-compose.yml that brings up a single-node Kiseki server with supporting services:

Start the stack:

docker compose up --build -d

Wait for all services to become healthy:

docker compose ps

The kiseki-server container sets KISEKI_BOOTSTRAP=true, which creates an initial shard for immediate use.

Verify the Deployment

Health Check

The data-path gRPC port responds to TCP connections when the server is ready:

# TCP probe on the data-path port
timeout 1 bash -c 'echo > /dev/tcp/127.0.0.1/9100'
echo $?  # 0 = healthy

Prometheus Metrics

curl -s http://localhost:9090/metrics | head -20

Jaeger Tracing

Open http://localhost:16686 in a browser to view distributed traces. The server exports traces via OTLP to Jaeger automatically.

Vault (Dev Mode)

Vault runs in dev mode with root token kiseki-e2e-token:

curl -s http://localhost:8200/v1/sys/health | python3 -m json.tool

Keycloak

Keycloak is available at http://localhost:8080 with admin credentials admin / admin.

S3 Operations

With aws-cli configured to point at the local S3 gateway:

# Configure a local profile (no real AWS credentials needed)
export AWS_ACCESS_KEY_ID=kiseki
export AWS_SECRET_ACCESS_KEY=kiseki
export AWS_DEFAULT_REGION=us-east-1

# Create a bucket (maps to a Kiseki namespace)
aws --endpoint-url http://localhost:9000 s3 mb s3://test-bucket

# Upload a file
echo "hello kiseki" > /tmp/hello.txt
aws --endpoint-url http://localhost:9000 s3 cp /tmp/hello.txt s3://test-bucket/hello.txt

# Download and verify
aws --endpoint-url http://localhost:9000 s3 cp s3://test-bucket/hello.txt /tmp/hello-back.txt
cat /tmp/hello-back.txt

Or with curl directly:

# List buckets
curl -s http://localhost:9000/

# PUT an object
curl -X PUT http://localhost:9000/test-bucket/greeting.txt \
     -d "hello from curl"

# GET it back
curl -s http://localhost:9000/test-bucket/greeting.txt

Multi-Node Cluster

A three-node cluster configuration is also provided:

docker compose -f docker-compose.3node.yml up --build -d

This starts three kiseki-server instances that form Raft groups for shard replication.

Building from Source

# Clone and build
git clone https://github.com/your-org/kiseki.git
cd kiseki
cargo build --release

# Run the server
KISEKI_BOOTSTRAP=true \
KISEKI_DATA_DIR=/tmp/kiseki-data \
KISEKI_S3_ADDR=0.0.0.0:9000 \
KISEKI_NFS_ADDR=0.0.0.0:2049 \
KISEKI_DATA_ADDR=0.0.0.0:9100 \
KISEKI_METRICS_ADDR=0.0.0.0:9090 \
  ./target/release/kiseki-server

Next Steps

• S3 API – full list of supported S3 operations
• NFS Access – mount via NFS
• FUSE Mount – native client mount on compute nodes
• Python SDK – use Kiseki from Python workloads
• Client Cache & Staging – pre-stage datasets for training jobs

S3 API

Kiseki exposes an S3-compatible HTTP gateway on port 9000 (configurable via KISEKI_S3_ADDR). The gateway implements the subset of S3 API operations needed by HPC/AI workloads (ADR-014). Unsupported operations return 501 Not Implemented.

Endpoint

http://<node>:9000

In the Docker Compose development stack, the endpoint is http://localhost:9000.

Authentication

Kiseki supports AWS Signature Version 4 authentication:

• Authorization header – standard SigV4 signing for aws-cli, boto3, and other SDK clients.
• Presigned URLs – planned for a future release (not yet implemented).

In development mode (Docker Compose), any access key and secret key values are accepted.

Supported Operations

Bucket Operations

S3 buckets map to Kiseki namespaces. Creating a bucket creates a tenant-scoped namespace; deleting a bucket deletes the namespace.

Object Operations

Multipart Upload

For objects larger than a single PUT (large datasets, model weights):

Versioning

Conditional Operations

Examples

aws-cli

# Set up environment
export AWS_ACCESS_KEY_ID=kiseki
export AWS_SECRET_ACCESS_KEY=kiseki
export AWS_DEFAULT_REGION=us-east-1
ENDPOINT="--endpoint-url http://localhost:9000"

# Bucket operations
aws $ENDPOINT s3 mb s3://datasets
aws $ENDPOINT s3 ls

# Upload a directory
aws $ENDPOINT s3 sync ./training-data/ s3://datasets/imagenet/

# Download a file
aws $ENDPOINT s3 cp s3://datasets/imagenet/train.tar /tmp/train.tar

# Multipart upload (automatic for large files)
aws $ENDPOINT s3 cp ./large-model.bin s3://datasets/models/gpt.bin

# List objects with prefix
aws $ENDPOINT s3 ls s3://datasets/imagenet/ --recursive

# Delete
aws $ENDPOINT s3 rm s3://datasets/imagenet/train.tar

curl

# Create a bucket
curl -X PUT http://localhost:9000/my-bucket

# PUT an object
curl -X PUT http://localhost:9000/my-bucket/config.json \
     -H "Content-Type: application/json" \
     -d '{"epochs": 100, "batch_size": 32}'

# GET an object
curl -s http://localhost:9000/my-bucket/config.json

# HEAD an object (metadata only)
curl -I http://localhost:9000/my-bucket/config.json

# Byte-range read (first 1024 bytes)
curl -s http://localhost:9000/my-bucket/large-file.bin \
     -H "Range: bytes=0-1023"

# DELETE an object
curl -X DELETE http://localhost:9000/my-bucket/config.json

# List objects (ListObjectsV2)
curl -s "http://localhost:9000/my-bucket?list-type=2&prefix=models/"

# Delete a bucket
curl -X DELETE http://localhost:9000/my-bucket

Python (boto3)

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:9000",
    aws_access_key_id="kiseki",
    aws_secret_access_key="kiseki",
    region_name="us-east-1",
)

# Create bucket
s3.create_bucket(Bucket="training")

# Upload
s3.put_object(Bucket="training", Key="data.csv", Body=b"col1,col2\n1,2\n")

# Download
obj = s3.get_object(Bucket="training", Key="data.csv")
print(obj["Body"].read().decode())

# List
for item in s3.list_objects_v2(Bucket="training")["Contents"]:
    print(item["Key"], item["Size"])

Bucket-to-Namespace Mapping

Every S3 bucket maps 1:1 to a Kiseki namespace within the authenticated tenant’s scope. Bucket names become namespace identifiers. Buckets from different tenants are fully isolated – two tenants can have buckets with the same name without conflict.

Objects within a bucket map to Kiseki compositions. Each object version corresponds to a sequence of deltas in the shard that owns the namespace.

Encryption Handling

Kiseki always encrypts all data (invariant I-K1). S3 server-side encryption headers are handled as follows:

Limitations

The following S3 features are not implemented:

NFS Access

Kiseki exposes an NFS gateway on port 2049 (configurable via KISEKI_NFS_ADDR) supporting both NFSv3 and NFSv4.2. The gateway translates NFS operations into reads and writes against materialized views and the composition log.

Protocol Support

Mounting

Basic Mount

mount -t nfs <node>:/ /mnt/kiseki

With explicit version and options:

# NFSv4.2
mount -t nfs -o vers=4.2,proto=tcp <node>:/ /mnt/kiseki

# NFSv3
mount -t nfs -o vers=3,proto=tcp <node>:/ /mnt/kiseki

Docker Compose (Development)

When using the development Docker Compose stack, the NFS port is published to the host:

mount -t nfs -o vers=4.2,proto=tcp,port=2049 127.0.0.1:/ /mnt/kiseki

fstab Entry

<node>:/ /mnt/kiseki nfs vers=4.2,proto=tcp,hard,intr 0 0

Authentication

In development (Docker Compose), AUTH_SYS is used with no additional configuration. For production deployments, Kerberos provides authentication and optional integrity/privacy protection on the wire.

Kiseki always encrypts data at rest regardless of the NFS authentication mode. The gateway performs tenant-layer encryption: clients send plaintext over TLS to the gateway, and the gateway encrypts before writing to the log and chunk store.

Supported Operations

Full Semantics

Limited Semantics

Not Supported

Namespace Mapping

The NFS root (/) lists the tenant’s namespaces as top-level directories. Each namespace contains the compositions (files and directories) belonging to that namespace. This is analogous to the S3 bucket mapping – the same namespace appears as a bucket via S3 and as a top-level directory via NFS.

/mnt/kiseki/
  training/          <- namespace "training"
    imagenet/
      train.tar
      val.tar
  checkpoints/       <- namespace "checkpoints"
    epoch-001.pt

Performance Considerations

• Readdir performance – directory listings are served from materialized views, not reconstructed from the log on each request. Views are updated incrementally by stream processors.

• Write path – writes flow through the gateway to the composition context, which appends deltas to the shard log. An fsync ensures the delta is committed to a Raft quorum before returning.

• Concurrent access – multiple NFS clients can read the same files concurrently. Write contention within a shard is serialized by the Raft leader.

• Large files – large files are chunked using content-defined chunking (Rabin fingerprinting). Byte-range reads are served by fetching only the relevant chunks.

Limitations Summary

1. No writable shared mmap – applications that use writable shared memory-mapped files must use write() instead. Read-only mmap works and is useful for model loading.

2. Cross-namespace rename returns EXDEV – renaming a file from one namespace to another requires a copy-and-delete at the application level, same as moving files across filesystem boundaries on a traditional system.

3. No POSIX ACLs – only standard Unix permissions (mode bits). Fine-grained access control is handled by Kiseki’s tenant IAM model, not filesystem-level ACLs.

4. Lock state is per-gateway – POSIX file locks (fcntl) are maintained by the gateway instance. If a gateway fails over, lock state is lost. Advisory locks (flock) are best-effort.

FUSE Mount

The Kiseki native client provides a FUSE mount that exposes the distributed storage as a local filesystem on compute nodes. Unlike the NFS gateway, the FUSE client runs in the workload’s process space and performs client-side encryption – plaintext never leaves the process.

Building

The FUSE mount is feature-gated. Build the client binary with the fuse feature:

cargo build --release --bin kiseki-client-fuse --features fuse

This requires the fuser crate, which depends on the FUSE kernel module being available on the host:

• Linux: install fuse3 or libfuse3-dev
• macOS: install macFUSE

Mounting

kiseki-client-fuse mount /mnt/kiseki \
    --data-addr <storage-node>:9100 \
    --tenant <tenant-id> \
    --namespace <namespace-id>

Mount Options

Options are passed with -o:

kiseki-client-fuse mount /mnt/kiseki \
    -o cache_mode=organic \
    -o cache_dir=/local-nvme/kiseki-cache \
    -o cache_l2_max=100G \
    -o meta_ttl_ms=5000

Environment Variables

Mount options can also be set via environment variables. Mount options take priority over environment variables.

Supported Operations

Read/Write

Directory Operations

Metadata and Links

Nested Directories and Write-at-Offset

The FUSE filesystem supports full directory trees within a namespace. Files can be created in nested directories, and writes at arbitrary offsets within a file are supported (the composition tracks chunk references and handles sparse regions with zero-fill).

mkdir -p /mnt/kiseki/experiments/run-42/logs
echo "epoch 1 loss: 0.3" > /mnt/kiseki/experiments/run-42/logs/train.log

# Write at offset (sparse file)
dd if=/dev/zero of=/mnt/kiseki/data/sparse.bin bs=1 count=1 seek=1048576

Not Supported

Cache Mode Selection

The cache mode determines how aggressively the client caches data on local storage. Choose the mode that matches your workload:

# Training job: pin the dataset
kiseki-client-fuse mount /mnt/kiseki -o cache_mode=pinned

# Interactive exploration
kiseki-client-fuse mount /mnt/kiseki -o cache_mode=organic

# Checkpoint writer
kiseki-client-fuse mount /mnt/kiseki -o cache_mode=bypass

See Client Cache & Staging for staging pre-fetch, Slurm integration, and policy configuration.

Transport Selection

The native client automatically selects the fastest available transport to reach storage nodes:

1. libfabric/CXI (Slingshot) – if available on the fabric
2. RDMA verbs – if InfiniBand/RoCE is available
3. TCP+TLS – universal fallback

Transport selection is automatic and requires no configuration. The client discovers available transports during fabric discovery at startup (ADR-008).

Unmounting

fusermount -u /mnt/kiseki    # Linux
umount /mnt/kiseki           # macOS

On clean unmount, the L2 cache pool is wiped (all chunk files are zeroized and deleted). On crash, the orphaned cache pool is cleaned up by the next client process or by the kiseki-cache-scrub service.

Python SDK

Kiseki provides Python bindings via PyO3, exposing the native client’s cache, staging, and workflow advisory APIs to Python workloads. The bindings are part of the kiseki-client crate, enabled with the python feature flag.

Building

Build and install the Python module using maturin:

pip install maturin
maturin develop --features python

This builds the native Rust code and installs the kiseki module into the active Python environment.

For a release build:

maturin build --release --features python
pip install target/wheels/kiseki-*.whl

Quick Start

import kiseki

# Create a client with organic caching (default)
client = kiseki.Client(cache_mode="organic", cache_dir="/tmp/kiseki-cache")

# Stage a dataset into the local cache
client.stage("/training/imagenet")

# ... workload reads via FUSE or native API ...

# Check cache statistics
stats = client.cache_stats()
print(stats)
# CacheStats(l1_hits=42, l2_hits=1500, misses=200, l1_bytes=134217728, l2_bytes=5368709120, wipes=0)

# Release the staged dataset
client.release("/training/imagenet")

# Clean up
client.close()

API Reference

kiseki.Client

The main entry point. Each Client instance manages its own cache pool (L1 in-memory + L2 NVMe) and advisory session.

Constructor

client = kiseki.Client(
    cache_mode="organic",           # "pinned", "organic", or "bypass"
    cache_dir="/tmp/kiseki-cache",  # L2 NVMe cache directory
    cache_l2_max=50 * 1024**3,      # L2 max bytes (default: 50 GB)
    meta_ttl_ms=5000,               # Metadata TTL in ms (default: 5000)
)

stage(namespace_path: str) -> None

Pre-fetch a dataset’s chunks into the local cache with pinned retention. The dataset is identified by its namespace path (e.g., "/training/imagenet"). Staging is idempotent – re-staging an already-staged dataset is a no-op.

client.stage("/training/imagenet")
client.stage("/training/imagenet")  # no-op, already staged

For directory paths, staging recursively enumerates all files up to a depth of 10 and a maximum of 100,000 files.

stage_status() -> list[str]

Return the namespace paths of all currently staged datasets.

paths = client.stage_status()
# ["/training/imagenet", "/models/gpt-3"]

release(namespace_path: str) -> None

Release a staged dataset, unpinning its chunks and making them eligible for eviction.

client.release("/training/imagenet")

release_all() -> None

Release all staged datasets.

client.release_all()

cache_stats() -> CacheStatsView

Return current cache statistics.

stats = client.cache_stats()
print(f"L1 hits: {stats.l1_hits}")
print(f"L2 hits: {stats.l2_hits}")
print(f"Misses:  {stats.misses}")
print(f"L1 used: {stats.l1_bytes / 1024**2:.0f} MB")
print(f"L2 used: {stats.l2_bytes / 1024**3:.1f} GB")
print(f"Wipes:   {stats.wipes}")

cache_mode() -> str

Return the current cache mode as a string.

print(client.cache_mode())  # "organic"

declare_workflow() -> int

Declare a new workflow for advisory integration. Returns a workflow ID (128-bit integer) that can be used to correlate operations with the advisory channel for telemetry feedback.

wf_id = client.declare_workflow()
# ... run training epochs ...
client.end_workflow(wf_id)

end_workflow(workflow_id: int) -> None

End a previously declared workflow.

wipe() -> None

Immediately wipe the entire cache (L1 + L2). All cached plaintext is zeroized before deletion.

close() -> None

Wipe the cache and release resources. Call this when the workload is done. Equivalent to wipe().

kiseki.CacheStatsView

Read-only statistics object returned by cache_stats().

Example: Training Workflow

import kiseki

def train():
    # Pin the dataset for the duration of training
    client = kiseki.Client(cache_mode="pinned", cache_dir="/local-nvme/cache")

    # Pre-stage the dataset (ideally done in Slurm prolog)
    client.stage("/training/imagenet-22k")

    # Declare a workflow for advisory telemetry
    wf_id = client.declare_workflow()

    try:
        for epoch in range(100):
            # Dataset reads hit L2 cache after first epoch
            # ... training loop reads from /mnt/kiseki/training/imagenet-22k/ ...
            pass

        stats = client.cache_stats()
        print(f"Cache hit rate: {(stats.l1_hits + stats.l2_hits) / "
              f"(stats.l1_hits + stats.l2_hits + stats.misses) * 100:.1f}%")
    finally:
        client.end_workflow(wf_id)
        client.release_all()
        client.close()

if __name__ == "__main__":
    train()

Example: Inference with Organic Caching

import kiseki

client = kiseki.Client(cache_mode="organic", cache_l2_max=20 * 1024**3)

# Model weights are cached on first load, then served from L2
# Prompt data is cached with LRU eviction

wf_id = client.declare_workflow()
try:
    # ... inference serving loop ...
    pass
finally:
    client.end_workflow(wf_id)
    client.close()

Example: Checkpoint Writer (No Caching)

import kiseki

# Bypass mode: checkpoint writes go straight to canonical
client = kiseki.Client(cache_mode="bypass")

# ... write checkpoints to /mnt/kiseki/checkpoints/ ...

client.close()

Environment Variable Overrides

The Python client respects the same environment variables as the FUSE mount and CLI:

Constructor parameters take priority over environment variables. All client-set values are clamped to the effective policy ceilings set by tenant and cluster administrators.

Client Cache & Staging

The client-side cache (ADR-031) eliminates repeated data transfers across the storage fabric by caching decrypted plaintext chunks on compute-node local NVMe. It is a library-level module in kiseki-client, shared across all access modes: FUSE, FFI, Python, and native Rust.

Architecture

canonical (fabric) -> decrypt -> cache store (NVMe) -> serve to caller
                                     ^
                           cache hit path (no fabric, no decrypt)

Two-Tier Storage

Read path: L1 -> L2 (with CRC32 verification) -> canonical (decrypt + SHA-256 verify + store in L1/L2).

L2 files are organized per-process with isolated cache pools:

$KISEKI_CACHE_DIR/
  <tenant_id_hex>/
    <pool_id>/                  <- per-process pool (128-bit CSPRNG)
      chunks/
        <prefix>/
          <chunk_id_hex>        <- plaintext + CRC32 trailer
      meta/
        file_chunks.db
      staging/
        <dataset_id>.manifest
      pool.lock                 <- flock proves process is alive

Each client process creates its own pool directory. Multiple concurrent same-tenant processes on the same node have fully independent pools with no contention.

Security Model

The cache stores decrypted plaintext on local NVMe. This is acceptable because:

• The compute node already holds decrypted data in process memory (computation requires plaintext)
• L2 NVMe is local to the compute node, same trust domain as process memory
• L2 is ephemeral – wiped on process exit and on long disconnect
• All cached data is overwritten with zeros (zeroize) before deallocation or eviction
• File permissions are 0600, owned by the process UID
• Orphaned pools from crashes are cleaned by the kiseki-cache-scrub service

Cache Modes

Three modes are available, selected per client instance at session establishment.

Pinned Mode

For workloads that declare their dataset upfront: training runs (epoch reuse), inference (model weights), climate simulations (boundary conditions).

• Chunks are retained against eviction until explicit release()
• Populated via the staging API or on first access
• Staging captures a point-in-time snapshot; canonical updates do not invalidate pinned data
• Capacity bounded by max_cache_bytes; staging beyond capacity returns CacheCapacityExceeded

Organic Mode

Default for mixed workloads. LRU with usage-weighted retention.

• Chunks cached on first read, evicted when capacity is reached
• Frequently accessed chunks promoted to L1
• L2 eviction: LRU by last-access timestamp, weighted by access count (chunks accessed N times survive N eviction rounds)
• Metadata cache with configurable TTL (default 5 seconds)

Bypass Mode

For workloads that do not benefit from caching: streaming ingest, one-shot scans, checkpoint writes.

• All reads go directly to canonical
• No L1 or L2 storage consumed
• Zero overhead beyond mode selection

Staging API

Client-local operation for pre-populating the cache in pinned mode. Pull-based – the client fetches from canonical.

CLI

# Stage a dataset
kiseki-client stage --dataset /training/imagenet

# Stage in daemon mode (for Slurm prolog)
POOL_ID=$(kiseki-client stage --dataset /training/imagenet --daemon)

# Check staging status
kiseki-client stage --status

# Release a dataset
kiseki-client stage --release /training/imagenet

# Release all
kiseki-client stage --release-all

Rust API

#![allow(unused)]
fn main() {
let result = cache_manager.stage("/training/imagenet").await?;
let datasets = cache_manager.stage_status();
cache_manager.release("/training/imagenet");
cache_manager.release_all();
}

Python API

client.stage("/training/imagenet")
paths = client.stage_status()
client.release("/training/imagenet")
client.release_all()

C FFI

kiseki_stage(handle, "/training/imagenet", timeout_secs);
kiseki_stage_status(handle, &status);
kiseki_release(handle, "/training/imagenet");

Staging Flow

1. Resolve namespace_path to compositions via canonical. For directory paths, recursively enumerate all files up to max_staging_depth (10) and max_staging_files (100,000).
2. Extract full chunk list from all resolved compositions.
3. For each chunk not already in L2: fetch from canonical, decrypt, verify content-address (SHA-256), store in L2 with CRC32 trailer and pinned retention.
4. Write a staging manifest listing all compositions and chunk IDs.
5. Report progress (chunks staged / total, bytes, elapsed).

Staging is idempotent – re-staging an already-staged dataset is a no-op. Partial staging (interrupted) can be resumed by re-running the command.

Slurm Integration

Staging Handoff

The staging CLI creates a cache pool and holds its pool.lock flock. The workload process adopts the pool instead of creating a new one:

1. Prolog: staging CLI fetches chunks in daemon mode, outputs pool_id.
2. Workload: sets KISEKI_CACHE_POOL_ID=<pool_id>, starts, adopts the existing pool, takes over the flock.
3. Staging daemon: detects flock loss, exits cleanly.

Prolog Script

#!/bin/bash
# prolog.sh -- run before the job starts

POOL_ID=$(kiseki-client stage --dataset /training/imagenet --daemon)
echo "export KISEKI_CACHE_POOL_ID=$POOL_ID" >> $SLURM_EXPORT_FILE

Epilog Script

#!/bin/bash
# epilog.sh -- run after the job completes

kiseki-client stage --release-all --pool $KISEKI_CACHE_POOL_ID

Lattice Integration

Lattice injects KISEKI_CACHE_POOL_ID into the workload environment after parallel staging completes across the node set. It queries stage --status to verify readiness before launching the workload.

Policy Hierarchy

Cache policy follows the same distribution mechanism as quotas, using the existing TenantConfig structure.

cluster default -> org override -> project override -> workload override
                                                         -> session selection

Each level narrows (never broadens) the parent’s settings.

Policy Attributes

Policy Resolution

At session establishment, the client resolves its effective policy through multiple paths:

1. Primary: GetCachePolicy RPC on the data-path gRPC channel to any storage node. No gateway or control plane access required.
2. Secondary: gateway’s locally-cached TenantConfig.
3. Stale tolerance: last-known policy persisted in the L2 pool directory (policy.json).
4. Fallback: conservative defaults (organic mode, 10 GB max, 5s TTL).

Policy changes apply to new sessions only. Active sessions continue under the policy effective at session establishment.

Configuration

Environment Variables

Mount Options (FUSE)

kiseki-client-fuse mount /mnt/kiseki \
    -o cache_mode=pinned \
    -o cache_dir=/local-nvme/kiseki \
    -o cache_l2_max=100G

API (Rust)

#![allow(unused)]
fn main() {
let config = CacheConfig {
    mode: CacheMode::Pinned,
    cache_dir: PathBuf::from("/local-nvme/kiseki"),
    max_cache_bytes: 100 * 1024 * 1024 * 1024,
    metadata_ttl: Duration::from_secs(5),
    ..CacheConfig::default()
};
}

API (Python)

client = kiseki.Client(
    cache_mode="pinned",
    cache_dir="/local-nvme/kiseki",
    cache_l2_max=100 * 1024**3,
)

Priority: API/mount options > environment variables > policy defaults. All client-set values are clamped to policy ceilings.

Cache Invalidation

Metadata

TTL-based only. No push invalidation from canonical. The metadata TTL (default 5 seconds) is the sole freshness mechanism and the upper bound on read staleness.

Write-through: when the client writes a file, the local metadata cache is updated immediately, providing read-your-writes consistency within a single process.

Crypto-Shred

When a tenant’s KEK is destroyed, all cached plaintext for that tenant must be wiped. Detection via three paths:

1. Periodic key health check (default every 30 seconds) – primary.
2. Advisory channel notification – fast path, best-effort.
3. KMS error on next operation – tertiary.

Maximum detection latency: min(key_health_interval, max_disconnect_seconds) = 30 seconds by default.

Disconnect

If the client cannot reach any canonical endpoint for max_disconnect_seconds (default 300 seconds), the entire cache is wiped. Background heartbeat RPCs (every 60 seconds) maintain the disconnect timer.

Capacity Management

Pinned chunks are never evicted by organic LRU. Organic eviction considers only non-pinned chunks.

Crash Recovery

• On process start: the client scans for orphaned cache pools (those whose pool.lock has no live flock holder), zeroizes their contents, and deletes them.
• kiseki-cache-scrub service: a systemd one-shot (or cron job) that runs on node boot and every 60 seconds, covering the case where no subsequent Kiseki process starts on the node after a crash.

Deployment

This guide covers deploying Kiseki in development, multi-node cluster, and bare-metal production environments.

Docker Compose (development)

The single-node development stack includes Kiseki plus supporting services for tracing, KMS, and identity.

Services

Starting the stack

# Build and start all services
docker compose up --build

# Run in background for e2e tests
docker compose up --build -d && pytest tests/e2e/

Port map (single-node)

Environment (dev defaults)

The development compose file sets these environment variables on the kiseki-server container:

KISEKI_DATA_ADDR: "0.0.0.0:9100"
KISEKI_ADVISORY_ADDR: "0.0.0.0:9101"
KISEKI_S3_ADDR: "0.0.0.0:9000"
KISEKI_NFS_ADDR: "0.0.0.0:2049"
KISEKI_METRICS_ADDR: "0.0.0.0:9090"
KISEKI_DATA_DIR: "/data"
KISEKI_BOOTSTRAP: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger:4317"
OTEL_SERVICE_NAME: "kiseki-server"

The KISEKI_BOOTSTRAP=true flag tells the node to create an initial shard on first start, enabling immediate use without manual cluster initialization.

Vault (dev mode)

Vault runs in dev mode with the root token kiseki-e2e-token. This is suitable only for development and testing. The Transit secrets engine is used by Kiseki as a tenant KMS backend (ADR-028 Provider 2).

# Verify Vault is ready
curl http://localhost:8200/v1/sys/health

Keycloak (dev mode)

Keycloak runs with start-dev and default admin credentials (admin/admin). Configure OIDC realms for tenant identity provider integration.

Docker Compose (3-node cluster)

The multi-node compose file (docker-compose.3node.yml) deploys a 3-node Raft cluster for testing consensus, replication, and failover.

Starting

docker compose -f docker-compose.3node.yml up --build -d

# Run multi-node tests
KISEKI_E2E_COMPOSE=docker-compose.3node.yml pytest tests/e2e/test_multi_node.py

Node configuration

All three nodes share the same Raft peer list and each has a unique KISEKI_NODE_ID:

The Raft peer list is configured identically on all nodes:

KISEKI_RAFT_PEERS=1=kiseki-node1:9300,2=kiseki-node2:9300,3=kiseki-node3:9300

Node 1 is the bootstrap node. Each node has an independent data volume (node1-data, node2-data, node3-data).

Verifying cluster health

# Check all nodes are healthy
for port in 9100 9110 9120; do
  curl -s http://localhost:${port/9100/9090}/health && echo " :$port OK"
done

# View cluster status via the admin dashboard
open http://localhost:9090/ui

Bare metal deployment

Build from source

Prerequisites: Rust stable toolchain, protobuf compiler (protoc), OpenSSL development headers, pkg-config.

# Clone and build
git clone https://github.com/your-org/kiseki.git
cd kiseki

# Release build (all binaries)
cargo build --release

# Binaries produced:
# target/release/kiseki-server      — storage node
# target/release/kiseki-keyserver    — system key manager (HA)
# target/release/kiseki-client-fuse  — FUSE client for compute nodes
# target/release/kiseki-control      — control plane

Optional feature flags:

# Enable CXI/Slingshot transport (requires libfabric)
cargo build --release --features kiseki-transport/cxi

# Enable RDMA verbs transport
cargo build --release --features kiseki-transport/verbs

# Enable tenant opt-in compression
cargo build --release --features kiseki-chunk/compression

Disk layout

Each storage node should follow the recommended disk layout:

Server node:
  System partition (RAID-1 on 2x SSD):
    /var/lib/kiseki/raft/log.redb       Raft log entries
    /var/lib/kiseki/keys/epochs.redb    Key epoch metadata
    /var/lib/kiseki/chunks/meta.redb    Chunk extent index
    /var/lib/kiseki/small/objects.redb   Small-file inline content
    /var/lib/kiseki/config/             Node config, TLS certs

  Data devices (JBOD, managed by Kiseki):
    /dev/nvme0n1 -> pool "fast-nvme"
    /dev/nvme1n1 -> pool "fast-nvme"
    /dev/sda     -> pool "bulk-ssd"
    /dev/sdb     -> pool "cold-hdd"

JBOD for data devices, RAID-1 for the system partition. Kiseki manages data durability via EC/replication across JBOD members. The system partition uses RAID-1 because redb and Raft log must survive a single disk failure without Kiseki’s own repair mechanism.

systemd unit: kiseki-server

[Unit]
Description=Kiseki Storage Node
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=simple
User=kiseki
Group=kiseki
ExecStart=/usr/local/bin/kiseki-server
Restart=on-failure
RestartSec=5

# Environment
Environment=KISEKI_DATA_ADDR=0.0.0.0:9100
Environment=KISEKI_ADVISORY_ADDR=0.0.0.0:9101
Environment=KISEKI_S3_ADDR=0.0.0.0:9000
Environment=KISEKI_NFS_ADDR=0.0.0.0:2049
Environment=KISEKI_METRICS_ADDR=0.0.0.0:9090
Environment=KISEKI_DATA_DIR=/var/lib/kiseki
Environment=KISEKI_NODE_ID=1
Environment=KISEKI_RAFT_PEERS=1=node1.example.com:9300,2=node2.example.com:9300,3=node3.example.com:9300
Environment=KISEKI_RAFT_ADDR=0.0.0.0:9300

# TLS
Environment=KISEKI_CA_PATH=/etc/kiseki/tls/ca.crt
Environment=KISEKI_CERT_PATH=/etc/kiseki/tls/server.crt
Environment=KISEKI_KEY_PATH=/etc/kiseki/tls/server.key

# Observability
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger.internal:4317
Environment=OTEL_SERVICE_NAME=kiseki-server
Environment=RUST_LOG=kiseki=info

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/kiseki
PrivateTmp=yes
MemoryDenyWriteExecute=yes
LimitCORE=0

[Install]
WantedBy=multi-user.target

systemd unit: kiseki-keyserver

[Unit]
Description=Kiseki System Key Manager
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=kiseki-keys
Group=kiseki-keys
ExecStart=/usr/local/bin/kiseki-keyserver
Restart=on-failure
RestartSec=5

Environment=KISEKI_DATA_DIR=/var/lib/kiseki-keys
Environment=KISEKI_RAFT_PEERS=1=keysrv1:9400,2=keysrv2:9400,3=keysrv3:9400
Environment=KISEKI_RAFT_ADDR=0.0.0.0:9400
Environment=KISEKI_CA_PATH=/etc/kiseki/tls/ca.crt
Environment=KISEKI_CERT_PATH=/etc/kiseki/tls/keyserver.crt
Environment=KISEKI_KEY_PATH=/etc/kiseki/tls/keyserver.key

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/kiseki-keys
PrivateTmp=yes
MemoryDenyWriteExecute=yes
LimitCORE=0

[Install]
WantedBy=multi-user.target

systemd unit: kiseki-client-fuse

[Unit]
Description=Kiseki FUSE Client
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/kiseki-client-fuse --mountpoint /mnt/kiseki
ExecStop=/bin/fusermount -u /mnt/kiseki
Restart=on-failure
RestartSec=5

Environment=KISEKI_DATA_ADDR=node1.example.com:9100,node2.example.com:9100,node3.example.com:9100
Environment=KISEKI_CA_PATH=/etc/kiseki/tls/ca.crt
Environment=KISEKI_CERT_PATH=/etc/kiseki/tls/client.crt
Environment=KISEKI_KEY_PATH=/etc/kiseki/tls/client.key
Environment=KISEKI_CACHE_MODE=organic
Environment=KISEKI_CACHE_DIR=/var/cache/kiseki
Environment=KISEKI_CACHE_L1_MAX=1073741824
Environment=KISEKI_CACHE_L2_MAX=107374182400

[Install]
WantedBy=multi-user.target

Configuration checklist

Before starting a production cluster, verify the following:

TLS certificates

• Cluster CA certificate generated and distributed to all nodes
• Per-node server certificate signed by Cluster CA
• Per-tenant client certificates signed by Cluster CA
• Key manager server certificate signed by Cluster CA
• CRL distribution point configured (if using CRL-based revocation)
• Certificate SANs include all node hostnames and IP addresses
• All certificates use ECDSA P-256 or RSA 2048+ keys

Data directories

• KISEKI_DATA_DIR exists and is owned by the kiseki user
• System partition has sufficient capacity for metadata (see Capacity Planning)
• Data devices formatted and accessible (raw block or file-backed)
• Separate RAID-1 for system partition

Bootstrap

• Exactly one node has KISEKI_BOOTSTRAP=true on first start
• After initial bootstrap, set KISEKI_BOOTSTRAP=false on the bootstrap node (or remove the variable)
• KISEKI_RAFT_PEERS is identical on all nodes
• KISEKI_NODE_ID is unique per node
• System key manager cluster is started before storage nodes

Network

• Data-fabric ports (9100, 9101) reachable between all nodes
• Raft port (9300) reachable between all nodes
• Metrics port (9090) accessible to monitoring infrastructure
• NFS port (2049) accessible to clients
• S3 port (9000) accessible to clients
• Management network separated from data fabric (recommended)

Observability

• Jaeger or OTLP-compatible collector endpoint configured
• Prometheus scrape target added for each node’s :9090/metrics
• RUST_LOG level set appropriately (production: kiseki=info)

Health verification

After deployment, verify the cluster is healthy:

HTTP health endpoint

# Returns "OK" when the node is ready
curl http://node1:9090/health

Prometheus metrics

# Verify metrics are being exported
curl -s http://node1:9090/metrics | head -20

Admin dashboard

Open http://node1:9090/ui in a browser. The dashboard shows:

• Cluster health (nodes healthy / total)
• Raft entries applied
• Gateway requests served
• Data written and read
• Active transport connections

Any node in the cluster serves the full cluster-wide view by scraping metrics from its peers.

Raft consensus

Verify that the Raft cluster has elected a leader:

# Check the cluster status via the admin API
curl -s http://node1:9090/ui/api/cluster | jq .

S3 connectivity

# Test S3 access (if a tenant namespace is configured)
aws --endpoint-url http://node1:9000 s3 ls

NFS connectivity

# Test NFS mount
mount -t nfs node1:/ /mnt/kiseki -o vers=4.2

FUSE client

# Mount via FUSE (on a compute node)
kiseki-client-fuse --mountpoint /mnt/kiseki
ls /mnt/kiseki

Configuration Reference

Kiseki is configured entirely through environment variables. There are no configuration files to manage. Every tunable parameter has a sensible default. Variables are grouped by function below.

Network addresses

All addresses accept the host:port format. Use 0.0.0.0 to bind to all interfaces or a specific IP to restrict to one network.

Cluster membership

Storage

Data directory layout

KISEKI_DATA_DIR/
  raft/log.redb            Raft log entries (bounded by snapshot policy)
  keys/epochs.redb         Key epoch metadata (<10 MB)
  chunks/meta.redb         Chunk extent index (scales with file count)
  small/objects.redb        Small-file encrypted content (capacity-managed)

TLS / mTLS

When KISEKI_CA_PATH is not set, the server runs without TLS. This is acceptable for development but must not be used in production.

Client-side cache (ADR-031)

These variables configure the native client cache on compute nodes running kiseki-client-fuse.

Cache behavior notes

• Pinned mode: Pre-staged datasets remain in cache until explicitly released. Best for training workloads that re-read the same data across epochs.
• Organic mode: LRU eviction with usage-weighted retention. Default for mixed workloads.
• Bypass mode: No caching at all. Best for checkpoint/restart and streaming workloads.
• On process restart, the client creates a new L2 pool (wiping orphaned pools). A kiseki-cache-scrub service cleans orphans on node boot.
• Disconnects longer than 300 seconds (configurable) wipe the entire cache.
• Crypto-shred events wipe all cached plaintext for the affected tenant within the key health check interval (default 30 seconds).

Metadata capacity (ADR-030)

These variables control the dynamic inline threshold for small-file placement.

The inline threshold determines whether a file’s encrypted content is stored in small/objects.redb (metadata tier, NVMe) or as a chunk extent on a raw block device (data tier). The threshold is computed per-shard as the minimum affordable threshold across all Raft voters, clamped between 128 bytes (floor) and 64 KB (ceiling).

Observability

Tuning parameters (runtime)

The following parameters are set at runtime via the StorageAdminService gRPC API (SetTuningParams / GetTuningParams), not via environment variables. They are listed here for reference.

Cluster-wide tuning

Per-pool tuning

Default capacity thresholds by device class:

All tuning parameter changes via SetTuningParams are recorded in the cluster audit shard with parameter name, old value, new value, timestamp, and admin identity (I-A6).

Environment variable summary

Quick reference of all environment variables:

# Network
KISEKI_DATA_ADDR=0.0.0.0:9100
KISEKI_ADVISORY_ADDR=0.0.0.0:9101
KISEKI_S3_ADDR=0.0.0.0:9000
KISEKI_NFS_ADDR=0.0.0.0:2049
KISEKI_METRICS_ADDR=0.0.0.0:9090
KISEKI_RAFT_ADDR=0.0.0.0:9300

# Cluster
KISEKI_NODE_ID=1
KISEKI_RAFT_PEERS=1=node1:9300,2=node2:9300,3=node3:9300
KISEKI_BOOTSTRAP=false

# Storage
KISEKI_DATA_DIR=/var/lib/kiseki

# TLS
KISEKI_CA_PATH=/etc/kiseki/tls/ca.crt
KISEKI_CERT_PATH=/etc/kiseki/tls/server.crt
KISEKI_KEY_PATH=/etc/kiseki/tls/server.key
KISEKI_CRL_PATH=/etc/kiseki/tls/crl.pem

# Cache (client only)
KISEKI_CACHE_MODE=organic
KISEKI_CACHE_DIR=/var/cache/kiseki
KISEKI_CACHE_L1_MAX=1073741824
KISEKI_CACHE_L2_MAX=107374182400
KISEKI_CACHE_META_TTL_MS=5000

# Metadata capacity
KISEKI_META_SOFT_LIMIT_PCT=50
KISEKI_META_HARD_LIMIT_PCT=75

# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
OTEL_SERVICE_NAME=kiseki-server
RUST_LOG=kiseki=info
KISEKI_LOG_FORMAT=json

Cluster Management

This guide covers day-to-day cluster operations: adding and removing nodes, managing shards and pools, maintenance mode, and schema migration.

Node management

Kiseki uses Raft consensus groups for metadata and log replication. Adding or removing nodes is done through Raft membership changes, which are zero-downtime and zero-data-loss operations.

Adding a node

1. Deploy kiseki-server on the new host with a unique KISEKI_NODE_ID and the full KISEKI_RAFT_PEERS list (including the new node).

2. Start the service. The node registers with the cluster and begins receiving Raft log entries as a learner.

3. Promote the node to a voter once it has caught up:

   kiseki-server node add --node-id 4
   

4. The node receives shard assignments and begins participating in Raft elections and commit quorums.

Catch-up requirement (I-SF3): A learner must fully catch up with the leader’s committed index before being promoted to voter. The old voter remains in membership until the new voter is promoted.

Removing a node

1. Drain the node to migrate its shard assignments to other nodes:

   kiseki-server node drain --node-id 4
   

2. Wait for all shards to be migrated. The drain operation uses Raft membership changes (add learner on target, promote, demote source) for each shard hosted on the node.

3. Once drained, remove the node from the cluster:

   kiseki-server node remove --node-id 4
   

4. Stop the kiseki-server process and decommission the hardware.

Safety: Removing a node without draining first triggers automatic shard repair, but this is reactive rather than proactive. Always drain first for orderly removal.

Cluster sizing

• Minimum: 3 nodes (Raft requires a majority quorum; 2-of-3 for writes).
• Recommended: 5+ nodes for production. Tolerates 2 simultaneous node failures.
• Key manager: Deploy on a dedicated 3-5 node HA cluster, separate from storage nodes. The system key manager must be at least as available as the log (I-K12).

Shard management

Shards are the smallest unit of totally-ordered deltas, backed by one Raft group. They split automatically when size or throughput thresholds are exceeded (I-L6).

Viewing shard status

# List all shards
kiseki-server shard list

# Get details for a specific shard
kiseki-server shard info --shard-id shard-0001

# Check shard health
kiseki-server shard health --shard-id shard-0001

Automatic shard split

Shards have a hard ceiling triggering mandatory split (I-L6). The ceiling is configurable across three dimensions:

• Delta count: Maximum number of deltas in a shard.
• Byte size: Maximum total size of shard data.
• Write throughput: Maximum sustained write rate.

Any dimension exceeding its ceiling forces a split. The split operation:

1. Selects a split boundary (key range partition).
2. Creates a new shard for the upper range.
3. Continues accepting writes during the split (I-O1).
4. Notifies the control plane, views, and clients of the new shard topology.

Manual shard split

kiseki-server shard split --shard-id shard-0001 --boundary "..."

Shard maintenance mode

Set a shard to read-only for maintenance operations:

# Enable maintenance mode (writes rejected with retriable error)
kiseki-server shard maintenance --shard-id shard-0001 --enabled

During maintenance mode (I-O6):

• Write commands are rejected with a retriable error.
• Read operations continue normally.
• In-progress compaction and GC continue but no new triggers fire from write pressure.
• Shard splits do not initiate.

Cross-shard operations

Cross-shard rename returns EXDEV (I-L8). Shards are independent consensus domains with no two-phase commit. Applications must handle cross-shard moves via copy + delete.

Pool management

Affinity pools are groups of storage devices sharing a device class. Pools are the unit of capacity management and durability policy.

Viewing pools

# List all pools
kiseki-server pool list

# Get pool details including capacity and health
kiseki-server pool status --pool-id fast-nvme

Creating a pool

kiseki-server pool create --pool-id fast-nvme --device-class NvmeU2 \
  --ec-data 4 --ec-parity 2

Important: EC parameters (ec_data_chunks, ec_parity_chunks) are immutable per pool after creation (I-C6). Changing them requires creating a new pool and migrating data via ReencodePool.

Setting pool durability

# Switch pool durability strategy (applies to new chunks only)
kiseki-server pool set-durability --pool-id fast-nvme \
  --ec-data 4 --ec-parity 2

Existing chunks retain their original EC config. Re-encoding requires an explicit ReencodePool RPC.

Rebalancing a pool

Rebalance distributes data evenly across devices in a pool:

# Start rebalance
kiseki-server pool rebalance --pool-id fast-nvme

# Cancel a running rebalance
kiseki-server pool cancel-rebalance --pool-id fast-nvme

Rebalance runs at the configured rebalance_rate_mb_s (default 50 MB/s) to limit impact on production traffic.

Device evacuation

When a device shows signs of failure (SMART wear > 90% for SSD, > 100 bad sectors for HDD), automatic evacuation is triggered (I-D3). Evacuation can also be initiated manually:

# Start evacuation
kiseki-server device evacuate --device-id nvme-0001

# Cancel evacuation
kiseki-server device cancel-evacuation --device-id nvme-0001

Evacuation migrates all chunks from the device to other devices in the same pool. Device removal (RemoveDevice) is rejected unless the device state is Removed (post-evacuation) (I-D5).

Device state transitions: Healthy -> Degraded -> Evacuating -> Failed -> Removed. All transitions are recorded in the audit log (I-D2).

Pool capacity thresholds

Pool writes are rejected when the pool reaches the Critical threshold (I-C5). Thresholds vary by device class to account for SSD/NVMe GC pressure at high fill levels:

Pool redirection stays within the same device class only. ENOSPC is returned when the pool is Full.

Maintenance mode

Cluster-wide or per-shard maintenance mode sets the cluster (or specific shards) to read-only (I-O6).

Enabling cluster-wide maintenance

# Via the admin dashboard
curl -X POST http://node1:9090/ui/api/ops/maintenance \
  -H 'Content-Type: application/json' \
  -d '{"enabled": true}'

# Via the kiseki-server CLI
kiseki-server maintenance on

Maintenance mode behavior

• All write commands are rejected with a retriable error code (MaintenanceMode). Clients can retry after maintenance ends.
• Read operations continue normally.
• In-progress compaction and GC complete their current run.
• New shard splits, compaction triggers, and GC triggers from write pressure are suppressed.
• Maintenance mode is the prerequisite for:
  • Schema migration on upgrade
  • Inline threshold increase (optional migration of small chunked files back to inline)
  • Full cluster re-encryption

Disabling maintenance

curl -X POST http://node1:9090/ui/api/ops/maintenance \
  -H 'Content-Type: application/json' \
  -d '{"enabled": false}'

Writes resume immediately. Clients that were retrying will succeed on their next attempt.

Schema migration on upgrade

Kiseki uses versioned on-disk formats. Upgrades that change the schema follow this procedure:

1. Read the release notes for migration requirements. Not every release requires migration.

2. Enable maintenance mode on the cluster to prevent writes during migration.

3. Stop all nodes in the cluster.

4. Upgrade the binaries on all nodes (kiseki-server, kiseki-keyserver, kiseki-client-fuse).

5. Start nodes one at a time. On startup, each node detects the old schema version (via the superblock on each data device and the redb metadata version) and applies migration automatically.

6. Verify migration by checking the admin dashboard and node logs.

7. Disable maintenance mode to resume normal operations.

Rolling upgrades

For minor releases that do not change the on-disk format, rolling upgrades are supported:

1. Drain a node (DrainNode).
2. Stop the node.
3. Upgrade the binary.
4. Start the node.
5. Wait for it to rejoin and catch up.
6. Repeat for the next node.

The superblock on each data device carries a format version (ADR-029). Format version mismatches are detected at device open and handled by the migration path.

Admin Dashboard

Kiseki includes a built-in web dashboard for cluster monitoring and basic operations. The dashboard is served by every storage node on the metrics HTTP port.

Access

http://<node>:9090/ui

Any node in the cluster serves the full cluster-wide view. The dashboard scrapes metrics from peer nodes in the background and aggregates them locally. There is no dedicated dashboard server; connect to whichever node is most convenient.

The metrics HTTP server also serves:

Technology

The dashboard is a single-page HTML application using:

• HTMX for live updates via HTML fragment polling.
• Chart.js for time-series and per-node comparison charts.
• No build step, no JavaScript framework, no node_modules.

The dashboard HTML is embedded in the kiseki-server binary at compile time (include_str!). No external files to deploy or manage.

Overview tab

The main view shows six metric cards at the top, a time-series chart in the middle, and a node table at the bottom. All data refreshes automatically via HTMX polling.

Metric cards

Numbers are formatted with SI suffixes (K, M, B) and byte units (KB, MB, GB, TB) for readability.

Time-series charts

The dashboard stores up to 3 hours of metric history (configurable) in memory. Time-series charts show:

• Raft entries over time
• Gateway request rate
• Chunk write/read throughput
• Connection count

Historical data is available via the API:

# Get 3 hours of history (default)
curl http://node1:9090/ui/api/history

# Get 1 hour of history
curl http://node1:9090/ui/api/history?hours=1

Node table

A table listing every node in the cluster with per-node metrics:

Click a node row to drill down to the node detail view.

Performance tab

The performance tab shows per-node comparison charts for identifying hotspots and imbalances:

• Write throughput by node: Bar chart comparing chunk bytes written per node.
• Read throughput by node: Bar chart comparing chunk bytes read per node.
• Request count by node: Bar chart comparing gateway requests per node.

Chart data is sourced from the chart-data API:

curl http://node1:9090/ui/fragment/chart-data
# Returns: {"labels": [...], "writes": [...], "reads": [...], "requests": [...]}

Alerts tab

The alerts tab shows health status and capacity warnings. Each alert is a row with a colored dot (green, yellow, red, blue), a message, and a timestamp.

Alert types

Alerts are generated by comparing the current cluster state against expected conditions. The alert endpoint returns HTML fragments for HTMX polling:

curl http://node1:9090/ui/fragment/alerts

Operations tab

The operations tab provides buttons for common administrative actions. Each action calls a REST endpoint and records an event in the diagnostic event store.

Available operations

Example:

# Enable maintenance mode
curl -X POST http://node1:9090/ui/api/ops/maintenance \
  -H 'Content-Type: application/json' \
  -d '{"enabled": true}'

# Trigger a scrub
curl -X POST http://node1:9090/ui/api/ops/scrub

All operations return {"status": "ok", "message": "..."} on success.

Node drill-down

Click a node in the node table to see its detailed view. The drill-down shows:

• Node-specific metric history (time-series)
• Device health for devices attached to that node
• Shard assignments on that node
• Raft role (leader/follower/learner) per shard

API endpoints

All dashboard data is available via JSON APIs for scripting and integration:

Event log query parameters

Example:

# Get last 50 error events in the past hour
curl 'http://node1:9090/ui/api/events?severity=error&hours=1&limit=50'

Response format:

{
  "count": 2,
  "events": [
    {
      "timestamp": "2026-04-23T14:30:00Z",
      "severity": "error",
      "category": "device",
      "source": "nvme-0001",
      "message": "Device SMART wear exceeds 90%"
    }
  ]
}

Cluster-wide view architecture

Every node in the cluster runs the same dashboard. The cluster-wide view is assembled by scraping /metrics from peer nodes:

1. Each node knows its peers from KISEKI_RAFT_PEERS.
2. A background task scrapes each peer’s /metrics endpoint at a configurable interval (default 10 seconds).
3. Scraped metrics are cached locally in a MetricsAggregator.
4. Dashboard requests aggregate local + cached peer metrics.

This means:

• No single point of failure. Any node serves the dashboard.
• Stale data tolerance. If a peer is unreachable, the dashboard shows the last known state and marks the node as “Unreachable.”
• No additional infrastructure. No dedicated monitoring server is needed for basic cluster visibility.

For production monitoring with alerting and long-term retention, use Prometheus and Grafana (see Monitoring).

Backup & Recovery

Kiseki’s primary disaster recovery mechanism is federation (async replication to a secondary site). External backup is additive and optional, providing defense-in-depth for deployments that require it.

Architecture overview

Federation as primary DR

Federated-async replication to a secondary site is the recommended DR strategy (ADR-016). Properties:

• RPO: Bounded by async replication lag (seconds to minutes).
• RTO: Secondary site is warm (has replicated data + tenant config); switchover requires KMS connectivity and control plane reconfiguration.
• Data replication: Ciphertext-only. No key material in the replication stream.

What is replicated

External backup

Cluster admins can configure external backup targets (S3-compatible object store). Backup data is encrypted with the system key at rest.

Backup operations

Creating a backup

# Via the admin dashboard
curl -X POST http://node1:9090/ui/api/ops/backup

# Via the kiseki-server CLI
kiseki-server backup create

Backup contents

Each backup snapshot contains:

1. Per-shard metadata: Raft log snapshots for each shard, capturing the delta history up to the snapshot point.
2. Chunk extent manifests: The chunks/meta.redb index mapping chunk IDs to device extents.
3. Inline content: The small/objects.redb database (small-file data below the inline threshold).
4. Control plane state: Tenant configuration, namespace mappings, quotas, compliance tags, federation peer registry.
5. Key epoch metadata: Key epoch records from keys/epochs.redb (key material itself is NOT included in backups; it is managed by the system key manager and tenant KMS independently).

All backup data is encrypted. No plaintext chunk data appears in backup output. Backups reference chunk ciphertext on data devices by extent coordinates, not by copying the raw ciphertext (which would require reading and re-encrypting terabytes of data).

Listing backups

kiseki-server backup list

Deleting a backup

kiseki-server backup delete --backup-id backup-20260423-001

Retention policy

Backup retention is configurable per cluster. Defaults:

Retention is enforced by a background task that runs on the Raft leader. Deletion of expired backups is recorded in the cluster audit log.

Recovery procedures

Single node failure

Recovery path: Raft re-election + EC repair.

1. The Raft group detects the failed node and elects a new leader (if the failed node was leader).
2. EC repair automatically rebuilds chunk fragments that were on the failed node’s devices.
3. RPO: 0 (committed data is on a majority of replicas). RTO: seconds to minutes.

No manual intervention required. Monitor the repair progress via:

kiseki-server repair list

Multiple node failure (quorum maintained)

Recovery path: Raft reconfiguration + EC repair.

If the cluster still has a Raft majority (e.g., 2 of 3 nodes alive), recovery is automatic:

1. Raft continues operating with the surviving majority.
2. EC repair rebuilds lost chunk fragments.
3. Deploy replacement nodes and add them to the cluster.

Multiple node failure (quorum lost)

Recovery path: Manual Raft reconfiguration.

If the majority is lost (e.g., 2 of 3 nodes down), Raft cannot make progress. Recovery requires manual intervention:

1. Identify the surviving node(s) with the most recent committed state.
2. Force a new Raft configuration with the surviving node(s) as the initial voter set.
3. Deploy replacement nodes and add them as learners.
4. Promote learners to voters once they catch up.

Data loss risk: Deltas committed on the failed majority but not yet replicated to the surviving minority may be lost.

Full site failure (with federation)

Recovery path: Failover to federated peer.

1. Redirect clients to the secondary site (DNS, load balancer, or manual reconfiguration).
2. The secondary site has replicated chunk data, log deltas, and control plane config.
3. Tenant KMS must be reachable from the secondary site (same KMS serves both sites).
4. The secondary site’s system key manager has its own master keys, but tenant data is accessible because tenant KEKs come from the shared tenant KMS.

RPO: Replication lag. RTO: Minutes to hours (depends on control plane reconfiguration speed).

Full site failure (without federation)

Recovery path: Restore from external backup.

1. Deploy a new cluster.
2. Restore the backup snapshot to the new cluster.
3. The system key manager on the new cluster generates new system master keys.
4. Tenant KMS must be reconfigured to point to the new cluster.
5. Re-wrap all envelopes with new system master keys.

RPO: Time since last backup. RTO: Hours (depends on data volume).

Tenant KMS loss

Unrecoverable (I-K11). If the tenant loses their KMS and has no backup of their KEK material, all data encrypted under those keys is permanently unreadable. Kiseki documents this requirement but provides no system-side escrow. The tenant controls and is responsible for their keys.

Recovery summary

Limitations

• No point-in-time restore. Backups are snapshots, not continuous journals. Recovery restores the cluster to the state at the snapshot time. Deltas committed after the snapshot are lost unless federation has replicated them.

• Backup does not include key material. System master keys and tenant KEKs are managed by their respective key managers. Backup and recovery of key material is the responsibility of the key manager operator (cluster admin for system keys, tenant admin for tenant KEKs).

• Chunk ciphertext is referenced, not copied. Backup manifests reference chunk extents on data devices. If data devices are destroyed, the chunk ciphertext is lost. Federation replicates the actual ciphertext to a secondary site, which is why it is the primary DR mechanism.

• Cross-site backup requires federation. There is no built-in mechanism to ship backup snapshots to a remote site outside of the federation framework. For cross-site backup without federation, operators must arrange their own transport of backup snapshots.

Monitoring & Observability

Kiseki provides three observability pillars: metrics (Prometheus), structured logging (tracing), and distributed traces (OpenTelemetry). All three are tenant-aware, respecting the zero-trust boundary between cluster admin and tenant admin (ADR-015).

Prometheus metrics

Every kiseki-server node exposes Prometheus metrics in text exposition format on the metrics HTTP port.

Endpoint

GET http://<node>:9090/metrics

Registered metrics

Metric scoping (zero-trust)

Per ADR-015, metric scoping respects the zero-trust boundary:

• Cluster admin sees: Aggregated metrics, per-node metrics, system health. Per-tenant metrics are anonymized (tenant_id replaced with opaque hash) unless the cluster admin has approved access for that tenant.
• Tenant admin sees: Their own tenant’s metrics via the tenant audit export.
• No metric exposes: File names, directory structure, data content, or access patterns attributable to a specific tenant (without approval).

Metric cardinality

Metric cardinality is bounded by design. Label values are drawn from fixed sets (shard IDs, pool names, HTTP methods, strategy names). There are no unbounded label values such as file paths, tenant IDs, or user identifiers in metrics labels.

Structured logging

Kiseki uses the tracing crate for structured logging. Every log event is a structured record with typed fields.

Configuration

Filter examples

# Default: info-level for all Kiseki modules
RUST_LOG=kiseki=info

# Debug for the Raft subsystem, info for everything else
RUST_LOG=kiseki_raft=debug,kiseki=info

# Trace-level for the chunk subsystem (very verbose)
RUST_LOG=kiseki_chunk=trace,kiseki=info

# Warnings only (quiet)
RUST_LOG=warn

JSON output format

In production, set KISEKI_LOG_FORMAT=json for structured log aggregation (ELK, Loki, Datadog, etc.):

{
  "timestamp": "2026-04-23T14:30:00.123Z",
  "level": "INFO",
  "target": "kiseki_raft",
  "message": "Raft leader elected",
  "shard": "shard-0001",
  "node_id": 1,
  "term": 42
}

Log levels

Security in logs

• Tenant-identifying fields (tenant_id, namespace) are present for correlation.
• Content fields (file names, chunk plaintext, key material) are never logged (I-K8).
• Logs ship to the same audit/observability pipeline.

Distributed tracing (OpenTelemetry)

Kiseki uses OpenTelemetry for distributed tracing across the full write/read path.

Configuration

Trace propagation

Every write/read path carries a trace ID via OpenTelemetry context propagation. Traces span:

client -> gateway -> composition -> log -> chunk -> view

For the native client path:

client (FUSE) -> transport -> composition -> log -> chunk

Jaeger integration

The development Docker Compose stack includes Jaeger for trace visualization:

• Jaeger UI: http://localhost:16686
• OTLP gRPC receiver: localhost:4317

Trace scoping

Traces respect the zero-trust boundary:

• Tenant-scoped traces are visible only to the tenant admin (via tenant audit export).
• Cluster admin sees system-level spans. No tenant content appears in span attributes visible to the cluster admin.
• Trace overhead is approximately 1-2% on the data path (acceptable for production).

Event store

The admin dashboard maintains an in-memory event store for diagnostic events. Events are categorized and severity-tagged.

Event categories

Event severities

Event API

# All events from the last 3 hours
curl http://node1:9090/ui/api/events

# Errors from the last hour
curl 'http://node1:9090/ui/api/events?severity=error&hours=1'

# Device events, last 50
curl 'http://node1:9090/ui/api/events?category=device&limit=50'

# Security events from the last 24 hours
curl 'http://node1:9090/ui/api/events?category=security&hours=24'

Historical metrics API

# Metric snapshots from the last 3 hours
curl http://node1:9090/ui/api/history

# Last 6 hours
curl 'http://node1:9090/ui/api/history?hours=6'

The history endpoint returns time-series data points suitable for charting. The default retention is 3 hours in memory. For longer retention, use Prometheus.

Grafana integration

For production monitoring with alerting and long-term storage, configure Prometheus to scrape Kiseki metrics and visualize with Grafana.

Prometheus scrape configuration

scrape_configs:
  - job_name: 'kiseki'
    scrape_interval: 15s
    static_configs:
      - targets:
          - 'node1:9090'
          - 'node2:9090'
          - 'node3:9090'
    metrics_path: '/metrics'

Recommended Grafana dashboards

Cluster overview dashboard:

• Cluster health (up/down per node)
• Total Raft entries/sec (rate of kiseki_raft_entries_total)
• Gateway request rate (rate of kiseki_gateway_requests_total)
• Gateway latency p50/p99 (kiseki_gateway_request_duration_seconds)
• Pool utilization (kiseki_pool_capacity_used_bytes / kiseki_pool_capacity_total_bytes)

Per-node dashboard:

• Raft commit latency histogram (kiseki_raft_commit_latency_seconds)
• Chunk read/write throughput
• Transport connection count
• Shard delta count per shard

Capacity dashboard:

• Pool fill percentage over time
• Pool capacity trend (linear projection for capacity planning)
• Delta count growth rate (shard split prediction)

Key management dashboard:

• Key rotation count over time (kiseki_key_rotation_total)
• Crypto-shred count (kiseki_crypto_shred_total)

Alerting rules

Recommended Prometheus alerting rules:

groups:
  - name: kiseki
    rules:
      - alert: KisekiNodeDown
        expr: up{job="kiseki"} == 0
        for: 1m
        labels:
          severity: critical

      - alert: KisekiPoolCapacityWarning
        expr: >
          kiseki_pool_capacity_used_bytes / kiseki_pool_capacity_total_bytes > 0.85
        for: 5m
        labels:
          severity: warning

      - alert: KisekiPoolCapacityCritical
        expr: >
          kiseki_pool_capacity_used_bytes / kiseki_pool_capacity_total_bytes > 0.92
        for: 1m
        labels:
          severity: critical

      - alert: KisekiGatewayLatencyHigh
        expr: >
          histogram_quantile(0.99, rate(kiseki_gateway_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning

      - alert: KisekiRaftCommitLatencyHigh
        expr: >
          histogram_quantile(0.99, rate(kiseki_raft_commit_latency_seconds_bucket[5m])) > 0.1
        for: 5m
        labels:
          severity: warning

Key Management

Kiseki uses a two-layer encryption model where system-level encryption protects data at rest and tenant-level key wrapping controls access. This page covers operational aspects of key management: rotation, re-encryption, crypto-shred, and external KMS integration.

Encryption model

Kiseki implements Model (C) from ADR-002: single data encryption pass at the system layer, with tenant access via key wrapping. No double encryption.

Plaintext chunk
  |
  v
System DEK (AES-256-GCM)  -->  Ciphertext (stored on disk)
  |
  v
System KEK (wraps DEK derivation material)
  |
  v
Tenant KEK (wraps system DEK derivation parameters per tenant)

System keys

• System DEK: Per-chunk symmetric key derived locally on each storage node via HKDF-SHA256 (ADR-003). Never stored, never transmitted. Derivation: HKDF(master_key[epoch], chunk_id, "kiseki-chunk-dek-v1").
• System master key: Per-epoch master key stored in the system key manager (kiseki-keyserver). Storage nodes fetch it at startup and on epoch rotation, then derive per-chunk DEKs locally. The key manager never sees individual chunk IDs.
• System KEK: Wraps system master keys. Managed by the cluster admin.

Tenant keys

• Tenant KEK: Key wrapping key managed by the tenant’s chosen KMS backend. Wraps access to system DEK derivation parameters (epoch + chunk_id). Destroying the tenant KEK = crypto-shred (data becomes unreadable).
• No Tenant DEK: Model (C) does not double-encrypt. The tenant layer is key-wrapping, not data-encryption.

Invariants

• I-K1: No plaintext chunk is ever persisted to storage.
• I-K2: No plaintext payload is ever sent on the wire.
• I-K7: Authenticated encryption (AES-256-GCM) everywhere.
• I-K8: Keys are never logged, printed, transmitted in the clear, or stored in configuration files.

System key manager

The system key manager (kiseki-keyserver) is a dedicated HA service backed by its own Raft consensus group.

Deployment

Deploy on 3-5 dedicated nodes, separate from storage nodes. The system key manager must be at least as available as the log (I-K12) because its unavailability blocks all chunk writes cluster-wide.

Key distribution

kiseki-keyserver:
  Stores: master_key per epoch (Raft-replicated)
  Serves: master_key to authenticated kiseki-server processes (mTLS)
  Never sees: individual chunk_ids or per-chunk operations

kiseki-server:
  Caches: master_key (mlock'd, MADV_DONTDUMP, seccomp)
  Derives: per-chunk DEK = HKDF(master_key, chunk_id) -- locally
  Never sends: chunk_ids to the key manager

This design prevents the key manager from building an index of all chunk IDs, which would leak per-tenant access patterns.

Key rotation

System key rotation

System key rotation creates a new epoch with a new master key. The rotation process:

1. Cluster admin initiates rotation via RotateSystemKey().
2. The key manager generates a new master key and assigns a new epoch.
3. Storage nodes are notified and fetch the new master key.
4. New writes use the new epoch. Old data retains its epoch.
5. Two epochs coexist during the rotation window (I-K6).

Old master keys are retained until all data encrypted under them has been re-encrypted or deleted. Full re-encryption is available as an explicit admin action.

Tenant key rotation

Tenant key rotation creates a new epoch for the tenant’s KEK:

1. Tenant admin initiates rotation via RotateTenantKey(tenant).
2. The tenant KMS generates or rotates the key (provider-specific).
3. New envelope wrappings use the new epoch.
4. Old wrapped material remains valid until background re-wrapping completes.

Background re-encryption

A background monitor detects envelopes wrapped under old epochs and schedules re-wrapping. The rewrap worker:

1. Reads envelopes with old-epoch tenant wrapping.
2. Unwraps with old KEK.
3. Re-wraps with current KEK.
4. Writes the updated envelope.

For providers that support server-side rewrap (e.g., Vault Transit), the rewrap operation never exposes plaintext derivation material to the storage node.

Crypto-shred

Crypto-shred is the authoritative deletion mechanism in Kiseki. Destroying the tenant KEK renders all tenant data unreadable.

Process

1. Tenant admin initiates via CryptoShred(tenant).
2. The tenant KMS destroys the KEK (provider-specific: Vault key deletion, AWS KMS key scheduling, PKCS#11 key destruction).
3. All cached key material for the tenant is invalidated across the cluster.
4. Native clients detect the shred via key health checks (default every 30 seconds) and wipe their caches (I-CC12).

What happens after crypto-shred

• Data is semantically deleted: No component can decrypt the tenant’s data because the KEK is destroyed.
• Ciphertext remains on disk: Physical GC runs separately when chunk refcount = 0 AND no retention hold is active (I-C2b).
• Audit trail preserved: Crypto-shred events are recorded in the audit log.

Ordering requirement

If retention holds are needed, they must be set before crypto-shred:

Set retention hold -> Crypto-shred -> Hold expires -> GC eligible

This prevents a race between crypto-shred and GC (I-C2b).

Detection latency

Crypto-shred detection is bounded by: min(key_health_interval, max_disconnect_seconds).

Default key health check interval: 30 seconds. Configurable per tenant within [5s, 300s], default 60s (I-K15).

External KMS providers (ADR-028)

Kiseki supports five tenant KMS backends via the TenantKmsProvider trait. The provider is selected per-tenant at onboarding.

Provider comparison

Provider invariants

• I-K16: Provider abstraction is opaque to callers. No correctness decision depends on which backend is selected.
• I-K17: Wrap/unwrap operations include AAD (chunk_id) binding. A wrapped blob cannot be spliced from one envelope to another.
• I-K18: Provider is validated on configuration: connectivity test, wrap/unwrap round-trip, certificate chain. Validation failure prevents tenant activation.
• I-K19: Internal provider stores tenant KEKs in a separate Raft group from system master keys.
• I-K20: Provider migration (e.g., Internal to Vault) requires re-wrapping all existing envelopes. Migration is background, audited, and preserves data availability throughout.

Provider 1: Kiseki Internal (default)

Zero-configuration default. Kiseki manages tenant KEKs internally in a Raft group separate from system master keys. Suitable for single-operator deployments.

Security trade-off: Internal mode does not provide the full two-layer security guarantee. Compromise of both the system key store and the tenant key store yields full access. Compliance-sensitive tenants should use an external provider.

Provider 2: HashiCorp Vault

Uses Vault’s Transit secrets engine for encryption-as-a-service:

Provider 3: KMIP 2.1

Standards-based integration with enterprise KMS and HSM appliances. Uses mTLS over TTLV binary protocol.

Provider 4: AWS KMS

Cloud-native KMS integration. Key material never leaves AWS. All wrap/unwrap operations are remote HTTPS calls. Suitable for hybrid cloud deployments.

Provider 5: PKCS#11 v3.0

Direct HSM integration via the PKCS#11 C API (FFI). Key material stays in the HSM. Highest security level, requires HSM hardware on or accessible from storage nodes.

OIDC integration

Tenant identity providers can be integrated for second-stage authentication (I-Auth2). This is optional and orthogonal to the KMS provider choice.

When configured, workload-level identity is validated against the tenant admin’s authorization via OIDC/JWT tokens, providing “authorized by my tenant admin” on top of the mTLS-based “belongs to this cluster” identity.

Keycloak is included in the development Docker Compose stack for OIDC testing.

Operational checklist

Key rotation schedule

Monitoring key health

# Check key manager health
kiseki-server keymanager health

# Check tenant KMS connectivity
kiseki-server keymanager check-kms

# Monitor key rotation metrics
curl -s http://node1:9090/metrics | grep kiseki_key_rotation_total

# Monitor crypto-shred events
curl -s http://node1:9090/metrics | grep kiseki_crypto_shred_total

Key material security

• Master keys are mlock’d in memory on storage nodes (prevent swapping).
• Core dumps are disabled (LimitCORE=0 in systemd, MADV_DONTDUMP).
• seccomp filters restrict system calls on key-handling threads.
• Runtime integrity monitor detects ptrace, /proc/pid/mem access, and debugger attachment (I-O7).
• Keys are zeroized on deallocation (Zeroizing<Vec<u8>>).

System Overview

Kiseki is a distributed storage system designed for HPC and AI workloads. It provides a unified data fabric with POSIX (FUSE), NFS, and S3 access paths, two-layer encryption with tenant-controlled crypto-shred, and pluggable HPC transports (CXI/Slingshot, InfiniBand, RoCEv2).

Workspace structure

The codebase is a single Rust workspace with 18 crates:

Bounded contexts

The domain is organized into eight bounded contexts, each with a distinct responsibility, failure domain, and scaling concern:

1. Log – Delta ordering, Raft consensus, shard lifecycle
2. Chunk Storage – Encrypted chunk persistence, placement, EC, GC
3. Composition – Tenant-scoped metadata assembly, namespace management
4. View Materialization – Protocol-shaped materialized projections
5. Protocol Gateway – NFS and S3 wire protocol translation
6. Control Plane – Tenancy, IAM, quota, policy, federation
7. Key Management – System DEK/KEK, tenant KMS providers, crypto-shred
8. Workflow Advisory – Client hints, telemetry feedback (cross-cutting)

Additionally, Native Client runs on compute nodes as a separate trust boundary and Block I/O handles raw device management underneath chunk storage.

Data path

Client (plaintext) ──encrypt──► Gateway / Native Client
                                       │
                                       ▼
                                  Composition
                                  (assemble chunks, record delta)
                                       │
                              ┌────────┴────────┐
                              ▼                 ▼
                          Log (Raft)       Chunk Storage
                     (commit delta,      (write encrypted
                      replicate)          chunk to device)

Write path: The client (native or protocol) encrypts data with the tenant KEK wrapping a system DEK. The composition layer assembles chunk references and records a delta. The delta is committed through Raft on the owning shard. Chunks are written to affinity pools with erasure coding.

Read path: The client issues a view lookup (materialized from log deltas). The view resolves chunk references. Chunks are read from devices, decrypted, and returned to the client.

Control path

Admin ──► Control Plane (gRPC)
              │
              ├── Tenant / Namespace / Quota / Policy
              ├── Flavor management
              ├── Federation (async cross-site)
              └── Advisory policy (hint budgets, profiles)

The control plane manages tenant lifecycle, IAM, quotas, compliance tags, placement policy, and federation. It communicates with storage nodes via gRPC on the management network. The control plane depends only on kiseki-common and kiseki-proto (crate-graph firewall, ADR-027).

Advisory path (ADR-020)

Client ──hints──► Advisory Runtime ──telemetry──► Client
                      │
                      ├── Route hints to Chunk / View / Composition
                      ├── Emit caller-scoped telemetry feedback
                      └── Audit advisory events

The workflow advisory system is a cross-cutting concern (not a bounded context). It carries two flows over a bidirectional gRPC channel per declared workflow:

• Hints (client to storage): advisory steering signals for prefetch, affinity, priority, and phase-adaptive tuning. Never authoritative (I-WA1).
• Telemetry feedback (storage to client): caller-scoped signals about backpressure, locality, materialization lag, and QoS headroom (I-WA5).

The advisory runtime runs on a dedicated tokio runtime, isolated from the data path. Advisory failures never block data-path operations (I-WA2).

Network ports

Binaries

Bounded Contexts

Eight bounded contexts form the core domain model. Each has a distinct responsibility, failure domain, and scaling concern. This page describes each context’s purpose, implementing crate, key types, and governing invariants.

1. Log

Crate: kiseki-log

Purpose: Accept deltas, assign them a total order within a shard, replicate via Raft, persist durably, and support range reads for view materialization and replay.

Key types: Delta, DeltaEnvelope, Shard, ShardConfig, ShardInfo

Key invariants:

Failure domain: Per-shard. Leader loss causes transient latency (election). Quorum loss makes the shard unavailable.

2. Chunk Storage

Crate: kiseki-chunk (with kiseki-block for device I/O)

Purpose: Store and retrieve opaque encrypted chunks. Manage placement across affinity pools. Handle erasure coding and replication. Run GC based on refcounts and retention holds.

Key types: Chunk, ChunkId, Envelope, AffinityPool, DeviceBackend

Key invariants:

Failure domain: Per-chunk or per-device. Chunk loss recoverable via EC parity or replicas.

3. Composition

Crate: kiseki-composition

Purpose: Maintain tenant-scoped metadata structures describing how chunks assemble into data units (files, objects). Manage namespaces. Record mutations as deltas in the log.

Key types: Composition, Namespace, CompositionMutation

Key invariants:

Failure domain: Coupled to Log. If a shard fails, its compositions are affected.

4. View Materialization

Crate: kiseki-view

Purpose: Consume deltas from shards and maintain materialized views per view descriptor. Handle view lifecycle (create, discard, rebuild) and MVCC read pins.

Key types: View, ViewDescriptor, StreamProcessor, MvccPin

Key invariants:

Failure domain: Per-view. A fallen-behind view serves stale data. A lost view can be rebuilt from the log.

5. Protocol Gateway

Crate: kiseki-gateway

Purpose: Translate wire protocol requests (NFS, S3) into operations against views and the log. Serve reads from views. Route writes as deltas to the log via composition. Perform tenant-layer encryption for protocol-path clients.

Key types: Protocol gateway instance, protocol plugin

Trust boundary: NFS/S3 clients send plaintext over TLS to the gateway. The gateway encrypts before writing to log/chunks. Plaintext exists in gateway memory only ephemerally.

Failure domain: Per-gateway. Crash disconnects affected clients. Restart and client reconnect recovers.

6. Control Plane

Crate: kiseki-control

Purpose: Declarative API for tenancy, IAM, policy, placement, discovery, compliance tagging, and federation. Manages cluster-level and tenant-level configuration.

Key types: Organization, Project, Workload, Flavor, ComplianceRegime, RetentionHold, FederationPeer

Key invariants:

Failure domain: Control plane unavailability prevents new tenant creation and policy changes, but the existing data path continues with last-known configuration.

7. Key Management

Crates: kiseki-keymanager, kiseki-crypto

Purpose: Custody, rotation, escrow, and issuance of all key material. Two layers: system keys (cluster admin) and tenant key wrapping (tenant admin via tenant KMS). Orchestrate crypto-shred.

Key types: SystemDek, SystemKek, TenantKek, KeyEpoch, Envelope, TenantKmsProvider

Tenant KMS providers (ADR-028): Five pluggable backends implementing the TenantKmsProvider trait – Kiseki-Internal, HashiCorp Vault, KMIP 2.1, AWS KMS, and PKCS#11.

Key invariants:

Failure domain: KMS unavailability blocks new encrypt/decrypt operations. This context’s availability is as critical as the Log’s.

8. Workflow Advisory (cross-cutting)

Crate: kiseki-advisory

Purpose: Carry workflow hints from clients to storage and telemetry feedback from storage back to clients. Route advisory signals to the bounded context best able to act on them.

Key types: WorkflowRef, OperationAdvisory, PoolHandle, PoolDescriptor, HintBudget

Key invariants:

Runtime isolation: The advisory runtime runs on a dedicated tokio runtime separate from the data-path runtime (ADR-021). No data-path crate depends on kiseki-advisory.

Cross-context relationships

Data Flow

This page describes the write, read, inline, and cross-node data paths through the Kiseki system.

Write path

┌──────────┐    plaintext     ┌──────────────────┐
│  Client  │ ──────────────► │  Gateway /        │
│          │   (over TLS)    │  Native Client    │
└──────────┘                 └────────┬──────────┘
                                      │ 1. Encrypt with tenant KEK
                                      │    wrapping system DEK
                                      │ 2. Content-defined chunking
                                      │    (Rabin fingerprinting)
                                      ▼
                             ┌──────────────────┐
                             │   Composition    │
                             │                  │
                             │ 3. Record chunk  │
                             │    references    │
                             │ 4. Build delta   │
                             └───────┬──────────┘
                                     │
                            ┌────────┴────────┐
                            ▼                 ▼
                   ┌──────────────┐   ┌──────────────┐
                   │  Log (Raft)  │   │ Chunk Storage │
                   │              │   │               │
                   │ 5. Commit    │   │ 6. Write      │
                   │    delta via │   │    encrypted  │
                   │    Raft      │   │    chunk to   │
                   │ 7. Replicate │   │    device     │
                   │    to        │   │ 8. EC encode  │
                   │    majority  │   │    across     │
                   │              │   │    pool       │
                   └──────────────┘   └──────────────┘

Step-by-step

1. Client encrypt: The native client encrypts data before it leaves the process. Protocol-path clients (NFS/S3) send plaintext over TLS to the gateway, which encrypts on their behalf.

2. Content-defined chunking: Data is split into variable-size chunks using Rabin fingerprinting. Each chunk gets a content-addressed ID (SHA-256 hash of plaintext, or HMAC when tenant opts out of cross-tenant dedup).

3. Compose: The composition layer records chunk references and constructs a delta describing the mutation (create, update, delete).

4. Raft commit: The delta is appended to the owning shard’s Raft log. The leader replicates to a majority of voters before acknowledging.

5. Chunk write: Encrypted chunks are written to affinity pool devices with erasure coding (or N-copy replication, per pool policy).

6. Ack: The write is acknowledged to the client only after the delta is committed (I-L2) and all referenced chunks are durable (I-L5).

Read path

┌──────────┐                 ┌──────────────────┐
│  Client  │ ◄────────────── │  Gateway /        │
│          │   plaintext     │  Native Client    │
└──────────┘   (over TLS)   └────────┬──────────┘
                                      ▲ 5. Decrypt
                                      │
                             ┌────────┴──────────┐
                             │   View Lookup     │
                             │                   │
                             │ 1. Resolve path   │
                             │    to composition │
                             │ 2. Get chunk list │
                             └────────┬──────────┘
                                      │
                                      ▼
                             ┌──────────────────┐
                             │  Chunk Storage   │
                             │                  │
                             │ 3. Read chunks   │
                             │    from device   │
                             │ 4. EC decode if  │
                             │    degraded      │
                             └──────────────────┘

Step-by-step

1. View lookup: The client or gateway queries a materialized view to resolve a path (POSIX) or key (S3) to a composition and its chunk list.

2. Chunk read: Encrypted chunks are read from the storage devices. If a device is degraded, EC parity reconstructs the missing data.

3. Decrypt: The client (native path) or gateway (protocol path) unwraps the system DEK using the tenant KEK, then decrypts the chunk data with AES-256-GCM.

4. Return: Plaintext is returned to the client.

Inline path (ADR-030)

Small files below the configurable inline threshold bypass chunk storage entirely:

Client ──► Composition ──► Log (Raft)
                             │
                             ▼
                    Delta with inline payload
                             │
                             ▼
                    Raft replication to voters
                             │
                             ▼
                    State machine apply:
                    store in small/objects.redb

Threshold computation: The inline threshold for a shard is the minimum affordable threshold across all nodes hosting that shard’s voter set:

clamp(min(voter_budgets) / file_count_estimate, INLINE_FLOOR, INLINE_CEILING)

Key invariants:

• I-L9: Inlined payloads are immutable after write; threshold changes apply prospectively only
• I-SF5: Inline content is offloaded to small/objects.redb on state machine apply; snapshots include inline content from redb
• I-SF7: Per-shard Raft inline throughput capped at KISEKI_RAFT_INLINE_MBPS (default 10 MB/s)

Cross-node data paths

Raft replication

Each shard runs an independent Raft group (ADR-026). The leader replicates log entries (deltas) to followers via the Raft RPC transport. Replication uses mTLS on the data fabric.

Leader ──► Follower 1 (AppendEntries)
       ──► Follower 2 (AppendEntries)
       ──► Follower 3 (AppendEntries)

Committed entries are persisted in RedbRaftLogStore on each voter.

Snapshot transfer

When a follower is too far behind or a new voter joins, the leader sends a full snapshot. Snapshots are transferred as length-prefixed JSON over the Raft transport connection.

For shards with inline data, the snapshot includes all entries from small/objects.redb (I-SF5).

Chunk replication and EC

Chunks are placed across distinct physical devices within a pool using deterministic hashing (CRUSH-like). No two EC fragments of the same chunk reside on the same device (I-D4).

Device failure triggers automatic repair from EC parity or replicas (I-D1).

Federation

Federated sites replicate data asynchronously. Only ciphertext is replicated – no key material in the replication stream (I-CS3). All federated sites for a tenant connect to the same tenant KMS.

Encryption Model

Kiseki uses a two-layer encryption architecture (ADR-002, model C) that separates data encryption from access control. One encryption pass protects data; key wrapping controls who can read it.

Two-layer architecture

┌─────────────────────────────────────────────────┐
│              Tenant Layer (access)              │
│                                                 │
│  Tenant KEK (controlled by tenant admin)        │
│  wraps the system DEK for tenant-scoped access  │
│                                                 │
│  Destroying the tenant KEK = crypto-shred       │
│  (all tenant data rendered unreadable)           │
├─────────────────────────────────────────────────┤
│              System Layer (data)                │
│                                                 │
│  System DEK encrypts chunk data (AES-256-GCM)   │
│  System KEK wraps system DEKs                    │
│  Always on -- no unencrypted chunks              │
└─────────────────────────────────────────────────┘

System layer: The system DEK encrypts every chunk using AES-256-GCM. System DEKs are derived per-chunk using HKDF-SHA256 from a master key (ADR-003). The system KEK wraps system DEKs and is managed by the cluster admin via the system key manager.

Tenant layer: The tenant KEK wraps the system DEK for tenant-scoped access control. There is no double encryption – one data encryption pass, with key wrapping for access control. The tenant admin controls the tenant KEK via the tenant KMS.

Envelope structure

Each chunk is stored as an envelope containing:

┌──────────────────────────────────────────┐
│  Envelope                                │
│                                          │
│  ┌──────────────────────────────────┐    │
│  │  Ciphertext (AES-256-GCM)       │    │
│  │  (encrypted chunk data)          │    │
│  └──────────────────────────────────┘    │
│                                          │
│  auth_tag (16 bytes, GCM tag)            │
│  nonce (12 bytes, unique per chunk)      │
│  system_key_epoch (current epoch)        │
│  tenant_key_epoch (current epoch)        │
│  chunk_id (content-addressed)            │
│  algorithm_id (for crypto-agility)       │
│                                          │
│  System wrapping metadata                │
│  Tenant wrapping metadata                │
└──────────────────────────────────────────┘

The envelope carries algorithm identifiers for crypto-agility (I-K7). All metadata is authenticated – unauthenticated encryption is never acceptable.

Key derivation

System DEKs are derived locally on each storage node using HKDF-SHA256 (ADR-003). No DEK-per-chunk RPC is required:

system_dek = HKDF-SHA256(
    ikm  = master_key[epoch],
    salt = chunk_id,
    info = "kiseki-chunk-dek-v1"
)

The master key is fetched from the system key manager at startup and on rotation events. DEK derivation is deterministic – the same chunk ID and epoch always produce the same DEK.

Key rotation

Key rotation is epoch-based (I-K6):

1. The admin triggers rotation (system or tenant level)
2. A new epoch is created with fresh key material
3. New data is encrypted with the current epoch’s keys
4. Old data retains its epoch until background re-encryption migrates it
5. Two epochs coexist during the rotation window
6. Full re-encryption available as an explicit admin action for key-compromise incidents

Crypto-shred

Destroying the tenant KEK renders all tenant data unreadable (I-K5):

1. Set retention hold (if compliance requires)
2. Destroy tenant KEK at tenant KMS
3. All wrapped system DEKs for this tenant become unwrappable
4. Chunk ciphertext remains on disk (system-encrypted) until GC
5. Physical GC runs separately when refcount = 0 AND no retention hold

The ordering contract (I-C2b): set hold before crypto-shred to prevent race with GC.

Client-side detection: periodic key health check (default 30s) detects KEK_DESTROYED and triggers immediate cache wipe (I-CC12). Maximum detection latency: min(key_health_interval, max_disconnect_seconds).

Chunk ID derivation

When a tenant opts out of cross-tenant dedup (I-X2, I-K10), chunk IDs are derived using HMAC with a tenant-specific key, making it impossible to determine whether two tenants store the same data.

Tenant KMS providers (ADR-028)

Five pluggable backends implement the TenantKmsProvider trait:

Provider selection is per-tenant at onboarding. The trait fully encapsulates protocol differences – callers never branch on provider type (I-K16). Wrap/unwrap operations include AAD (chunk_id) binding to prevent envelope splicing (I-K17).

FIPS compliance

Kiseki uses aws-lc-rs with the FIPS feature flag for FIPS 140-2/3 validated cryptographic operations. The kiseki-crypto crate provides:

• AES-256-GCM authenticated encryption
• HKDF-SHA256 key derivation
• SHA-256 hashing
• HMAC-SHA256 for opted-out chunk ID derivation
• zeroize integration for all key material in memory

Delta encryption

Log delta payloads (filenames, attributes, inline data) are encrypted with the system DEK, wrapped with the tenant KEK (I-K3). The delta envelope has structurally separated:

• System-visible header (cleartext or system-encrypted): sequence number, shard ID, hashed_key, operation type, timestamp
• Tenant-encrypted payload: the actual mutation data

Compaction operates on headers only and never decrypts tenant-encrypted payloads (I-O2).

Raft Consensus

Kiseki uses Raft for ordering and replicating deltas within each shard. The implementation is based on openraft 0.10 with a custom TCP transport and redb-backed persistent storage.

Per-shard Raft groups

Each shard runs an independent Raft group (ADR-026, Strategy A). This provides:

• Independent scaling: shard count grows with data volume and throughput
• Isolated failure domains: quorum loss in one shard does not affect others
• No cross-shard coordination: cross-shard rename returns EXDEV (I-L8)

The system key manager also runs its own Raft group for high availability (ADR-007), as do audit log shards (ADR-009).

openraft integration

The kiseki-raft crate defines KisekiTypeConfig used by all Raft groups:

• Node identity: u64 node IDs
• Async runtime: tokio
• Log store: RedbRaftLogStore (persistent) or MemLogStore (testing)
• Entry format: customized per context (log deltas, key manager ops, audit events)

Each context (log, key manager, audit) defines its own request (D) and response (R) types while sharing the node identity, entry format, and async runtime configuration.

Persistent log: RedbRaftLogStore

Raft log entries are persisted using redb (ADR-022), a pure-Rust embedded key-value store. The RedbRaftLogStore provides:

• Durable append and truncation of log entries
• Vote persistence (current term, voted-for)
• Snapshot metadata storage
• Crash-safe operations (redb uses write-ahead logging internally)

For shards with inline data (ADR-030), the state machine offloads inline content to small/objects.redb on apply. The in-memory state machine does not hold inline content after apply (I-SF5).

Snapshot transfer

When a follower falls behind or a new voter joins the group, the leader sends a full snapshot:

1. Leader serializes the current state machine as length-prefixed JSON
2. For shards with inline data, the snapshot includes all entries from small/objects.redb
3. The snapshot is streamed over the Raft transport connection
4. The follower installs the snapshot and resumes normal replication

Transport and security

Raft RPCs use a custom TCP transport with mTLS:

• All Raft communication is authenticated via per-node mTLS certificates signed by the Cluster CA (I-Auth1)
• The transport runs on the data fabric (not the management network)
• Connection pooling and keepalive are managed by the transport layer

The Raft transport address is configured via KISEKI_RAFT_ADDR.

Dynamic membership changes

Raft membership changes follow the standard joint-consensus protocol:

1. Add voter: new node starts as learner, catches up to committed index, then promoted to voter
2. Remove voter: validated that removal does not break quorum (safety check via can_remove_safely)
3. Shard migration: target node must fully catch up (learner state matches leader’s committed index) before old voter is removed (I-SF3)

Membership changes are validated by validate_membership_change in kiseki-raft, which checks quorum preservation and prevents unsafe removal.

Shard lifecycle

Shard splits do not block writes to the existing shard during the split operation (I-O1).

Consistency guarantees

Transport Layer

The kiseki-transport crate provides a pluggable transport abstraction for bidirectional byte-stream connections. It ships with a TCP+TLS reference implementation and feature-flagged support for HPC fabric transports.

Transport trait

The Transport trait is the core abstraction:

#![allow(unused)]
fn main() {
pub trait Transport: Send + Sync + 'static {
    type Connection: Connection;
    async fn connect(&self, addr: SocketAddr) -> Result<Self::Connection>;
    async fn listen(&self, addr: SocketAddr) -> Result<Listener>;
}

pub trait Connection: AsyncRead + AsyncWrite + Send + Unpin + 'static {
    fn peer_identity(&self) -> Option<&PeerIdentity>;
}
}

All components (client, server, Raft) use this trait, enabling transport selection without code changes.

TCP+TLS (reference implementation)

The TcpTlsTransport is always available and serves as the universal fallback:

• mTLS: Cluster CA validation with per-tenant certificates (I-Auth1, I-K13)
• SPIFFE: SAN-based SVID validation for workload identity (I-Auth3)
• CRL: Optional certificate revocation list support via KISEKI_CRL_PATH
• Connection pooling: Configurable pool size per peer
• Keepalive: TCP keepalive for connection health
• Timeouts: Configurable connect, read, and write timeouts

Configuration: TlsConfig with CA cert, node cert, node key, and optional CRL path.

RDMA verbs (feature: verbs)

Native InfiniBand and RoCEv2 support for low-latency HPC fabrics:

• InfiniBand: Direct RDMA over InfiniBand fabric (VerbsIb)
• RoCEv2: RDMA over Converged Ethernet (VerbsRoce)
• Device selection: Auto-detects the first available IB device, or uses the device named in KISEKI_IB_DEVICE
• Zero-copy: RDMA read/write for chunk data transfer

The verbs module uses unsafe code for FFI calls to libibverbs. Each unsafe block has a per-block SAFETY comment.

CXI/libfabric (feature: cxi)

HPE Slingshot fabric support via libfabric:

• CXI provider: Lowest-latency transport on Slingshot-equipped systems
• libfabric: Uses the libfabric API (fi_* calls) for fabric operations
• Feature-flagged: Only compiled when cxi feature is enabled

The CXI module uses unsafe code for FFI calls to libfabric.

FabricSelector

The FabricSelector provides priority-based transport selection with automatic failover:

Priority 0: CXI        (Slingshot, lowest latency)
Priority 1: VerbsIb    (InfiniBand)
Priority 2: VerbsRoce  (RoCEv2)
Priority 3: TcpTls     (always available, universal fallback)

At boot, the selector probes for available transports (hardware presence check). On connection, it selects the highest-priority available transport. On failure, it falls back to the next-best transport.

The TransportHealthTracker monitors transport health and marks transports as unhealthy after repeated failures, temporarily removing them from selection until they recover.

GPU-direct (planned)

Future support for direct GPU memory access:

• NVIDIA cuFile (feature: gpu-cuda): GPUDirect Storage for direct NVMe-to-GPU data transfer
• AMD ROCm (feature: gpu-rocm): ROCm-based GPU direct access

These features bypass CPU memory for chunk data, reducing latency for AI training workloads.

NUMA-aware thread pinning

The NumaTopology module provides NUMA-aware thread pinning for optimal memory locality:

• Auto-detects NUMA topology on Linux via sched_setaffinity
• Pins I/O threads to the NUMA node closest to the network device
• Reduces cross-NUMA memory access latency for high-throughput workloads

Metrics and health

The transport layer exports Prometheus metrics via TransportMetrics:

• Connection count per transport type
• Bytes sent/received per transport
• Connection errors and failover events
• Latency histograms per transport

Health tracking (TransportHealthTracker) provides per-transport health status for the selector’s failover decisions.

Invariant mapping

Client-Side Cache (ADR-031)

The native client (kiseki-client) includes a two-tier read-only cache of decrypted plaintext chunks. The cache is a performance feature, not a correctness mechanism – it is ephemeral and wiped on process restart or extended disconnect.

Architecture

┌────────────────────────────────────────────┐
│            kiseki-client process           │
│                                            │
│  ┌──────────────────────────────────────┐  │
│  │  L1: In-memory cache               │  │
│  │  Zeroizing<Vec<u8>> entries         │  │
│  │  Content-addressed by ChunkId       │  │
│  └──────────────┬───────────────────────┘  │
│                 │ miss                      │
│  ┌──────────────▼───────────────────────┐  │
│  │  L2: Local NVMe cache pool          │  │
│  │  CRC32 integrity per entry          │  │
│  │  Per-process, per-tenant isolation   │  │
│  └──────────────┬───────────────────────┘  │
│                 │ miss                      │
│                 ▼                           │
│         Fetch from canonical               │
│         (verify by ChunkId SHA-256)        │
└────────────────────────────────────────────┘

L1 (in-memory): Fast access to recently-used chunks. Entries use Zeroizing<Vec<u8>> so plaintext is overwritten with zeros on eviction or deallocation (I-CC2).

L2 (local NVMe): Larger cache on local storage. Each entry has a CRC32 checksum trailer computed at insert time (I-CC13). On read, the CRC32 is verified before serving; mismatch triggers bypass to canonical and entry deletion (I-CC7).

Cache modes

Three modes are available per client session (selected at session establishment):

Mode is per session, not per file. The admin controls which modes are available for each workload.

Staging API

Staging pre-fetches a dataset’s chunks into the L2 cache with pinned retention:

kiseki-client stage --dataset /path/to/data

1. Takes a namespace path and recursively enumerates compositions
2. Fetches and verifies all chunks from canonical (SHA-256 match)
3. Stores chunks in L2 with pinned retention
4. Produces a manifest file listing staged compositions and chunk IDs

Staging is idempotent and resumable. Limits: max_staging_depth (10), max_staging_files (100,000).

Pool handoff

The staging daemon and workload process can be different processes (e.g., Slurm prolog stages, then the workload runs):

1. Staging daemon holds the L2 pool via flock on pool.lock
2. Workload process adopts the pool via KISEKI_CACHE_POOL_ID env var
3. Workload takes over the flock

Each cache pool is identified by a 128-bit CSPRNG pool_id, isolated per process and per tenant.

Freshness and staleness

Metadata TTL (default 5s): File-to-chunk-list mappings are cached with a configurable TTL. Within the TTL, cached metadata is authoritative and may serve data for files that have since been modified (I-CC3, I-CC5).

Chunk data: No TTL needed. Chunks are immutable (I-C1), so a verified chunk remains correct indefinitely absent crypto-shred.

Crypto-shred detection

On crypto-shred (tenant KEK destruction), all cached plaintext must be wiped (I-CC12):

Detection mechanisms (in priority order):

1. Advisory channel notification (if active)
2. KMS error on next operation
3. Periodic key health check (default every 30s)

Response: Immediate wipe of L1 and L2 with zeroize.

Maximum detection latency: min(key_health_interval, max_disconnect_seconds).

Disconnect handling

If the client loses connectivity to all canonical endpoints for longer than max_disconnect_seconds (default 300s), the entire cache (L1 + L2) is wiped (I-CC6).

A background heartbeat RPC (every 60s) maintains the last_successful_rpc timestamp for disconnect detection.

Error handling

Any local cache error bypasses to canonical unconditionally (I-CC7):

• L2 I/O failure: bypass and flag pool for scrub
• CRC32 mismatch: bypass, delete corrupt entry
• Metadata lookup failure: bypass to canonical

Invariants

Environment variables

Security Model

Kiseki is designed with security as a foundational constraint, not a bolted-on feature. The system enforces strong tenant isolation, mandatory encryption, and a zero-trust boundary between infrastructure operators and tenants.

Zero-trust boundary

Kiseki enforces a strict separation between two administrative domains:

Cluster admin (infrastructure operator)

• Manages nodes, global policy, system keys, pools, devices.
• Cannot access tenant config, logs, or data without explicit tenant admin approval (I-T4).
• Sees operational metrics in tenant-anonymous or aggregated form.
• Modifications to pools containing tenant data are audit-logged to the affected tenant’s audit shard (I-T4c).

Tenant admin (data owner)

• Controls tenant keys, projects, workload authorization, compliance tags, user access.
• Grants or denies cluster admin access requests.
• Receives tenant-scoped audit exports sufficient for independent compliance demonstration.
• Can crypto-shred to render all tenant data unreadable.

Access request flow

When a cluster admin needs access to tenant resources (for debugging, migration, etc.):

1. Cluster admin submits an access request via the control plane.
2. The request is recorded in the audit log.
3. Tenant admin reviews and approves or denies.
4. If approved, access is time-bounded and scoped.
5. All access is audit-logged to the tenant’s shard.

Encryption at rest

Every chunk stored on disk is encrypted. There are no exceptions.

• Algorithm: AES-256-GCM (authenticated encryption with associated data).
• Key derivation: HKDF-SHA256 derives per-chunk DEKs from a system master key and the chunk ID (ADR-003).
• Envelope: Each chunk carries an envelope containing ciphertext, system-layer wrapping metadata, tenant-layer wrapping metadata, and authenticated metadata (chunk ID, algorithm identifiers, key epoch).

What is encrypted

What is NOT encrypted

• Delta headers: Compaction operates on headers only (I-O2). Headers contain no tenant-attributable content.
• Prometheus metrics: Aggregated counters and histograms. No tenant-attributable data in metric labels.
• Health/liveness probes: 200 OK response.

Encryption in transit

All data-fabric communication uses mTLS. No plaintext data crosses the network.

• Data path: mTLS with per-tenant certificates signed by the Cluster CA (I-K2).
• Raft consensus: mTLS between Raft peers.
• Key manager: mTLS between storage nodes and the key manager.
• Client to gateway: TLS (clients send plaintext over TLS; the gateway encrypts before writing).
• Native client: Client-side encryption (plaintext never leaves the workload process).

Protocol gateway encryption

Protocol gateway clients (NFS, S3) send plaintext over TLS to the gateway. The gateway performs tenant-layer encryption before writing to the storage layer. This means plaintext exists in gateway process memory but never on the wire in cleartext and never at rest.

Native client encryption

Native clients (FUSE, FFI, Python) perform tenant-layer encryption themselves. Plaintext never leaves the workload process and never traverses the data fabric.

FIPS 140-2/3 compliance

Kiseki uses aws-lc-rs as its cryptographic backend, which provides a FIPS 140-2/3 validated implementation of:

• AES-256-GCM (authenticated encryption)
• HKDF-SHA256 (key derivation)
• SHA-256 (content-addressed chunk IDs)
• HMAC-SHA256 (per-tenant chunk IDs for opted-out tenants)

The FIPS feature is controlled by the kiseki-crypto/fips feature flag at compile time.

Crypto-agility

Envelope metadata carries algorithm identifiers for crypto-agility. If a new algorithm is needed (e.g., post-quantum), envelopes can carry the new algorithm identifier alongside the existing one during a transition period.

No plaintext past gateway boundary (I-K1, I-K2)

This is the fundamental security invariant. Kiseki guarantees:

1. No plaintext chunk is ever persisted to storage (I-K1).
2. No plaintext payload is ever sent on the wire between any components (I-K2).
3. The system can enforce access to ciphertext without being able to read plaintext without tenant key material (I-K4).

Where plaintext exists

Plaintext exists only in:

• Client process memory: For native clients that perform client-side encryption.
• Gateway process memory: Transiently, while the gateway encrypts protocol-path data.
• Stream processor memory: Stream processors cache tenant key material and are in the tenant trust domain (I-O3).
• Client cache (L1): In-memory cache of decrypted chunks (zeroized on eviction or deallocation, I-CC2).
• Client cache (L2): On-disk cache of decrypted chunks on local NVMe (zeroized before unlink, I-CC2).

Content-addressed chunk IDs

Chunk identity is derived from content, serving both dedup and integrity:

• Default: chunk_id = SHA-256(plaintext). Enables cross-tenant dedup.
• Opted-out tenants: chunk_id = HMAC-SHA256(plaintext, tenant_key). Cross-tenant dedup is impossible. Zero co-occurrence leak (I-K10).

Tenants that opt out of cross-tenant dedup pay a storage overhead (identical data stored separately per tenant) but gain the guarantee that no metadata (chunk IDs, refcounts) leaks information about data similarity across tenants.

Audit trail

All security-relevant events are recorded in an append-only, immutable audit log with the same durability guarantees as the data log (I-A1).

Audit events include:

• Data access (read/write by tenant, workload, client)
• Key lifecycle (rotation, crypto-shred, KMS health)
• Admin actions (pool changes, device management, tuning parameters)
• Policy changes (quotas, compliance tags, advisory policy)
• Authentication events (mTLS success/failure, cert revocation)

Audit scoping

• Tenant audit export: Filtered to the tenant’s own events plus relevant system events. Delivered on the tenant’s VLAN (I-A2). Sufficient for independent compliance demonstration (HIPAA Section 164.312 audit controls).
• Cluster admin audit view: System-level events only. Tenant-anonymous or aggregated (I-A3).

Runtime integrity

An optional runtime integrity monitor detects attempts to access Kiseki process memory (I-O7):

• ptrace detection
• /proc/pid/mem access monitoring
• Debugger attachment detection
• Core dump attempt detection

On detection, the monitor alerts both cluster admin and tenant admin. Optional auto-rotation of keys can be configured as a response.

STRIDE Threat Analysis

Systematic analysis of Kiseki’s attack surfaces using the STRIDE framework.

Spoofing (identity)

Residual risk: NFS AUTH_SYS is inherently spoofable. Production deployments MUST use Kerberos or restrict NFS to trusted networks.

Tampering (data integrity)

Repudiation (deniability)

Information disclosure (confidentiality)

Denial of service (availability)

Elevation of privilege (authorization)

Summary

Both residual risks have documented mitigations:

1. NFS AUTH_SYS → deploy Kerberos or restrict to trusted networks
2. NVMe FTL data remanence → deploy OPAL/SED with per-boot key rotation

Authentication

Kiseki uses a layered authentication model. The primary mechanism is mTLS with certificates signed by a Cluster CA. Optional second-stage authentication via tenant identity providers adds workload-level authorization.

mTLS with Cluster CA (I-Auth1)

The Cluster CA is the trust root for all data-fabric authentication. Every participant in the data fabric (storage nodes, gateways, clients, stream processors) presents a certificate signed by the Cluster CA.

Certificate hierarchy

Cluster CA (managed by cluster admin)
  |
  +-- Server certificates (per storage node)
  |     SAN: node hostname, IP address
  |     OU: kiseki-server
  |
  +-- Key manager certificates (per key server)
  |     SAN: keyserver hostname, IP address
  |     OU: kiseki-keyserver
  |
  +-- Admin certificates (cluster admin)
  |     OU: kiseki-admin
  |
  +-- Tenant certificates (per tenant)
        SAN: tenant identifier
        OU: tenant-{org_id}

Properties

• No real-time auth server on data path (I-Auth1). Certificates are local credentials. Authentication is a TLS handshake, not an RPC to a central authority. This eliminates a latency-sensitive dependency on the data path.
• Per-tenant certificates: Each tenant’s clients and gateways present certificates that identify the tenant. The storage layer validates the certificate chain and extracts the tenant identity.
• Certificate revocation: Supported via CRL (KISEKI_CRL_PATH). The CRL is reloaded periodically. Revoked certificates are rejected at the TLS handshake.

Configuration

# On storage nodes
KISEKI_CA_PATH=/etc/kiseki/tls/ca.crt
KISEKI_CERT_PATH=/etc/kiseki/tls/server.crt
KISEKI_KEY_PATH=/etc/kiseki/tls/server.key
KISEKI_CRL_PATH=/etc/kiseki/tls/crl.pem  # optional

# On client nodes
KISEKI_CA_PATH=/etc/kiseki/tls/ca.crt
KISEKI_CERT_PATH=/etc/kiseki/tls/client.crt
KISEKI_KEY_PATH=/etc/kiseki/tls/client.key

Certificate generation example

# Generate Cluster CA (do this once)
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
  -keyout ca.key -out ca.crt -days 3650 -nodes \
  -subj "/CN=Kiseki Cluster CA"

# Generate server certificate
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
  -keyout server.key -out server.csr -nodes \
  -subj "/CN=node1.example.com/OU=kiseki-server"

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365 \
  -extfile <(echo "subjectAltName=DNS:node1.example.com,IP:10.0.0.1")

# Generate tenant client certificate
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
  -keyout tenant.key -out tenant.csr -nodes \
  -subj "/CN=workload-1/OU=tenant-acme-corp"

openssl x509 -req -in tenant.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out tenant.crt -days 365

SPIFFE SVID (I-Auth3)

SPIFFE (Secure Production Identity Framework for Everyone) is available as an alternative to raw mTLS certificate management.

SPIFFE ID structure

spiffe://kiseki.example.com/tenant/{org_id}/workload/{workload_id}
spiffe://kiseki.example.com/tenant/{org_id}/project/{project_id}/workload/{workload_id}

The SPIFFE ID maps directly to the tenant hierarchy (organization/project/workload).

SPIRE integration

SPIRE (the SPIFFE Runtime Environment) handles certificate issuance and rotation automatically:

1. SPIRE Server acts as the Cluster CA (or delegates to it).
2. SPIRE Agent runs on each node (storage and compute).
3. Workloads receive SVIDs via the Workload API.
4. Certificates rotate automatically (no manual renewal).

Benefits over raw mTLS

• Automatic certificate rotation (no manual renewal ceremonies).
• Workload attestation (verify the workload binary, not just the certificate).
• Short-lived certificates reduce the window of compromise.

S3 SigV4 authentication

The S3 gateway supports AWS Signature Version 4 authentication for S3 API clients.

How it works

1. The S3 client signs each request with an access key and secret key.
2. The gateway validates the signature.
3. The access key is mapped to a tenant identity via the control plane.
4. Subsequent authorization is based on the tenant identity.

Configuration

Access keys are provisioned via the control plane:

kiseki-server s3-credentials create --tenant-id acme-corp --workload-id training-job-1

Compatibility

The SigV4 implementation supports standard S3 clients:

# AWS CLI
aws --endpoint-url http://node1:9000 s3 ls

# boto3
import boto3
s3 = boto3.client('s3', endpoint_url='http://node1:9000',
                  aws_access_key_id='...', aws_secret_access_key='...')

NFS authentication

The NFS gateway supports two authentication mechanisms:

Kerberos (recommended for production)

NFSv4.2 with Kerberos provides strong authentication:

• krb5 — Authentication only.
• krb5i — Authentication + integrity.
• krb5p — Authentication + integrity + privacy (encrypted).

The Kerberos principal maps to a tenant identity.

AUTH_SYS (development only)

AUTH_SYS (traditional UNIX UID/GID authentication) is supported for development and testing. It provides no real security and should not be used in production. When AUTH_SYS is used, the NFS gateway maps the export path to a tenant identity.

OIDC/JWT second-stage authentication (I-Auth2)

Optional second-stage authentication validates workload identity against the tenant admin’s authorization. This provides an additional layer beyond the mTLS “belongs to this cluster” identity.

Architecture

Workload
  |
  v
mTLS (Cluster CA)  -->  "This workload belongs to tenant X"
  |
  v
OIDC/JWT (Tenant IdP)  -->  "This workload is authorized by tenant X's admin"

Integration

1. Tenant admin configures their identity provider (Keycloak, Okta, Azure AD, etc.) in the control plane.
2. Workloads obtain JWT tokens from the tenant IdP.
3. On connection, the workload presents both:
   • mTLS certificate (Cluster CA trust chain)
   • JWT token (tenant IdP authorization)
4. The storage node validates both independently.

Token validation

• JWT signature verification against the tenant IdP’s JWKS endpoint.
• Token expiry and audience validation.
• Claims mapping to tenant hierarchy (org, project, workload).
• No real-time IdP dependency on the data path: JWKS keys are cached and refreshed periodically.

gRPC role-based authorization

After authentication (mTLS + optional OIDC), gRPC services enforce role-based authorization:

Roles

Authorization enforcement

• StorageAdminService: Cluster admin only (mTLS cert with admin OU). SRE read-only role for monitoring.
• ControlService: Cluster admin for system operations, tenant admin for tenant-scoped operations.
• Data-path services (LogService, ChunkOps, CompositionOps, ViewOps): Any authenticated tenant workload, scoped to the tenant’s own data.
• WorkflowAdvisoryService: Any authenticated tenant workload. Per-operation authorization (I-WA3): every request re-validates the caller’s mTLS identity against the workflow’s owning workload.

Cluster admin isolation (I-T4)

The cluster admin certificate grants access to infrastructure management but explicitly does NOT grant access to:

• Tenant configuration
• Tenant audit logs
• Tenant data (read or write)
• Tenant key material

Access to tenant resources requires an explicit access request approved by the tenant admin.

Client identity

Client ID (native client)

Each native client process generates a stable identifier at startup:

• 128-bit CSPRNG value.
• Bound to the workload’s mTLS certificate at first use.
• Scoped within (org, project, workload).
• Never reused across processes (I-WA4).

The client ID ties an operation stream to a single process instance. It is not a user identity and not a session token.

Workflow reference

For advisory-enabled workloads, a workflow reference is attached to data-path RPCs as a gRPC binary metadata entry (x-kiseki-workflow-ref-bin). This is a 16-byte opaque handle, generated with 128+ bits of entropy, never reused, and verified against the caller’s mTLS identity on every request (I-WA3, I-WA10).

Tenant Isolation

Tenant isolation is a foundational invariant of Kiseki. Tenants are fully isolated with no cross-tenant data access, no delegation tokens, and no cross-tenant key sharing (I-T1).

Isolation model

Kiseki implements hierarchical tenancy with strict isolation boundaries:

Organization (billing, admin, master key authority)
  |
  +-- Project (optional: resource grouping, key delegation)
  |     |
  |     +-- Workload (runtime isolation unit)
  |     +-- Workload
  |
  +-- Workload (directly under org, if no projects)

Isolation guarantees

Per-tenant encryption keys

Each tenant has their own KEK (Key Encryption Key) managed by their chosen KMS backend (ADR-028). The tenant KEK wraps access to system DEK derivation parameters for that tenant’s data.

Key isolation

• System DEKs are derived per-chunk and are the same for identical chunks across tenants (enabling cross-tenant dedup by default).
• Tenant KEKs are unique per tenant. Even if two tenants store the same data, each tenant wraps access to the DEK derivation parameters independently.
• Tenant keys are not accessible to other tenants or to shared system processes (I-T3).

Key storage isolation

When using the internal KMS provider (default), tenant KEKs are stored in a separate Raft group from system master keys (I-K19). Compromise of one group does not expose the other.

When using external KMS providers (Vault, KMIP, AWS KMS, PKCS#11), tenant key material is managed entirely outside of Kiseki’s storage, under the tenant’s own operational control.

HMAC-keyed chunk IDs for opted-out tenants

By default, chunk IDs are derived from plaintext content: chunk_id = SHA-256(plaintext). This enables cross-tenant deduplication: identical data stored by different tenants produces the same chunk ID and shares storage.

Tenants that require stronger isolation can opt out of cross-tenant dedup (I-X2, I-K10):

Default:     chunk_id = SHA-256(plaintext)
Opted-out:   chunk_id = HMAC-SHA256(plaintext, tenant_key)

What opt-out provides

• No cross-tenant dedup: Identical data from different tenants produces different chunk IDs. Each tenant’s data is stored independently.
• Zero co-occurrence leak: An observer cannot determine whether two tenants store the same data by comparing chunk IDs.
• Storage overhead: Duplicate data across tenants consumes additional storage.

When to opt out

Opt-out is recommended for tenants with:

• Regulatory requirements prohibiting any form of cross-tenant data correlation (even at the metadata level).
• High-sensitivity data where the existence of shared content is itself sensitive information.
• Compliance regimes (HIPAA, ITAR) where data co-location with other tenants must be minimized.

Audit log scoping

The audit log is append-only, immutable, and system-wide (I-A1). Audit visibility is strictly scoped:

Tenant audit export (I-A2)

Each tenant receives a filtered projection of the audit log:

• All events originating from the tenant’s own operations.
• Relevant system events sufficient for a coherent, complete audit trail (e.g., a cluster admin modifying a pool that contains the tenant’s data).
• Delivered on the tenant’s VLAN.
• Sufficient for independent compliance demonstration (e.g., HIPAA Section 164.312 audit controls).

The tenant admin consumes this export. No events from other tenants appear in the export.

Cluster admin audit view (I-A3)

The cluster admin sees:

• System-level events (node joins, pool changes, key rotations).
• Tenant-anonymous or aggregated metrics.
• No tenant-attributable content.

Cluster admin modifications to pools containing tenant data are audit-logged to the affected tenant’s audit shard (I-T4c), so the tenant can review.

Advisory audit scoping (I-WA8)

Workflow advisory events (declare, end, phase-advance, hint accept/reject, etc.) are written to the tenant’s audit shard.

• Semantic phase tags and workflow IDs are tenant-scoped.
• Cluster-admin views see opaque hashes only (consistent with I-A3).
• High-volume events (hint-accepted, hint-throttled) may be batched or sampled, but at least one event per unique (workflow_id, rejection_reason) tuple is written per second.

Cache isolation (ADR-031)

The client-side cache maintains strict per-tenant isolation:

L1 (in-memory) isolation

• The L1 cache operates within a single client process.
• A client process is authenticated as a specific tenant via mTLS.
• L1 entries are decrypted plaintext chunks, keyed by chunk ID.
• On process termination, L1 entries are zeroized (I-CC2).

L2 (on-disk) isolation

• Each client process creates its own L2 cache pool on local NVMe.
• Pool isolation is enforced by:
  • Unique pool ID: 128-bit CSPRNG value per process.
  • flock: Ownership proven by file lock on pool.lock.
  • Per-process directory: No cross-process sharing.
• Concurrent same-tenant processes have independent pools. There is no cross-process cache sharing.
• Orphaned pools (no live flock holder) are scavenged on startup or by kiseki-cache-scrub.
• On eviction or cache wipe, L2 entries are overwritten with zeros before unlink (I-CC2).

Crypto-shred cache wipe (I-CC12)

When a crypto-shred event is detected for a tenant:

1. All cached plaintext for that tenant is wiped from L1 and L2.
2. L1 entries: Zeroizing<Vec<u8>> ensures memory-level erasure.
3. L2 entries: File contents overwritten with zeros before unlink.
4. Detection mechanisms:
   • Periodic key health check (default 30 seconds).
   • Advisory channel notification.
   • KMS error on next operation.

Maximum detection latency: min(key_health_interval, max_disconnect_seconds).

Physical-level erasure note

Logical-level erasure (zeroize before deallocation) provides strong protection against software-level attacks. For protection against physical-level attacks on flash storage (e.g., reading NAND cells after logical deletion), hardware encryption (OPAL/SED) on the compute node’s local NVMe is required. This is outside Kiseki’s control but should be part of the compute node security policy.

Network isolation

Data fabric

All data-fabric traffic is mTLS-encrypted. Tenant identity is extracted from the client certificate and validated on every RPC.

Management network

The management network (control plane, admin API) is separate from the data fabric. Cluster admin access requires admin-OU certificates.

Tenant VLAN

Tenant audit exports are delivered on the tenant’s VLAN, providing network-level isolation of audit data.

Advisory isolation (I-WA1, I-WA2, I-WA5, I-WA6)

The workflow advisory subsystem enforces strict tenant isolation:

• No existence oracles (I-WA6): A client cannot determine the existence of resources it is not authorized to observe. Unauthorized and absent targets return identical responses (same error code, payload size, and latency distribution).
• No content oracles (I-WA11): Advisory fields never include cluster-internal identifiers (shard IDs, chunk IDs, node IDs, device IDs, rack labels).
• Telemetry scoping (I-WA5): Every telemetry value is computed over resources the caller is authorized to read. Aggregate metrics use k-anonymous bucketing (minimum k=5).
• Covert-channel hardening (I-WA15): Response timing and size do not vary with neighbor-workload state.

Pool handle isolation (I-WA19)

Affinity pools are referenced via opaque pool handles, not cluster-internal pool IDs:

• Handles are valid for one workflow’s lifetime only.
• Never reused across workflows.
• Never equal or leak the cluster-internal pool identity.
• Multiple tenants can see the same opaque label attached to different internal pools; correlation across tenants is impossible because handles differ.

Compliance support

Kiseki’s tenant isolation model supports the following compliance regimes:

Compliance tags attach at any level of the tenant hierarchy (organization, project, workload) and inherit downward. Tags may impose additional constraints:

• Prohibit compression (HIPAA namespaces, I-K14).
• Set staleness floor (minimum 2 seconds for HIPAA).
• Require external KMS provider (no internal mode).
• Restrict pool placement (data residency).

Troubleshooting

This guide covers common issues, diagnostic tools, and resolution procedures for Kiseki clusters.

Diagnostic tools

Health endpoint

# Quick liveness check (returns "OK" or connection refused)
curl http://node1:9090/health

Event log

The event log captures categorized diagnostic events in memory. Query via the admin API:

# All events from the last 3 hours
curl http://node1:9090/ui/api/events

# Error events only
curl 'http://node1:9090/ui/api/events?severity=error'

# Critical events from the last 24 hours
curl 'http://node1:9090/ui/api/events?severity=critical&hours=24'

# Device-related events
curl 'http://node1:9090/ui/api/events?category=device'

# Raft events (elections, membership changes)
curl 'http://node1:9090/ui/api/events?category=raft'

Node status

# Per-node metrics and health
curl http://node1:9090/ui/api/nodes

# Cluster summary
curl http://node1:9090/ui/api/cluster

Structured logs

# Tail logs for errors (systemd)
journalctl -u kiseki-server -f --priority=err

# Search for specific errors in JSON logs
journalctl -u kiseki-server --output=json | jq 'select(.level == "ERROR")'

# Raft-specific logs
journalctl -u kiseki-server | grep kiseki_raft

Common issues

Connection refused on data-path port (9100)

Symptoms: Clients cannot connect. curl http://node:9090/health returns OK but gRPC connections to port 9100 fail.

Diagnosis:

1. Verify the port is listening:

   ss -tlnp | grep 9100
   

2. Check firewall rules:

   iptables -L -n | grep 9100
   

3. Check the server logs for bind errors:

   journalctl -u kiseki-server | grep "bind\|listen\|9100"
   

Common causes:

• Port conflict: Another process is using port 9100.
• Bind address: KISEKI_DATA_ADDR is set to 127.0.0.1:9100 instead of 0.0.0.0:9100.
• Firewall: Port 9100 is not open between nodes or to clients.

mTLS authentication failures

Symptoms: AuthenticationFailed errors in logs. Clients receive gRPC UNAUTHENTICATED (16) status.

Diagnosis:

# Verify certificate validity
openssl x509 -in /etc/kiseki/tls/server.crt -noout -dates -subject -issuer

# Verify certificate chain
openssl verify -CAfile /etc/kiseki/tls/ca.crt /etc/kiseki/tls/server.crt

# Test TLS handshake
openssl s_client -connect node1:9100 \
  -cert /etc/kiseki/tls/client.crt \
  -key /etc/kiseki/tls/client.key \
  -CAfile /etc/kiseki/tls/ca.crt

Common causes:

• Certificate expired: Renew the certificate.
• CA mismatch: Client and server certificates signed by different CAs.
• Missing SAN: Server certificate does not include the hostname or IP the client is connecting to.
• CRL revocation: Certificate revoked via KISEKI_CRL_PATH. Check the CRL:

  openssl crl -in /etc/kiseki/tls/crl.pem -text -noout
  

• Wrong OU: Tenant certificate has wrong OU, or admin certificate does not have kiseki-admin OU.

Capacity full (ENOSPC)

Symptoms: Write operations return PoolFull errors. S3 PutObject returns HTTP 507. NFS writes return EIO or ENOSPC.

Diagnosis:

# Check pool capacity
curl -s http://node1:9090/metrics | grep kiseki_pool_capacity

# Check system disk usage
df -h /var/lib/kiseki

Resolution:

1. Add devices to the pool to increase capacity.
2. Rebalance to distribute data more evenly:

   kiseki-server pool rebalance --pool-id fast-nvme
   

3. Evacuate devices from an over-full pool to a different pool (within the same device class).
4. Delete data: Remove compositions/objects to free space. GC runs periodically (default every 300 seconds).
5. Adjust thresholds if the defaults are too conservative for your deployment:

   kiseki-server pool set-thresholds --pool-id fast-nvme \
     --warning-pct 80 --critical-pct 90
   

Metadata disk full (system partition)

Symptoms: Inline threshold drops to floor (128 bytes). Alert: “system disk metadata usage exceeds hard limit.” Raft may stall if the system disk is completely full.

Diagnosis:

# Check system partition usage
df -h /var/lib/kiseki

# Check individual redb sizes
du -sh /var/lib/kiseki/raft/log.redb
du -sh /var/lib/kiseki/chunks/meta.redb
du -sh /var/lib/kiseki/small/objects.redb

Resolution:

1. The system automatically reduces the inline threshold to the floor (128 bytes) when the hard limit is exceeded (I-SF2).
2. Trigger Raft log compaction to reduce raft/log.redb size:

   kiseki-server compact
   

3. Run GC to clean up orphaned entries in small/objects.redb (I-SF6).
4. Consider migrating shards to nodes with larger system disks.
5. If the system partition is persistently undersized, upgrade to larger NVMe for the system RAID-1.

Raft diagnostics

Leader election issues

Symptoms: ShardUnavailable errors. Writes fail intermittently.

Diagnosis:

# Check shard health
kiseki-server shard health --shard-id shard-0001

# Check Raft events
curl 'http://node1:9090/ui/api/events?category=raft'

# Check election metrics
curl -s http://node1:9090/metrics | grep kiseki_raft

Common causes:

• Network partition: Raft peers cannot communicate. Check connectivity on port 9300 between all nodes.
• Clock skew: Large clock differences can cause election timeouts. Verify NTP synchronization. Nodes with Unsync clock quality are flagged (I-T6).
• Disk latency: HDD system disks cause 5-10ms fsync latency per Raft commit. Use NVMe or SSD for the system partition.

Quorum loss

Symptoms: All writes fail. Reads may succeed (depending on consistency model).

Diagnosis:

# Check how many nodes are reachable
for node in node1 node2 node3; do
  echo -n "$node: "
  curl -s http://$node:9090/health && echo "OK" || echo "DOWN"
done

Resolution:

• If one node is down (3-node cluster): The remaining 2 nodes form a majority. Raft continues. Repair or replace the failed node.
• If two nodes are down: Quorum is lost. See Backup & Recovery for recovery procedures.

Shard split stalls

Symptoms: Shard reports high delta count or throughput but split does not complete.

Diagnosis:

kiseki-server shard info --shard-id shard-0001

Resolution:

• Verify the shard is not in maintenance mode (I-O6).
• Check if the cluster-wide concurrent migration limit is reached (I-SF4): max(1, num_nodes / 10).
• Check the exponential backoff timer (I-SF4): Minimum 2 hours between placement changes per shard.
• Manually trigger a split if auto-split is not firing:

  kiseki-server shard split --shard-id shard-0001
  

Device issues

Integrity scrub

Trigger a manual integrity scrub to verify chunk data against EC parity:

# Scrub all devices
curl -X POST http://node1:9090/ui/api/ops/scrub

# Scrub a specific device
kiseki-server device scrub --device-id nvme-0001

The periodic scrub runs every 7 days by default (scrub_interval_h).

SMART warnings

Automatic evacuation triggers when a device reports:

• SSD: SMART wear indicator > 90%.
• HDD: > 100 bad sectors.

Check device health:

kiseki-server device info --device-id nvme-0001

Device evacuation

Monitor evacuation progress:

# List active repairs/evacuations
kiseki-server repair list

# Check device state
kiseki-server device info --device-id nvme-0001

Device state transitions: Healthy -> Degraded -> Evacuating -> Failed -> Removed (I-D2).

A device in Evacuating state can be cancelled:

kiseki-server device cancel-evacuation --device-id nvme-0001

RemoveDevice is rejected unless the device state is Removed (post-evacuation) (I-D5).

Key management issues

Key manager unreachable

Symptoms: KeyManagerUnavailable errors. All chunk writes fail cluster-wide (I-K12).

Diagnosis:

# Check key manager health
kiseki-server keymanager health

# Check connectivity from storage node
curl -s http://node1:9090/metrics | grep kms_reachability

Resolution:

• The key manager is a Raft-replicated HA service. If one node is down, the remaining majority continues serving.
• If the entire key manager cluster is unreachable, storage nodes use cached master keys (mlock’d in memory) for reads but cannot process new writes.
• Restore key manager connectivity as soon as possible.

Tenant KMS unreachable

Symptoms: TenantKmsUnreachable errors for operations involving the affected tenant. Other tenants are unaffected.

Diagnosis:

kiseki-server keymanager check-kms --tenant-id acme-corp

Resolution:

• Check network connectivity to the tenant’s KMS endpoint.
• Check KMS credentials and certificate validity.
• The tenant admin is responsible for their KMS availability (I-K11).

Crypto-shred verification

After a crypto-shred, verify that all clients have wiped their caches:

# Check crypto-shred count
curl -s http://node1:9090/metrics | grep kiseki_crypto_shred_total

# Check security events
curl 'http://node1:9090/ui/api/events?category=security'

Gateway issues

S3 errors

Common S3 error codes returned by the gateway:

NFS errors

Performance Tuning

Kiseki is designed for HPC and AI workloads running at 200+ Gbps per NIC. This guide covers tuning levers for maximizing throughput and minimizing latency.

Transport selection

The transport layer abstracts the network fabric. Kiseki automatically selects the best available transport, but manual override is possible.

Transport hierarchy (fastest to slowest)

Enabling high-performance transports

# Build with CXI support (requires libfabric development headers)
cargo build --release --features kiseki-transport/cxi

# Build with RDMA verbs support (requires rdma-core)
cargo build --release --features kiseki-transport/verbs

The client automatically detects available transports and selects the fastest one. Override with:

# Force TCP transport (e.g., for debugging)
KISEKI_TRANSPORT=tcp kiseki-client-fuse --mountpoint /mnt/kiseki

Transport tuning

• Connection pooling: The transport layer maintains a pool of connections per peer. Pool size adapts to workload.
• Keepalive: Connections are kept alive to avoid handshake overhead. Configure via KISEKI_TRANSPORT_KEEPALIVE_MS.
• Zero-copy: CXI and verbs transports use zero-copy DMA where possible.

NUMA pinning

For multi-socket servers, NUMA-aware placement is critical for avoiding cross-socket memory traffic.

Recommendations

• Pin kiseki-server to the NUMA node closest to the NIC:

  numactl --cpunodebind=0 --membind=0 kiseki-server
  

• Pin NVMe interrupts to the same NUMA node:

  echo 0 > /proc/irq/<irq>/smp_affinity_list
  

• Pin data devices to the NUMA node closest to their PCIe root complex.

systemd integration

[Service]
# Pin to NUMA node 0
ExecStart=/usr/bin/numactl --cpunodebind=0 --membind=0 /usr/local/bin/kiseki-server

Verification

# Check NUMA topology
numactl --hardware

# Check NIC NUMA node
cat /sys/class/net/eth0/device/numa_node

# Check NVMe NUMA node
cat /sys/block/nvme0n1/device/numa_node

Erasure coding parameters

EC parameters control the trade-off between storage overhead, repair bandwidth, and read performance.

Common configurations

Performance implications

• Read amplification: Reading a chunk requires reading data_chunks fragments. More data chunks = more read I/O.
• Write amplification: Writing a chunk requires writing data_chunks + parity_chunks fragments.
• Repair bandwidth: Repairing a lost fragment requires reading data_chunks fragments and writing 1. Higher data_chunks = more repair bandwidth.
• Minimum pool size: The pool must have at least data_chunks + parity_chunks devices.

EC parameters are immutable per pool after creation (I-C6). Choose carefully. Changing requires creating a new pool and migrating data.

Inline threshold (ADR-030)

The inline threshold determines whether small files are stored in the metadata tier (NVMe, redb) or the data tier (block device extents).

Tuning the threshold

The system automatically adjusts the threshold per-shard based on system disk capacity (I-SF1, I-SF2). Manual adjustment:

# Set cluster-wide default for new shards
kiseki-server tuning set --inline-threshold-bytes 8192

Trade-offs

Monitoring

# Check system disk usage
df -h /var/lib/kiseki

# Check per-store sizes
du -sh /var/lib/kiseki/small/objects.redb
du -sh /var/lib/kiseki/raft/log.redb

The Raft inline throughput guard (I-SF7) automatically reduces the threshold to the floor if inline write rate exceeds KISEKI_RAFT_INLINE_MBPS (default 10 MB/s per shard). This prevents inline data from starving metadata-only Raft operations during write storms.

Cache tuning (ADR-031)

L1 cache (in-memory)

The L1 cache holds decrypted plaintext chunks in process memory.

L2 cache (local NVMe)

The L2 cache uses local NVMe on compute nodes.

Metadata TTL

Cache mode selection

Staging for training workloads

Pre-stage datasets before training begins to avoid cold-start latency:

# Slurm prolog script
kiseki-client-fuse --stage /datasets/imagenet --mountpoint /mnt/kiseki
export KISEKI_CACHE_POOL_ID=$(cat /var/cache/kiseki/pool_id)

# Workload picks up the staged cache via KISEKI_CACHE_POOL_ID
srun --export=ALL python train.py

Raft tuning

Snapshot interval

kiseki-server tuning set --raft-snapshot-interval 10000

• Lower values (1000-5000): More frequent snapshots. Faster catch-up for new nodes. More I/O.
• Higher values (50000-100000): Less snapshot overhead. Slower catch-up.

Compaction rate

kiseki-server tuning set --compaction-rate-mb-s 200

Higher compaction rate reduces Raft log size faster but consumes more I/O bandwidth.

View materialization poll interval

kiseki-server tuning set --stream-proc-poll-ms 50

Lower poll interval reduces view staleness but increases CPU usage.

Benchmark harness

Kiseki includes a transport benchmark for measuring raw fabric throughput:

# Run transport benchmarks (if available)
tests/hw/run_transport_bench.sh

What it measures

• Bandwidth: Sequential read/write throughput per transport.
• Latency: Round-trip latency (p50, p99, p999) per transport.
• IOPS: Random read/write IOPS per transport.
• Concurrency: Throughput scaling with connection count.

Interpreting results

System tuning checklist

Kernel parameters

# Increase maximum open files
echo "fs.file-max = 1048576" >> /etc/sysctl.conf

# Increase socket buffer sizes for high-bandwidth transports
echo "net.core.rmem_max = 67108864" >> /etc/sysctl.conf
echo "net.core.wmem_max = 67108864" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem = 4096 87380 67108864" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem = 4096 65536 67108864" >> /etc/sysctl.conf

# Disable transparent hugepages (can cause latency spikes)
echo never > /sys/kernel/mm/transparent_hugepage/enabled

NVMe tuning

# Set I/O scheduler to none (best for NVMe)
echo none > /sys/block/nvme0n1/queue/scheduler

# Increase queue depth
echo 1024 > /sys/block/nvme0n1/queue/nr_requests

Process limits

# /etc/security/limits.d/kiseki.conf
kiseki  soft  nofile  1048576
kiseki  hard  nofile  1048576
kiseki  soft  memlock unlimited
kiseki  hard  memlock unlimited

Performance Tests

Benchmark results for kiseki on GCP infrastructure.

Test Environment

Results (2026-04-24)

Network Bandwidth

S3 Gateway

All S3 tests run from client nodes (n2-standard-8) with 8-way parallelism.

Write Throughput (single client → leader)

Read Throughput

PUT Latency (1 KB objects, sequential)

Aggregate Write (3 clients, parallel)

NFS / pNFS / FUSE

Not yet tested on GCP. NFS mount from client nodes requires SSH key distribution from the ctrl node (OS Login configuration pending). FUSE requires the kiseki-client binary installed on client nodes.

Local testing (3-node cluster on localhost) confirms all protocols functional via unit and integration tests.

Prometheus Metrics

Gateway request counters showed 0 during the test. The requests_total atomic counter in InMemoryGateway is not wired to the Prometheus metrics exporter yet.

Local Test Results (same binary, localhost)

For comparison, local 3-node cluster results (loopback network, no disk I/O latency, 32-way parallelism):

Observations

1. Small object writes improved 9.6x after ADR-032 (async GatewayOps + lock-free composition writes). The composition lock is no longer held during Raft consensus, allowing concurrent writes to proceed in parallel.

2. Read throughput exceeds write. Reads bypass Raft consensus (served from the local composition + chunk store) and hit 1.1 GB/s even for 1 MB objects.

3. GCP outperforms localhost for large objects. The GCP network (15+ Gbps) and n2-standard-16 nodes have more bandwidth than localhost loopback under contention. 16 MB writes: 1,102 MB/s (GCP) vs 341 MB/s (local).

4. Latency is network-bound. p50 latency on GCP (7.6 ms) includes network RTT + Raft consensus (5-node quorum). Local latency is dominated by CPU contention on shared machine.

5. Single Raft group is the write bottleneck. All writes go through one leader. Multi-shard deployment would distribute leaders across nodes, scaling write throughput linearly.

Known Issues

• Concurrent write deadlock (fixed in ADR-032). The sync→async bridge (run_on_raft) caused thread starvation under concurrent load. Fixed by making GatewayOps and LogOps fully async, and moving log emission out of the composition lock scope. Result: 1 MB writes improved from 39.5 to 380.2 MB/s (9.6x).

• NFS mount on GCP. Requires SSH key distribution from ctrl to client nodes. The ctrl service account needs osAdminLogin role and OS Login key registration.

• Prometheus counters. gateway_requests_total not exported to /metrics endpoint.

Running the Benchmark

# Local 3-node test
cargo build --release --bin kiseki-server
# Start 3 nodes (see examples/cluster-3node.env.node{1,2,3})
# Run: bash infra/gcp/benchmarks/perf-suite.sh

# GCP deployment
cd infra/gcp
terraform apply -var="project_id=PROJECT" -var="zone=ZONE" \
  -var="release_tag=v2026.1.332"
# Deploy perf-suite.sh to ctrl node and run

See infra/gcp/benchmarks/perf-suite.sh for the full benchmark script and infra/gcp/benchmarks/run-perf.sh for the local deployment wrapper.

Comparison with Ceph and Lustre

Single-Leader Kiseki vs Typical Deployments (similar hardware scale)

Why aggregate throughput is lower

All writes go through a single Raft leader (single Raft group). Ceph distributes across PGs/OSDs, Lustre stripes across OSTs. They parallelize writes across all nodes; kiseki serializes through one leader. This is a deployment constraint, not an architectural limit.

Where kiseki is strong

1. Per-leader throughput is excellent. 1.1 GB/s per leader with full AES-256-GCM encryption is comparable to Ceph RGW without encryption. The crypto overhead is nearly invisible (aws-lc-rs with AES-NI).

2. Read throughput matches. Reads bypass Raft consensus entirely and serve from local composition + chunk store. Multi-node reads scale linearly since any node can serve.

3. Latency is reasonable. 7.6 ms includes Raft consensus over network + encryption. Ceph’s 2-5 ms S3 latency is lower but typically without encryption. Lustre’s sub-ms is POSIX (kernel bypass), not comparable to HTTP/S3.

Bottleneck analysis

• Not bottlenecked by crypto – AES-256-GCM at 1.1 GB/s means the CPU encrypts faster than the network/Raft can deliver.
• Not bottlenecked by network – 15 Gbps available, using <10 Gbps.
• Bottlenecked by Raft consensus – 7.6 ms per round-trip for small objects, amortized for large ones.
• Multi-shard is the path to parity – linear scaling with shard count, same model as Ceph PGs and Lustre OSTs.

Projected multi-shard performance

At 5 shards on the same hardware, kiseki reaches parity with Ceph and approaches Lustre – while encrypting all data at rest and in transit, on commodity GCP VMs with network-attached storage (not local NVMe or InfiniBand).

Capacity Planning

Kiseki separates metadata and data onto different storage tiers. Proper sizing of both tiers is critical for stable operation at scale.

Storage tiers

Each storage node has two distinct storage tiers:

System disk (metadata tier)

The system partition hosts:

• Raft log (raft/log.redb): Bounded by snapshot interval. Grows with write rate, compacted periodically.
• Key epochs (keys/epochs.redb): Tiny (<10 MB). One entry per key epoch.
• Chunk metadata (chunks/meta.redb): Scales linearly with file count. Approximately 80 bytes per file.
• Inline content (small/objects.redb): Variable. Controlled by the dynamic inline threshold (ADR-030).

Requirements:

• NVMe or SSD strongly recommended. HDD system disks trigger a boot warning because Raft fsync latency will be 5-10ms per commit.
• RAID-1 on 2x SSD for redundancy (the system disk is not protected by Kiseki’s EC; it uses traditional RAID).
• Size based on expected file count and inline content.

Data devices (data tier)

Data devices are JBOD-managed by Kiseki. They store chunk ciphertext as extents on raw block devices (ADR-029).

Requirements:

• NVMe, SSD, or HDD depending on the pool’s device class.
• Multiple devices per node for EC placement (I-D4: no two EC fragments on the same device).
• JBOD (no RAID): Kiseki manages durability via EC or replication.

Metadata capacity sizing

Per-file metadata footprint

Capacity examples

10 billion files, 50-node cluster, RF=3, no inline:

At 168 GB per node, 256 GB NVMe system disks are tight. Larger system disks (512 GB or 1 TB) provide comfortable headroom.

10 billion files, 50-node cluster, RF=3, with inline (4 KB threshold):

This exceeds 256 GB system disks. The dynamic inline threshold (ADR-030) prevents this by automatically reducing the threshold when system disk usage approaches the soft limit.

Capacity monitoring

The system automatically monitors metadata disk usage and adjusts:

Alerts use out-of-band gRPC health reports (not Raft) so that a full-disk node can signal without writing Raft entries (I-SF2).

Dynamic inline threshold (ADR-030)

The inline threshold is computed per-shard as the minimum affordable threshold across all Raft voters:

available = min(node.small_file_budget_bytes for node in shard.voters)
projected_files = shard.file_count_estimate
raw_threshold = available / max(projected_files, 1)
shard_threshold = clamp(raw_threshold, INLINE_FLOOR, INLINE_CEILING)

Where:

Threshold behavior

• Decrease: Automatic and safe. New files use the chunk path. Existing inline data is not retroactively migrated (I-L9).
• Increase: Requires cluster admin decision. May trigger optional background migration of small chunked files back to inline.
• Emergency: If any voter reports hard-limit breach, the leader commits a threshold reduction via Raft (2/3 majority; the full-disk node’s vote is not required).

Raft throughput guard (I-SF7)

The effective inline threshold is further clamped by a per-shard Raft log throughput budget (KISEKI_RAFT_INLINE_MBPS, default 10 MB/s). If the shard’s inline write rate exceeds this budget, the threshold temporarily drops to the floor until the rate subsides. This prevents inline data from starving metadata-only Raft operations during write storms.

Pool capacity thresholds

Data-tier capacity is managed per pool. Thresholds vary by device class to account for SSD/NVMe GC pressure at high fill levels (ADR-024):

Why NVMe/SSD thresholds are lower

NVMe and SSD devices experience write amplification from garbage collection at high fill levels. Above ~80% fill, GC pressure increases sharply, causing:

• Increased write latency (10-100x during GC storms).
• Reduced effective write bandwidth.
• Accelerated wear.

Enterprise storage arrays (VAST, Pure) operate at 95%+ because they have global wear leveling across all flash in the system. JBOD devices do not have this capability, so Kiseki’s thresholds are more conservative.

Growth estimation

File count growth

Monitor kiseki_shard_delta_count to track delta (file) accumulation:

# Current delta count per shard
curl -s http://node1:9090/metrics | grep kiseki_shard_delta_count

Use the rate of delta count increase to project when the metadata tier will reach capacity.

Data volume growth

Monitor pool capacity metrics:

# Current pool utilization
curl -s http://node1:9090/metrics | grep kiseki_pool_capacity

Projection formula

days_until_full = (capacity_total - capacity_used) / daily_write_rate

For metadata:

metadata_per_file = 280 bytes (no inline) or 280 + avg_inline_size (with inline)
days_until_full = (system_disk_capacity * soft_limit_pct - current_used) /
                  (new_files_per_day * metadata_per_file * replication_factor)

Sizing recommendations

Small deployment (development/testing)

Medium deployment (departmental HPC)

Large deployment (institutional HPC/AI)

Rules of thumb

• System disk: Size at 2x the expected metadata footprint for comfortable headroom. Include inline content estimates.
• Data devices: At least ec_data_chunks + ec_parity_chunks devices per pool (for EC placement across distinct devices, I-D4).
• Network: CXI or InfiniBand for clusters where storage bandwidth is critical. TCP is acceptable for cold-tier pools.
• Memory: At least 64 GB per storage node for Raft state, chunk metadata caching, and stream processor buffers.

Capacity alerts

Configuring alerts

Use Prometheus alerting rules (see Monitoring) to detect capacity issues before they become critical:

- alert: KisekiSystemDiskWarning
  expr: >
    node_filesystem_avail_bytes{mountpoint="/var/lib/kiseki"} /
    node_filesystem_size_bytes{mountpoint="/var/lib/kiseki"} < 0.50
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "System disk above 50% on {{ $labels.instance }}"

- alert: KisekiSystemDiskCritical
  expr: >
    node_filesystem_avail_bytes{mountpoint="/var/lib/kiseki"} /
    node_filesystem_size_bytes{mountpoint="/var/lib/kiseki"} < 0.25
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "System disk above 75% on {{ $labels.instance }}"

When to add capacity

• System disk above 50% (soft limit): Plan for capacity expansion. Inline threshold will start decreasing.
• System disk above 75% (hard limit): Urgent. Inline threshold is at floor. Add nodes or upgrade system disks.
• Pool above Warning threshold: Monitor growth. Plan for device additions.
• Pool above Critical threshold: Writes are being rejected. Add devices immediately or evacuate data to another pool.

gRPC Services

Kiseki exposes several gRPC services across two network ports. Data-path services run on port 9100. The advisory service runs on a separate listener at port 9101 (isolated runtime, ADR-021).

LogService

Port: 9100 (data fabric) Provider: kiseki-log (via kiseki-server) Consumers: Composition, View stream processors, Gateway, Client

KeyManagerService

Port: Internal network (dedicated key manager cluster) Provider: kiseki-keymanager (via kiseki-keyserver) Consumers: Storage nodes (chunk encryption), Gateway, Client

System DEK derivation is local (HKDF, no RPC). Only master key fetch and tenant KEK operations require network calls (ADR-003).

ControlService

Port: Management network Provider: kiseki-control Consumers: Admin CLI, storage nodes, advisory runtime

Tenant management

Namespace and policy

IAM

Operations

Federation

Advisory policy

WorkflowAdvisoryService

Port: 9101 (data fabric, separate listener) Provider: kiseki-advisory (via kiseki-server, isolated tokio runtime) Consumers: Native client, any authorized tenant caller

Advisory stream message types

Inbound hints (client to storage):

• Access pattern declaration
• Prefetch range (up to 4096 tuples per hint, I-WA16)
• Affinity pool preference (via opaque pool handles, I-WA19)
• Priority class (within policy-allowed maximum)
• Retention intent
• Dedup intent
• Collective checkpoint announcement
• Deadline hint

Outbound telemetry (storage to client):

• Backpressure signal (ok / soft / hard severity with retry-after)
• Placement locality class (local-node / local-rack / same-pool / remote / degraded)
• Materialization lag
• Prefetch effectiveness
• QoS headroom
• Hotspot detection (caller-owned compositions only)

StorageAdminService (ADR-025)

Port: Management network Provider: kiseki-server Consumers: Cluster admin, SRE (read-only role)

DiscoveryService

Port: 9100 (data fabric) Provider: kiseki-server Consumers: Native client

Used by the native client to discover shards, views, and gateways from the data fabric without requiring direct control plane access (I-O4, ADR-008).

Protocol binding

• Protobuf definitions: proto/kiseki/v1/*.proto
• Generated code: kiseki-proto crate
• Workflow ref header: x-kiseki-workflow-ref-bin (16 raw bytes as gRPC binary metadata, not a proto field, per ADR-021)

REST & Admin API

The kiseki-server binary exposes an HTTP server (default port 9090) for health checks, Prometheus metrics, and an admin dashboard. All endpoints are served via axum.

Health and metrics

GET /health

Liveness probe for load balancers and orchestrators.

Response: 200 OK with body ok when the server is running.

GET /metrics

Prometheus text-format metrics endpoint.

Response: 200 OK with text/plain body containing all registered Prometheus metrics including:

• Raft state per shard (leader, follower, candidate)
• Chunk operations (reads, writes, dedup hits)
• Transport metrics (connections, bytes, errors per transport type)
• Pool utilization (capacity, used, free per pool)
• View materialization lag
• Advisory budget usage

Admin dashboard

GET /ui

HTML admin dashboard with HTMX live polling. Provides a visual overview of cluster health, node status, and operational metrics.

The dashboard polls the JSON API endpoints below for live updates.

JSON API endpoints

GET /ui/api/cluster

Cluster-wide summary with aggregated metrics from all nodes.

Response: JSON object with node count, total capacity, total used, shard count, and aggregated health status.

GET /ui/api/nodes

List of all known nodes with per-node metrics.

Response: JSON array of node objects, each with node ID, address, status, device count, shard count, and key metrics.

GET /ui/api/history

Metric time series for charting.

Query parameters: