Performance Tests

Benchmark results for kiseki on GCP infrastructure.

Test Environment

Results (2026-04-24)

Network Bandwidth

S3 Gateway

All S3 tests run from client nodes (n2-standard-8) with 8-way parallelism.

Write Throughput (single client → leader)

Read Throughput

PUT Latency (1 KB objects, sequential)

Aggregate Write (3 clients, parallel)

NFS / pNFS / FUSE

Not yet tested on GCP. NFS mount from client nodes requires SSH key distribution from the ctrl node (OS Login configuration pending). FUSE requires the kiseki-client binary installed on client nodes.

Local testing (3-node cluster on localhost) confirms all protocols functional via unit and integration tests.

Prometheus Metrics

Gateway request counters showed 0 during the test. The requests_total atomic counter in InMemoryGateway is not wired to the Prometheus metrics exporter yet.

Local Test Results (same binary, localhost)

For comparison, local 3-node cluster results (loopback network, no disk I/O latency, 32-way parallelism):

Observations

1. Small object writes improved 9.6x after ADR-032 (async GatewayOps + lock-free composition writes). The composition lock is no longer held during Raft consensus, allowing concurrent writes to proceed in parallel.

2. Read throughput exceeds write. Reads bypass Raft consensus (served from the local composition + chunk store) and hit 1.1 GB/s even for 1 MB objects.

3. GCP outperforms localhost for large objects. The GCP network (15+ Gbps) and n2-standard-16 nodes have more bandwidth than localhost loopback under contention. 16 MB writes: 1,102 MB/s (GCP) vs 341 MB/s (local).

4. Latency is network-bound. p50 latency on GCP (7.6 ms) includes network RTT + Raft consensus (5-node quorum). Local latency is dominated by CPU contention on shared machine.

5. Single Raft group is the write bottleneck. All writes go through one leader. Multi-shard deployment would distribute leaders across nodes, scaling write throughput linearly.

Known Issues

• Concurrent write deadlock (fixed in ADR-032). The sync→async bridge (run_on_raft) caused thread starvation under concurrent load. Fixed by making GatewayOps and LogOps fully async, and moving log emission out of the composition lock scope. Result: 1 MB writes improved from 39.5 to 380.2 MB/s (9.6x).

• NFS mount on GCP. Requires SSH key distribution from ctrl to client nodes. The ctrl service account needs osAdminLogin role and OS Login key registration.

• Prometheus counters. gateway_requests_total not exported to /metrics endpoint.

Running the Benchmark

# Local 3-node test
cargo build --release --bin kiseki-server
# Start 3 nodes (see examples/cluster-3node.env.node{1,2,3})
# Run: bash infra/gcp/benchmarks/perf-suite.sh

# GCP deployment
cd infra/gcp
terraform apply -var="project_id=PROJECT" -var="zone=ZONE" \
  -var="release_tag=v2026.1.332"
# Deploy perf-suite.sh to ctrl node and run

See infra/gcp/benchmarks/perf-suite.sh for the full benchmark script and infra/gcp/benchmarks/run-perf.sh for the local deployment wrapper.

Comparison with Ceph and Lustre

Single-Leader Kiseki vs Typical Deployments (similar hardware scale)

Why aggregate throughput is lower

All writes go through a single Raft leader (single Raft group). Ceph distributes across PGs/OSDs, Lustre stripes across OSTs. They parallelize writes across all nodes; kiseki serializes through one leader. This is a deployment constraint, not an architectural limit.

Where kiseki is strong

1. Per-leader throughput is excellent. 1.1 GB/s per leader with full AES-256-GCM encryption is comparable to Ceph RGW without encryption. The crypto overhead is nearly invisible (aws-lc-rs with AES-NI).

2. Read throughput matches. Reads bypass Raft consensus entirely and serve from local composition + chunk store. Multi-node reads scale linearly since any node can serve.

3. Latency is reasonable. 7.6 ms includes Raft consensus over network + encryption. Ceph’s 2-5 ms S3 latency is lower but typically without encryption. Lustre’s sub-ms is POSIX (kernel bypass), not comparable to HTTP/S3.

Bottleneck analysis

• Not bottlenecked by crypto – AES-256-GCM at 1.1 GB/s means the CPU encrypts faster than the network/Raft can deliver.
• Not bottlenecked by network – 15 Gbps available, using <10 Gbps.
• Bottlenecked by Raft consensus – 7.6 ms per round-trip for small objects, amortized for large ones.
• Multi-shard is the path to parity – linear scaling with shard count, same model as Ceph PGs and Lustre OSTs.

Projected multi-shard performance

At 5 shards on the same hardware, kiseki reaches parity with Ceph and approaches Lustre – while encrypting all data at rest and in transit, on commodity GCP VMs with network-attached storage (not local NVMe or InfiniBand).