Transcribed from the production blueprint in the Go repository. A label changes only when new evidence changes it — never because code exists.
Decentralized.Host files each capability under one label: VERIFIED (exercised by an automated test with real processes and failures), LIMITED (works with a stated limitation), NOT IMPLEMENTED, or NOT RUN. All VERIFIED results so far come from one macOS machine over loopback.
Implemented and exercised by an automated test or chaos scenario with real processes, sockets and failures (single-machine scope unless stated).
Works, but with a stated limitation or narrower evidence than production needs.
Absent. Where the system can detect the gap, it reports it.
The tooling exists, but the run that would establish the claim has not happened.
Every automated run so far is multi-process on one macOS machine over loopback. No test has crossed physical machines, a real WAN or NAT, or run on a Linux host. Linux validation (PV1-S1-A03) is pending.
Every assignment passes 18 ordered checks on the host before anything runs: signatures, audience, generation, clock, capability chain, revocation, freeze, freshness, tier, runtime, digest, attestation, federation and resource caps. The control plane proposes; the host decides.
Hosts report what they actually measured in Ed25519-signed observations with monotonic sequence numbers. Replayed, forged or stale-generation messages are rejected, and signed bundles carry a state index so the control plane cannot be rolled back.
If the control plane is frozen, unreachable or stale, admitted work keeps running (hold, not stop), new admissions are refused, and observations queue in an outbox that flushes on recovery.
Limitation: One M1 subtest ("host-mode offline-hold" ledger entry) failed intermittently in 2 of 3 full-suite runs on 2026-09-24 and has not reproduced since. The cause is not established; it stays open until explained.
The control plane keeps a hash-chained ledger with signed checkpoints, and every host keeps its own fsynced journal. `dh audit verify` fetches the ledger and verifies the chain locally; backups and exports verify offline.
BLAKE3 content addressing with FastCDC chunking, verified on every read. Snapshots commit only when 2 replicas return signed evidence; an interrupted write rolls back to the last committed snapshot, and a lost host is restored from peers byte for byte.
Replicas compare 256-bucket Merkle roots and repair divergent or corrupt objects from peers, recording repair evidence per object. Corrupt objects are quarantined, not served.
Volumes are replicated. `durability.erasure` accepts only `none`.
Snapshots are automatic and restore happens through rescheduling; `dh volume export` downloads and verifies a committed snapshot. Explicit snapshot and restore commands do not exist yet.
Authority flows as chains of Ed25519 blocks anchored in the cluster root: root → member, member → per-assignment capability bound to host, generation, digest and resource ceilings, expiring operator sessions, and single-use join tokens.
Host keys rotate with dual signatures and a grace window; a single key can be revoked in an emergency; the root key rotates with the old root signing the rotation and assignments re-signed.
Limitation: Root key custody (offline signer or HSM) is a hardening item, not implemented.
wireguard-go on a gVisor netstack (no root, no kernel interface) with signed key bindings, and SWIM gossip inside the tunnel. Workloads are reached through per-assignment forwarders, not routed per-workload IPs.
Limitation: Many real host processes, each with its own identity and WireGuard device, but all on one machine. No cross-machine, NAT or WAN run yet (PV1-S2 and PV1-S4).
Edge hosts probe replicas over the mesh and route only to healthy ones, with routing / draining / ejected / pending states. Hung replicas are ejected; rolling updates are make-before-break for stateless replicas.
HTTP-01, DNS-01 and wildcard issuance, with certificate state (REQUESTED … ISSUED … UNKNOWN) reported in observations.
Limitation: Verified only against Pebble, locally. Never exercised against a public CA or public DNS (P0-3).
Not implemented; the edge reports `UDP 443: NOT LISTENING`.
Three or five members, a deterministic state machine, a leader-only bundle issuer, membership changes and leadership transfer. Leader loss does not interrupt workloads.
Control-plane backups are signed, verify offline, and restore into a freshly bootstrapped cluster after losing every member.
Members serve TLS before they have credentials; bootstrap pins the throwaway certificate by fingerprint, then members serve root-issued certificates. Hosts join over HTTPS with a token that pins the root, and Raft uses mutual TLS.
Run a signed executable from the cluster CAS (`process`) or a digest-pinned container image (`docker`, with enforced --memory/--cpus and OOM detection).
Limitation: The process runtime does not enforce CPU or memory limits and says so in admission details (P0-6). containerd, gVisor and Firecracker are not implemented.
14 screens served by every control-plane member with no third-party requests. Desired, admitted and observed state are separate everywhere; every value carries a truth basis (OBSERVED, DERIVED, CONFIGURED, PLANNED, UNKNOWN); a stalled view turns into LAST KNOWN STATE — NOT CURRENT.
Limitation: No automated browser test suite or accessibility audit yet (P0-7).
`dh export` writes manifests, rosters, trust configuration, host records and policies, volume metadata, artifact references, audit and federation state; `dh import` verifies and re-applies it.
Limitation: No automated export → import → equivalence test yet (P0-5).
No outbound calls except those the operator configures (ACME, optional Postgres mirror). The console CSP is default-src 'self'.
Leader crash, total control-plane outage, host crash, agent restart, journal corruption, network partition, packet chaos, clock skew, stale generation, replay/forgery, revoked host, storage replica loss, disk full, OOM, interrupted deployment, interrupted storage write and Postgres outage — each under traffic with invariants checked and a signed report.
`dh chaos soak` exists but has not been run on the final build. The target is 24 hours under traffic on a multi-machine cluster (P0-2).
Root-signed agreements let one cluster grant another bounded capacity. Placements are re-signed by the grantor, hosts still apply their own policy (quotas, allowFederated), and revocation stops the work.
The dh/v1 specification, 136 test vectors, an adapter-protocol runner, and an independent Python implementation that passes canonical JSON, identity, envelopes, audit, capabilities, chunking and Merkle roots.
Limitation: Manifests, admission decisions, bundle verification and mesh bindings are specified but not yet pinned by vectors (P1-4).
From dh chaos list. Topology is control-plane members / hosts / edges; every scenario runs on its own disposable cluster, under traffic. All 17 passed in REF-MAC-A03 — on one machine.
| Scenario | Topology | Injection | Invariant checked |
|---|---|---|---|
| leader-crash | 3cp/3h/1e | SIGKILL the raft leader while traffic flows through the edge | a new leader is elected, committed writes survive, new writes succeed, and no workload request fails |
| cp-total-outage | 1cp/2h/1e | SIGKILL every control-plane member for 15s, then restart | hosts enter offline-hold, admitted workloads keep running with the same PIDs, no request fails, buffered observations are delivered after recovery |
| host-crash | 1cp/4h/1e | SIGKILL a host agent and all of its workloads (machine loss) | the edge stops routing to the dead replica without failed requests, the host is marked lost, and the replica is rescheduled and observed running elsewhere |
| agent-restart | 1cp/2h/0e | SIGKILL dh-noded only (workloads keep running) and restart it | the restarted agent re-adopts every workload by PID and kernel start time without restarting it |
| journal-corruption | 1cp/2h/0e | stop a host agent, flip bytes inside one entry of its hash-chained journal, restart it | the host detects the named break, enters ledger-corrupt mode, keeps admitted work, refuses new work, and never appends to the broken chain; after the local operator seals the journal (kept byte-for-byte) the host admits work again and its new chain records the break |
| network-partition | 1cp/4h/1e | drop every WireGuard packet to and from one host at the fault proxies and cut its control-plane channel for 25s, then heal | the isolated host keeps its admitted workload, gossip suspects it, the control plane marks it lost and reschedules, no request fails, and after healing the cluster converges to exactly the desired replicas |
| packet-chaos | 1cp/3h/1e | on every WireGuard path: 5% loss, 10% duplication, 10% reordering, +40ms delay for 20s | WireGuard and TCP absorb the faults: no request fails, the measured mesh RTT reflects the injected delay, and it returns to baseline when faults are cleared |
| clock-skew | 1cp/2h/0e | shift one host's clock +120s, then −120s, then restore | a skewed host detects it from signed bundle timestamps, holds admitted work, refuses new work (CLOCK_SKEW / stale plane), and returns to normal when the clock is corrected |
| stale-generation | 1cp/3h/1e | cut one host from the control plane, roll the app to a new generation, heal | the cut-off host keeps the generation it admitted (never runs an unsigned one), the console shows it as stale, other hosts roll within maxUnavailable, and after healing every replica runs the new generation |
| replay-forgery | 1cp/2h/0e | resend a committed host observation; send one signed by a foreign key; send one with a flipped field | each is rejected with a specific reason and recorded; the host's real state is unchanged |
| revoked-host | 1cp/4h/1e | revoke a host that serves a replica while traffic flows | the revoked host keeps its admitted workload but is removed from the mesh and the edge routing table, the replica is rescheduled, and no request fails |
| storage-replica-loss | 1cp/4h/1e | delete every stored object on one replica host while it runs | anti-entropy detects the missing objects by Merkle comparison, repairs them from peers with signed evidence, and the volume returns to HEALTHY with identical data |
| disk-full | 1cp/4h/1e | make every new write on one replica host fail with ENOSPC (storage-layer injection), write new data, then clear it | the volume is reported DEGRADED — never falsely HEALTHY — while the replica cannot store data; writes still commit on the other replicas; after the disk frees, anti-entropy restores HEALTHY |
| oom | 1cp/1h/0e | run a container limited to 32 MiB that allocates without bound | the container runtime kills it (OOMKilled=true), the host reports oom-killed from runtime evidence, audits it, and restarts it with backoff |
| interrupted-deployment | 3cp/3h/1e | apply a new generation and SIGKILL the raft leader as soon as the first replica has rolled | the new leader resumes the rollout, it completes, and requests keep succeeding (maxUnavailable respected) |
| interrupted-storage-write | 1cp/4h/1e | write 24 MiB, then SIGKILL the primary host (agent and workload) while the new snapshot is still pending | committed data is intact after recovery, and the interrupted write is either fully committed or absent — never partial |
| postgres-outage | 1cp/1h/0e | stop the Postgres evidence mirror, keep operating, restart it | while Postgres is down the console reports JOURNAL ONLY (never DB COMMITTED), writes still commit through raft, and the mirror catches up to DB COMMITTED after recovery |
Source of truth: docs/BLUEPRINT.md. If this page and the blueprint disagree, the blueprint wins.