Talking between nodes: the CBOR wire envelope and a SWIM cluster that gossips

Hew is aimed at service backends — the kind of system that ends up as more than one process on more than one machine, whether you wanted it to or not. So the distributed substrate isn’t a bolt-on for me; it’s the part of the language that has to be right before the rest of it means anything. This week I spent most of my time below the actor layer, on the two things that decide whether a Hew cluster is trustworthy: what crosses the wire, and how a node decides a peer is dead.

I’ll start with the wire, because that’s where the worst code in the runtime was.

The format I was embarrassed by

The original inter-node wire format was something I hand-built early on, when I just needed two processes to exchange an actor message and didn’t want to think about it. It had a magic number, a fixed field layout, manual offset arithmetic, and a 2,498-line wire.rs to encode and decode it all. It worked. It also had exactly the property you don’t want in a format that two independent processes have to agree on: the only specification of it was the code that read and wrote it.

That’s fine when there’s one reader and one writer and they’re the same binary. It is not fine the moment you want a second implementation, a debugging tool that can decode a captured frame, or a version of the runtime six months from now that has to stay compatible with this one. The format lived in my head and in wire.rs, and those are the same single point of failure.

It took me a while to commit to replacing it, because the hand-rolled thing worked. The replacement is wholesale: CBOR (a compact binary encoding, the binary cousin of JSON, with a real IETF spec) for the frames, described in CDDL, a schema language for CBOR. CDDL is the doc-of-truth now. The Rust types follow the schema, not the other way around.

Here’s the shape of it. A runtime frame is a definite-length CBOR map with integer keys — the keys are small integers rather than strings, so there’s no field-name overhead on every message. There are two top-level rules, and frame_type picks the branch:

wire-frame = control-frame / envelope-frame

envelope-frame = {
    version:          1 => wire-version,
    frame_type:       2 => 1,            ; 1 = envelope
    target_actor_id:  3 => actor-id,
    source_actor_id:  4 => actor-id,
    msg_type:         5 => int,
    payload:          6 => bstr,         ; serialised message body
    request_id:       7 => uint,         ; 0 = fire-and-forget
    source_node_id:   8 => node-id,
}

wire-version = 1   ; decoders reject unknown values
actor-id     = uint
node-id      = uint

The version => wire-version line is the load-bearing one. A decoder that sees a version it doesn’t recognize returns an UnknownVersion error and stops — it does not guess, does not try to parse the rest, does not fall through to the old format. That last clause matters more than it looks: when I swapped the six live call sites from the old format to the envelope, every one of them routes a decode failure through a typed error. There is no silent fallback to the legacy format anywhere. If a frame doesn’t decode as CBOR, it fails closed, because a wire format that silently accepts garbage is a wire format that will silently corrupt state in production.

To be clear about what lives where: the envelope above is the runtime’s plumbing, and your message rides inside it as opaque bytes in payload: bstr. What goes into those bytes is your own wire type, encoded by the stdlib codecs, not by the envelope’s CBOR. User-facing message types describe themselves with @N field tags. You tag a struct for the wire and give each field a number:

#[wire]
type Task {
    id: i64 @1,
    payload: string @2,
}
 
fn main() {
    let t = Task { id: 7, payload: "reindex" };
    println(t.payload);
}

The field tags are the stable identity — rename payload to body and the wire doesn’t care, because the wire only knows @2. That’s the same backwards-compatibility discipline protobuf gets from its field numbers, and it’s why the tags are part of the type and not an afterthought.

Once every call site was on the envelope, deleting wire.rs was almost anticlimactic. One commit, 2,498 lines gone, plus the hew_wire_* C exports that nothing called anymore. I grepped for the old magic number and the old encode/decode symbols across the whole runtime afterward and got zero hits. The format that only existed in my head and in that file now exists in a schema that a tool — or another person — can read.

(The doctrine I’m settling on, beyond this one format: CBOR-with-CDDL for the runtime’s own inter-process messaging, and the stdlib’s protobuf/msgpack/JSON codecs for the wire types users define for their own services. The runtime’s plumbing and the user’s payloads are different concerns and shouldn’t share one format just because both are “binary serialization”.)

Deciding a peer is dead is harder than it sounds

The other half of the week was the cluster. Hew’s membership uses SWIM — a gossip-based protocol where each node periodically pings a random peer, falls back to indirect pings through other peers if the direct one is silent, and piggybacks membership changes on the ping/ack traffic so the whole cluster converges on who’s alive without a central coordinator. That part was already in place. What I changed is the single most consequential decision SWIM makes: when does a peer go from ALIVE to SUSPECT?

It used to be a flat timeout. ping_timeout_ms = 500, and if a peer hadn’t answered in half a second, suspect it. The problem with a fixed timeout is that it forces one trade-off across every network you’ll ever run on. Set it short and a sleepy WAN link produces spurious suspicions of perfectly healthy peers every time a packet takes a scenic route. Set it long and a snappy LAN waits seconds longer than it needs to before reacting to a node that genuinely fell over. There is no single number that’s right for both, and the runtime doesn’t get to know in advance which one you have.

So I replaced the threshold with a phi-accrual failure detector — the Hayashibara et al. design that Akka and Cassandra use. The idea is to stop asking “has it been longer than X?” and start asking “given everything I’ve learned about this peer’s timing, how surprising is this silence?”

Each peer gets a sliding window of the last 200 heartbeat inter-arrival intervals. From that window the detector estimates a mean and standard deviation, and on every tick it computes

t = now − last_heartbeat
φ = −log₁₀(P(interval > t | learned distribution))

Phi is a confidence value, not a duration. It climbs as the silence gets more improbable relative to what that peer normally does, and the peer goes SUSPECT once phi crosses 8.0 — roughly a one-in-a-hundred-million chance the silence is benign jitter. The crucial property is that the wall-clock wait emerges from the data instead of being a constant. A peer whose heartbeats land like a metronome accrues phi fast the moment it goes quiet. A peer on a jittery link is granted proportional slack before the substrate is willing to call it suspect. Same threshold, different real-world latency, and I didn’t have to pick either one.

There were two details I had to get right or the whole thing would quietly misbehave.

The first is cold start. Mean and standard deviation are meaningless on a handful of samples, so a detector that has seen fewer than ten intervals returns phi of zero — it never raises suspicion on its own. While the window is still warming, SWIM falls back to the old fixed ping_timeout_ms so a brand-new or very quiet peer isn’t a blind spot. This is the fail-closed choice: a fresh detector abstains rather than fabricating a high phi from noise. (It is genuinely tempting to let a new detector be aggressive, and genuinely wrong. Three samples can’t tell you anything about a distribution.)

The second is recovery. When a peer that was SUSPECT finally sends a heartbeat again, that one enormous inter-arrival interval is recorded as an anchor but is deliberately not folded into the distribution. If I let the multi-second recovery gap into the window, I’d be teaching the detector that long silences are normal for this peer — which would dull every future detection of a real failure. The recovery proves the peer is back; it must not be allowed to make the detector more forgiving.

One thing I was careful not to change: phi-accrual alters when the substrate fires a SUSPECT event, never what a consumer sees. The membership-event surface is identical — same events, same shape — so nothing above the cluster had to know any of this happened. The whole change is a swap of the trigger logic behind a stable boundary, and the tuning constants live as module-level values rather than on the C-ABI config struct, so re-tuning later is a one-line edit and not a compatibility break.

A seam for testing the thing you can’t schedule

The hardest part of distributed-systems code is that the interesting behaviour only happens during a partition, and you can’t make a partition happen on demand in a unit test. So the protocol decision logic sits behind a ClusterProtocol trait — the membership state machine doesn’t bake in how the alive/suspect/dead decision is made, it calls through the trait to whatever detector is installed. Today that’s a SWIM implementation driving the phi-accrual detector; the trait is what lets me swap the strategy without rewriting the membership machinery around it.

The trait also carries the seam I built specifically for testing: when a member is declared dead, the protocol calls into a partition registry that fans a partition signal out to every live receiver registered for that node. In tests, that’s how I inject a partition deterministically — I don’t have to drop real packets or sever a real socket, I tell the registry the node is dead and watch every blocked receiver on that node resolve. The thing you normally can’t schedule becomes a function call, which is the only way I know to write a failure-handling test that fails for the right reason.

Encrypting the mesh, and refusing strangers

Transport selection is configuration, not code. The runtime ships a TCP transport and a QUIC mesh, and you pick the mesh either from the environment or from inside the program:

fn main() {
    Node.set_transport("quic");
    Node.start("127.0.0.1:9000");
    // ... register actors, accept peers ...
    Node.shutdown();
}

The QUIC mesh exists because the older QUIC transport opened a single bidirectional stream per connection, which means a slow consumer on that one stream stalls every actor sharing the connection — head-of-line blocking at the application level, even though QUIC eliminates it at the transport level. The mesh opens a fresh stream per actor-pair instead, each independently flow-controlled, so a slow consumer on one stream can’t stall its siblings.

The part I want to call out, though, is the handshake. Connections run over QUIC’s TLS 1.3, and peer certificates are checked against a per-node SPKI allowlist — the set of public-key fingerprints (the SubjectPublicKeyInfo bytes of each peer’s leaf certificate) that this node is willing to talk to. The allowlist starts empty, and that is the whole point: a node with an empty allowlist accepts no other node. (The one SPKI it always trusts is its own, so loopback to itself keeps working.) You register the SPKI of each peer you intend to trust before you start listening, and anything else is rejected at the TLS handshake — fail-closed, before a single application byte is exchanged. The verifier is installed on both ends — as a client-cert verifier on the server side and a server-cert verifier on the client side — so the rejection is mutual. The default posture of the mesh is “I trust no one,” which is the only sane default for a thing that listens on a network. (There’s an honest seam here I’m not hiding: the mesh’s certificate identity and the runtime’s existing Noise-handshake identity use different key types and aren’t unified yet. Unifying them needs a signing-capable static key the whole runtime shares, and that’s a later piece of work. For now a caller that uses both has to register both.)

The part I actually care about: sends that stay type-checked

Everything above is plumbing. Here’s the part that’s the reason I’m building a language and not a library.

When you look up an actor on another node, you don’t get back an untyped handle that accepts any message and fails at runtime if you got it wrong. You get a typed, fallible result:

record Job {
    id: i64,
}
 
actor Worker {
    receive fn run(msg: Job) {}
}
 
impl ActorMsg for Worker {
    type Msg = Job;
    type Reply = ();
}
 
fn main() {
    let found: Result<RemotePid<Worker>, LookupError> = Node.lookup("worker");
    match found {
        .Ok(remote) => println("found remote worker"),
        .Err(_) => println("no worker registered"),
    }
}

(Update, June 2026: if you run this today, the checker emits a spurious orphan-rule warning on the impl ActorMsg block, even though Worker is defined right above it — the orphan check doesn’t yet count a local actor as a locally-defined type. A diagnostics bug, not a problem with the code.)

Node.lookup("worker") returns Result<RemotePid<Worker>, LookupError>. The lookup can fail — the name might not be registered on any node the cluster knows about — and the type makes you say what happens when it does. There’s no nullable handle to forget to check; the Err arm is not optional. And the success value isn’t a generic remote pid, it’s a RemotePid<Worker>, carrying the actor type all the way across the node boundary.

That type parameter is what keeps the send honest. A RemotePid<Worker> only accepts Worker’s message type, checked at compile time, exactly like a local send:

fn dispatch(remote: RemotePid<Worker>) -> Result<(), SendError> {
    remote.tell(Job { id: 7 })
}

remote.tell(Job { id: 7 }) reads like a local actor send because, as far as the type system is concerned, it is one — the message has to match the actor’s declared Msg or the program doesn’t compile. The difference is entirely under the surface. The checker rewrites every tell on a RemotePid receiver into a direct call to the runtime’s send-by-id path (hew_actor_send_by_id), which serializes the message into a CBOR envelope and routes it to the actor’s node. You write a method call against a typed handle; the compiler turns it into a wire send. The remote-ness is a lowering detail, not something the programmer has to hand-marshal and pray about.

I should be straight about where the seams are, because this is real-but-young. The wire envelope is live and the lookup-and-tell path type-checks and lowers end to end, but the full cross-process routing behind it is still being wired up — a tell to a peer that isn’t reachable fails closed with a SendError rather than pretending it landed, which is the behaviour I want but not yet the behaviour of a battle-tested cluster. Node::lookup resolving across gossiped names from real remote peers is the next piece, not a finished one.

(Update, June 2026: that piece has since landed. Registry names now gossip between nodes over the CBOR control frame, and the two-process case — spawn two real processes, Node.lookup a name registered on the other one, tell it — runs end to end in the runtime’s test suite. The fail-closed SendError behaviour described above is unchanged; the routing behind it is now real.)

But the shape is the thing I was after, and the shape is settled. A distributed message in Hew is a typed method call on a typed handle, the lookup that produced the handle is a Result you can’t ignore, and the bytes that cross the wire are described by a schema instead of living in 2,498 lines of offset arithmetic. The compiler knows it’s a remote send and you mostly don’t have to.