Wiring the QUIC Mesh

Two programs, two terminals. Server:

actor Counter {
    let count: int;
    receive fn increment(n: int) {
        count = count + n;
    }
    receive fn get_count() -> int {
        count
    }
}
 
fn main() {
    Node::set_transport("quic");
    Node::start("127.0.0.1:9000");
    let counter = spawn Counter;
    Node::register("counter", counter);
    sleep_ms(5000);
    Node::shutdown();
}

actor Counter {
    var count: i64;
    receive fn increment(n: i64) {
        count = count + n;
    }
    receive fn get_count() -> i64 {
        count
    }
}
 
fn main() {
    Node.set_transport("quic");
    Node.start("127.0.0.1:9000");
    let counter = spawn Counter(count: 0);
    Node.register("counter", counter);
    sleep_ms(5000);
    Node.shutdown();
}

Client:

fn main() {
    Node::set_transport("quic");
    Node::start("127.0.0.1:9001");
    Node::connect("127.0.0.1:9000");
    let counter: Counter = Node::lookup("counter");
    counter.increment(10);
    counter.increment(20);
    let result = await counter.get_count();
    Node::shutdown();
}

The syntax is intentionally local-looking. counter.increment(10) on a remote actor is identical to the same call on a local one. The PID is an i64 either way — what changes is the codegen path. If the actor ID is in the local range, the runtime does a direct mailbox push. If it’s in the remote range, it serializes the message and sends it over QUIC. The calling code doesn’t know and doesn’t need to.

Why QUIC

The original plan was TCP with a Noise protocol handshake on top. The distributed actors v2 branch already had TCP infrastructure — the I/O poller, the connection pool, the gossip layer. Adding Noise would have given us mutual authentication and forward secrecy.

I switched to QUIC instead, and the reason was simple: QUIC has TLS 1.3 built in. No separate handshake layer to implement, no key derivation to get wrong at 1am, no “I wonder if I implemented the transcript hash correctly.” The quinn crate handles all of it. One bidirectional QUIC stream per logical connection, established lazily on first send. ALPN protocol identifier hew-mesh/1 so a QUIC server that isn’t a Hew node rejects the connection early.

The one awkward part: quinn is async (tokio), and the Hew runtime is synchronous. A background tokio task accepts incoming connections and feeds them through a std::sync::mpsc channel, which the synchronous vtable side reads. It’s a thin async-to-sync bridge, but it keeps tokio isolated — it doesn’t touch the scheduler or the actor runtime.

The hard part: distributed reply channels

Fire-and-forget sends are easy. You serialize the message, write it to the QUIC stream, move on. What’s hard is await counter.get_count() — where the calling thread needs to block until a response comes back from another machine.

Local await uses a condvar: the calling thread registers a reply slot, sends the ask, and waits on the condvar. The actor’s receive thread signals it when the return value is ready. That doesn’t work across machines, because the return value isn’t coming back through the local scheduler — it’s arriving as bytes on a network connection.

The fix was two new fields in the wire envelope: request_id (field 5, varint) and source_node_id (field 6, varint), both zero for fire-and-forget messages. On a remote ask, the caller generates a request ID, registers a slot in a process-global reply table keyed by that ID, sends the envelope, and blocks on a condvar with a 5-second timeout. On the remote side, an incoming-message listener spawns a thread to perform a local blocking ask, then packages the result into a reply envelope and sends it back to source_node_id. When the reply arrives, the receive loop deposits it in the table and signals the condvar.

Because the new fields are only encoded when non-zero, the wire format is backward-compatible. A fire-and-forget message is the same size as before. A request message is a few bytes larger. Old nodes that don’t understand the new fields ignore them.

The 5-second timeout is a // FIXME: configurable. For now it’s enough to detect a crashed remote node; the caller gets an error, and the supervision layer can decide what to do with it. (Distributed supervision is still the gap from the v0.1.9 post — that hasn’t changed.)

Service discovery

Node::lookup("counter") needs to know which node holds the actor registered under that name, and what its PID is. The gossip layer already propagates membership information — which nodes are alive, which have left. I extended it with two new event types: GOSSIP_REGISTRY_ADD and GOSSIP_REGISTRY_REMOVE.

When a node calls Node::register("counter", counter), it packages up (name, actor_pid, node_id, incarnation) into a GOSSIP_REGISTRY_ADD event and lets gossip disseminate it. Every node maintains a local registry. When a node shuts down, its registry entries get GOSSIP_REGISTRY_REMOVE events with a higher incarnation number, and peers purge them.

Node::lookup("counter") checks the local registry first. If the name is found and the registered node is the current node, it returns the local PID directly. If it’s on a remote node, it returns a remote PID — one in the range the codegen uses to route over the network.

The gossip approach means discovery is eventually consistent. If you call Node::register on one node and immediately call Node::lookup on another, you might not find it yet. In practice, with two nodes on localhost, the propagation delay is under a millisecond. On a real cluster with gossip fanout, you’d need to wait or retry. This is the same trade-off SWIM makes for membership — correctness in the common case, no guarantees in the window between write and read.

Wiring it through the type checker

Node::lookup("counter") returns a remote actor handle, but the type checker needs to know what kind of actor it is before it can verify that counter.increment(10) is a valid call.

The call site has a type annotation: let counter: Counter = Node::lookup("counter"). The type checker treats lookup’s return type as a fresh type variable, unifies it with Counter from the annotation, then — before emitting TypeCheckOutput — applies the substitution to every expression type in the program. By the time codegen runs, there are no unresolved type variables. The runtime handle carries the same Counter type as a locally-spawned actor.

This was the least exciting piece of the implementation and took the longest to get right. Type variable substitution at the wrong point in the pipeline leaves codegen with half-resolved types that produce valid MLIR but wrong code. The fix was moving the final substitution pass to immediately before TypeCheckOutput is emitted, so the type system view and the codegen view are always the same snapshot.

The commits

919e9560 is the QUIC transport — about 1179 lines, 33 unit tests covering connection setup, stream multiplexing, and the async-to-sync bridge. 031b909e is the Node API builtins through the type checker and into MLIR codegen. 3d5fc91e is the distributed reply channels — wire format extension, reply table, condvar wiring. 59956962 is the gossip extension for the cross-node registry.

Four commits, 72 hours. The QUIC transport was the big one; the reply channels the most interesting; the gossip extension mostly mechanical — two new event types over existing infrastructure. The type checker work was the frustrating one, for the reason they usually are: everything looks right until it isn’t, and the error is three layers below where you’re looking.