Comments Are Not Optional

Someone ran hew fmt on a file and then diffed it against the original. The code was identical — but the comments were gone.

Not mangled or misplaced — just gone. Run the formatter on this:

wire type UserProfile {
    name: String @1;    // display name
    email: String @2;   // primary contact
    age: int @3;        // years, not months
}

#[wire]
type UserProfile {
    name: string @1;    // display name
    email: string @2;   // primary contact
    age: i32 @3;        // years, not months
}

Get back:

wire type UserProfile {
    name: String @1;
    email: String @2;
    age: int @3;
}

#[wire]
type UserProfile {
    name: string @1;
    email: string @2;
    age: i32 @3;
}

No warning. No error. Just silence where the comments were.

The formatter had been in use for several weeks before anyone ran it on a file with struct field comments and actually checked the output.

How the formatter handles comments

The formatter maintains a cursor into the comment stream. Before emitting each construct, it calls flush_comments_before(pos) to drain any comments whose source position sits before pos. The idea is that comments accumulate between constructs and get flushed at the right time.

The problem was scope. flush_comments_before was being called at the enclosing construct’s boundary — the opening and closing of a struct body, say — but not at each element boundary inside that body. A comment on a struct field sits at a position that’s after the struct opens and before the struct closes, so it never triggers a flush. The formatter has no idea it was there.

Once I understood the pattern, I started looking for how many places had the same gap. The answer was seven. Enum variants, struct fields, match arms, else-if branch headers, extern fn declarations, and two others. Every one had been added independently without a systematic flush discipline. (This is what “no systematic flush discipline” looks like in practice: the first five cases work fine, the sixth has no comments in the test fixtures, and the bug ships quietly.)

Fixing it

The first problem was that some AST nodes didn’t carry their source position. TypeBodyItem::Field and VariantDecl didn’t have Span fields, so even if the formatter wanted to call flush_comments_before(item.span.start), there was no span to read. Those fields went in first — then the formatter could call:

flush_comments_before(item.span.start)
prev_source_pos = item.span.start
format(item)
flush_comments_before(item.span.end)   // trailing comment on the same line

The two-call pattern is important. The first flush catches comments that appear before the item on their own line. The second flush catches a trailing comment on the same line as the item — the // display name in the example above sits after the field ends, not before the next one starts.

The else-if case was trickier. A comment between } and else if documents the next branch, but it’s written at the outer indent level:

if cond {
    body
// this describes the else branch
} else if other {
    other_body
}

Whether that comment belongs to the closing block or to the next branch depends on its column position relative to the closing brace. Comments at or shallower than the brace column get emitted at the outer indent — decremented by one relative to the body — rather than as block-interior comments. Getting that wrong would either drop the comment or mis-indent it.

Testing

Six fixture tests went in — one per construct type: struct fields, enum variants, match arms, else-if headers, extern fns, and an end-of-block trailing comment. The fixture format is before/after pairs; the formatter is expected to produce the after from the before. These would have caught the bug immediately if they’d existed before the formatter shipped. (They didn’t because the test coverage was written against constructs that didn’t have inline comments in the test fixtures. The formatter worked correctly on everything it was tested on.)

After the fixture tests passed, I ran hew fmt on four stdlib files — std/fs.hew, std/net/net.hew, std/net/quic/quic.hew, and std/net/websocket/websocket.hew — and diffed the results against the originals. Combined, those files had 47 inline comments that had been disappearing on every fmt invocation. They’re all there now.

The commits are 8a23aecc and closed issues #1529, #1534, and #1536.

LSP formatting

With the formatter correct, wiring textDocument/formatting in the LSP was straightforward. The handler calls the parser formatter, gets back the canonical source, and returns either an empty edit list (already canonical — the editor sees a successful response with no changes) or a single whole-document TextEdit.

The one detail worth calling out: LSP positions use UTF-16 code units, not bytes. Any file with non-ASCII content — a string literal with a non-ASCII character, a comment in another language, a filename path — will have positions that don’t match a byte offset. offset_range_to_lsp builds a line-offset table during document parse and translates through it. Getting this wrong produces a formatting edit that lands at the wrong position and corrupts the document.

That landed in ceb33114.