Right now State.consume updates line/column by iterating over bytes in the source string:
for (let i = state.offset; i < end; i++) {
if (state.source[i] === "\n") { ... }
else { column++ }
}
That works for ASCII, but it's wrong for any non-ASCII text. A character like é (two bytes in UTF-8 / two UTF-16 code units in JS) or 👋 gets counted as multiple columns, so caret pointers and error locations drift relative to what the user actually sees.
We should iterate over code points instead of raw indices. Something like:
for (const ch of consumed) {
if (ch === "\n") { line++; column = 1; }
else { column++; }
}
This also ties into the Span overhaul in #X — once we have separate byteOffset and charOffset, the state update should populate both correctly.
I don't think we need full grapheme-cluster accuracy yet (that usually pulls in a dependency or heavy Intl usage), but code-point-aware columns would fix the vast majority of real-world Unicode issues.
Worth adding a test case with emoji and non-Latin text to lock this down.
Right now
State.consumeupdatesline/columnby iterating over bytes in the source string:That works for ASCII, but it's wrong for any non-ASCII text. A character like
é(two bytes in UTF-8 / two UTF-16 code units in JS) or👋gets counted as multiple columns, so caret pointers and error locations drift relative to what the user actually sees.We should iterate over code points instead of raw indices. Something like:
This also ties into the Span overhaul in #X — once we have separate
byteOffsetandcharOffset, the state update should populate both correctly.I don't think we need full grapheme-cluster accuracy yet (that usually pulls in a dependency or heavy
Intlusage), but code-point-aware columns would fix the vast majority of real-world Unicode issues.Worth adding a test case with emoji and non-Latin text to lock this down.