Skip to content

Make position tracking code-point-aware for Unicode input #12

Description

@saiashirwad

Right now State.consume updates line/column by iterating over bytes in the source string:

for (let i = state.offset; i < end; i++) {
  if (state.source[i] === "\n") { ... }
  else { column++ }
}

That works for ASCII, but it's wrong for any non-ASCII text. A character like é (two bytes in UTF-8 / two UTF-16 code units in JS) or 👋 gets counted as multiple columns, so caret pointers and error locations drift relative to what the user actually sees.

We should iterate over code points instead of raw indices. Something like:

for (const ch of consumed) {
  if (ch === "\n") { line++; column = 1; }
  else { column++; }
}

This also ties into the Span overhaul in #X — once we have separate byteOffset and charOffset, the state update should populate both correctly.

I don't think we need full grapheme-cluster accuracy yet (that usually pulls in a dependency or heavy Intl usage), but code-point-aware columns would fix the vast majority of real-world Unicode issues.

Worth adding a test case with emoji and non-Latin text to lock this down.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions