Remove 'prefix' field in favor of Token-level whitespace/comments?

[LPGhatguy]

Having to have an EndOfFile token is awkward. It might make more sense if the parser rules are a little smarter and comments and whitespace appeared in stream, next to tokens.

Tokens would have another layer added to them, something like:

enum Token<'a> {
    Semantic(ActualToken<'a>),
    Whitespace(Cow<'a, str>),
    Comment(Comment<'a>),
}

[Kampfkarren]

In From AST to Lossless Syntax Tree, the author has an alternative solution: tokens are assigned span IDs that are relevant to them. Alongside that, there is a vector of every span, which would include whitespace and comments.

In the code:

echo one  # comment
echo two

the result would look like:

(CommandList
  children: [
    (SimpleCommand
      words: [
        (CompoundWord parts:[(LiteralPart token:(token id:Lit_Chars val:echo span_id:0))])
        (CompoundWord parts:[(LiteralPart token:(token id:Lit_Chars val:one span_id:2))])
      ]
    )
    (SimpleCommand
      words: [
        (CompoundWord parts:[(LiteralPart token:(token id:Lit_Chars val:echo span_id:7))])
        (CompoundWord parts:[(LiteralPart token:(token id:Lit_Chars val:two span_id:9))])
      ]
    )
  ]
)

Spans (stored in an Arena):

    0 'echo'
    1 ' '
    2 'one'
    3 '  '
    4 '#'
    5 ' comment'
    6 '\n'
    7 'echo'
    8 ' '
    9 'two'
   10 '\n'
(11 spans)

[LPGhatguy]

Ooh dang, that's a good idea. I've heard of the idea of using a Vec (or IndexMap) approach for storing an AST, since it lets you assign additional data, but I didn't think of using it for styling information.

I think the holy grail here is probably to ditch a nested structure altogether and go all-in on using an ECS-like structure for the AST. I'd love to do that if I come back to Mab, maybe this idea will give me a shot of motivation!

[Kampfkarren]

How would entities and components look like in the context of a language parser in a way that wouldn't just end up being a tree?

[LPGhatguy]

It would just end up being a tree, but with every node stored in a flat Vec with entity IDs indicating references. You'd have a whole new big bag of problems (preventing cycles, pruning the tree) in exchange for being able to access nodes by ID in constant time.