This document is normative for the P-374/S-205 baseline. It gives compiler-oriented token rules; cooked literal values and static typing are defined by the consolidated specification.
The notation uses quoted terminals, character ranges, | for alternatives,
? for optional elements, * for repetition, and + for one-or-more.
Source is valid, unnormalized UTF-8. U+0000 is forbidden. A single U+FEFF byte order mark is accepted only as the first code point of a non-shebang file. The only lexical whitespace outside comments and literals is:
space = U+0020
tab = U+0009
carriage = U+000D
newline = U+000A
whitespace = space | tab | carriage | newline
Only U+000A is a newline. U+000D is whitespace. Newlines do not insert semicolons. A missing final newline is accepted; canonical formatting emits exactly one.
A shebang is recognized only at byte offset zero:
shebang = "#!" { any-code-point-except-U+000A } (U+000A | EOF)
No BOM or whitespace may precede it.
line-comment = "//" { any-code-point-except-U+000A } (U+000A | EOF)
block-comment = "/*" { block-comment | non-delimiter-code-point } "*/"
Block comments nest. EOF does not close an unterminated block comment.
There are no distinct documentation-comment tokens. A consecutive standalone
group of ordinary // comments documents the immediately following
declaration. A blank line or intervening token breaks attachment.
ascii-letter = "A"…"Z" | "a"…"z"
decimal-digit = "0"…"9"
identifier-start = ascii-letter | "_"
identifier-rest = ascii-letter | decimal-digit | "_"
identifier = identifier-start identifier-rest*
Identifiers are case-sensitive and are not normalized. _ is a dedicated
discard/wildcard token; a longer underscore-prefixed spelling is an ordinary
identifier.
The globally reserved words are:
package import pub const let mut fn self struct enum type opaque
resource cleanup protocol conform if else match for in break continue
return spawn select case default arena recover with native offload true false
map and set are contextual collection-literal introducers before {; they
are ordinary identifiers elsewhere. as is contextual inside an import.
The lexer uses longest-match selection within this fixed vocabulary:
( ) { } [ ] , ; : . :: => #[ @ ? .. ..=
+ - * / % & | ^ &^ << >> == != < <= > >= ! && || =
+= -= *= /= %= &= |= ^= &^= <<= >>=
Whitespace may not split a multi-character token. # is otherwise invalid;
backslash has no meaning outside a literal.
digit-sequence = decimal-digit ("_"? decimal-digit)*
binary-digits = ("0" | "1") ("_"? ("0" | "1"))*
octal-digits = ("0"…"7") ("_"? ("0"…"7"))*
hex-digit = decimal-digit | "A"…"F" | "a"…"f"
hex-digits = hex-digit ("_"? hex-digit)*
decimal-integer = "0" | ("1"…"9") ("_"? decimal-digit)*
binary-integer = "0b" binary-digits
octal-integer = "0o" octal-digits
hex-integer = "0x" hex-digits
integer-literal = decimal-integer | binary-integer | octal-integer | hex-integer
exponent = ("e" | "E") ("+" | "-")? digit-sequence
float-literal = digit-sequence "." digit-sequence exponent?
| digit-sequence exponent
Non-decimal prefixes are lowercase. Numeric literals have no suffix. A sign is a separate unary operator. Multi-digit decimal integers beginning with zero are invalid.
A rune literal contains exactly one Unicode scalar written directly or by one recognized escape:
rune-literal = "'" (unicode-scalar-except-quote-backslash-newline | rune-escape) "'"
rune-escape = "\\" ("a"|"b"|"f"|"n"|"r"|"t"|"v"|"\\"|"'")
| "\\" octal-digit octal-digit octal-digit
| "\\x" hex-digit hex-digit
| "\\u" hex-digit{4}
| "\\U" hex-digit{8}
Escapes denoting a surrogate or value above U+10FFFF are invalid.
A byte literal is prefixed by adjacent b and admits exactly one ASCII byte or
single-byte escape:
byte-literal = "b'" (ascii-byte-except-quote-backslash-newline | byte-escape) "'"
byte-escape = "\\" ("a"|"b"|"f"|"n"|"r"|"t"|"v"|"\\"|"'")
| "\\" octal-digit octal-digit octal-digit
| "\\x" hex-digit hex-digit
The decoded value must fit 0…255. Unicode escapes and literal non-ASCII
characters are invalid in byte literals.
An interpreted string is delimited by ". It cannot contain a physical
newline. Its escapes are:
\\a \\b \\f \\n \\r \\t \\v \\\\ \\" \\uHHHH \\UHHHHHHHH \\${
Octal and \x byte escapes are invalid in strings. ${ begins interpolation;
its restricted slot grammar is defined in
Syntactic grammar. Interpreted strings cannot nest
inside interpolation slots.
A raw string is delimited by backticks. It has no escapes or interpolation, may contain physical newlines, and cannot contain a backtick. U+000D is discarded from its cooked value so LF and CRLF source produce the same string.
The lexer must be lossless over accepted UTF-8 source and must produce stable half-open byte ranges. Invalid source must produce bounded diagnostics and make forward progress. Parser recovery cannot reinterpret a lexically invalid token as valid syntax.