Parsing template with Twig 3 results in wrong Token type. #4756
Description
Hello, I think there has been some breaking change when parsing templates and how template is broken down into tokens.
Example I have this template with custom tags (mx_interests and mx_interest):
$template = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<title>Twig Test</title>
</head>
<body>
{% mx_interests %}
{% mx_interest [1] %}<p>Custom 0</p>{% end_mx_interest %}
{% mx_interest [] %}<p>Custom 1</p>{% end_mx_interest %}
{% end_mx_interests %}
</body>
</html>
';
$this->loader->setTemplate('template.html', $template);
$this->twig->render('template.html');
Parsing with Twig v3.23.0:
Parsing with Twig v2.16.1:
My question is, if the above template and tags are valid, why is there a difference in parsing? The token at position 6 with value "[" has type 9 (PUNCTUATION_TYPE) in Twig 2 which seems correct, but in Twig 3 it has type 8 (OPERATOR_TYPE) which seems wrong. Anyone could help me understand this? Could it be a bug? Or are we using a wrong syntax?