Returning an array of captures with IDs.

[zherczeg]

I saw an interesting request which made me thinking. While PCRE is perfectly capable of matching complex patterns, it could only return with a fixed set of captures. For example, it can parse a table with columns, but it cannot return with the column list, since the number of columns is not fixed.

The feature would look like this: (*push:<ID>PATTERN). The character range with its ID is pushed onto a stack. When the engine backtracks, the item is popped as well.

Since allocating new items may fail, this construct may break matching. By default, there is no stack, so using this construct always aborts the match. The user application must allocate the stack (with its allocation limit), and add it to match data. The cost is an extra pointer in match data for those applications, which does not use this feature.

What do you think?

[ltrzesniewski]

For information, the .NET regex engine does this by default for all groups:
https://learn.microsoft.com/en-us/dotnet/api/system.text.regularexpressions.group.captures

[NWilson]

That is interesting, and has a good use-case.

Instead of adding a new (*push:...) we could make it a new type of capture group. For example, (?+PATTERN) and (?+<name>PATTERN) would create list-based capture groups. Each time it's captured it pushes a new item onto the start of a linked list of captures; backtracking pops. \1 would match against the top of the stack (if capture group "1" has been designated a list-based capture group).

Then we can add new methods on the match_data to allow a user to access the rest of the stack via a PCRE2-specific API.

Alternatively, instead of a new piece of syntax, it could be a compile flag: set this flag, and all capture groups become stack-based.

There could be new kinds of conditional expression, or pattern assertions, to test how many times a group has been captured (once? more than once? N times?) or to do a backreference against a previously-captured value, such as \first{1} or something like that, or \g{1[-3]}.

This compile flag would match the .NET behaviour, which records the full history of all captures:

https://learn.microsoft.com/en-us/dotnet/api/system.text.regularexpressions.capturecollection?view=net-9.0

[zherczeg]

If other engines has such features, it is definitely worth thinking about it. The chain list is also a good idea, after a successful match it could be extracted if requested. Since it require an extra effort for the engine, it should not be done automatically, only if needed.