Add support for non-native-endian UTF-16 and UTF-32

[Rot127]

It would be great if the endianess of the input buffer could be changed for each match().
For our use case we can have strings which come in little and big endian encoding and we must support both.
The encoding normalization to UTF-8 naturally eats a lot of runtime.

Having this build into PCRE2 would be a blessing.

I am aware that the docs say:

UTF-16 and UTF-32 strings can indicate their endianness by special code knows as a byte-order mark (BOM).
The PCRE2 functions do not handle this, expecting strings to be in host byte order.

But would it be a possible extension? Or is it simply utopic because too complicated to implement?

[zherczeg]

The PCRE2 sources are compiled multiple times even now (with different configurations), so this could be supported without affecting the host endian support. The compiled functions would get another suffix. However, it is not a small amount of work. Not just the code, but testing and documentation as well.

[NWilson]

Normalising to UTF-8 is clearly going to be expensive.

But what you actually have is presumably UTF-16 or UTF-32, and you want to avoid having to byte-swap the input between UTF-16-BE and UTF-16-LE?

I'm curious what system you have, which actually processes UTF-16 or UTF-32 and does NOT use little-endian.

I would say that UTF-16-BE and UTF-32-BE are virtually non-existent in the wild, and that when they are encountered, you can do a pretty quick byte-swapping pass over the input before passing it to PCRE2. (Without SSE - it would be uint64_t *p; *p = ((*p & 0x00ff00ff00ff00ff) << 8) | ((*p & 0xff00ff00ff00ff00) >> 8); ++p; or something like that.)

To put it another way: running a regex is an expensive operation, far more time-consuming than the byte-swapping. Hence, you can accept the cost of converting to native-endian input, if you're willing to use a regex at all.

Adding the non-native-endian versions of UTF-16 and UTF-32 is not likely to happen for these reasons.

I hate to close a legitimate feature request, but this one does seem like it would useful to a very limited number of users.

[Rot127]

We use it for our reverse engineering framework and have to enable regex search in those rarer big endian binaries.
Potentially also for very big ones (1GB+), so swapping the whole input buffer (memcpy essentially) and passing it on is possible. Compared to PCRE2 matching directly against big endian this would perform badly in these more extreme cases though.

Our use case is pretty niche.

And I will go with the option to byte swap a few code points, match partially, and skip them if they don't match already.

[PhilipHazel]

Because of backtracking, PCRE2 may read the same characters a number of times while matching. If it were to do byte-swapping on the fly, it could be less efficient than a byte-swapping prepass (as @NWilson suggests). On the other hand, if a match is to occur at the start of a very long string, the opposite may be true. However, I fully agree that this would be a large amount of work for a very small number of use cases.

[Rot127]

Thanks for the feedback.
I guess it can be closed as won't fix then?

[NWilson]

That is interesting. So you have big-endian binaries (SPARC? POWER?), AND they have wide-character strings in UTF-16 or UTF-32. You want to do a regex search on them, on a little-endian ("normal") host machine.

Currently, PCRE2 is compiled three times, with macro definitions switched, to target UTF-8/16/32. In theory, you or we could add two non-native-endian modes. It just seems very unlikely that it would be of use to many people, especially as there's a reasonably-efficient workaround of converting the input.

If it's a 1GB binary, byte-swapping that in 64-bit chunks will still be quicker than PCRE2 scanning it, right (assuming no match)? So the only thing you're trying to optimise is: if there's a match early on, and PCRE2 matching wouldn't scan the whole input, you would get a speed-up if you didn't have to byte-swap it all.

[Rot127]

That is interesting. So you have big-endian binaries (SPARC? POWER?), AND they have wide-character strings in UTF-16 or UTF-32. You want to do a regex search on them, on a little-endian ("normal") host machine.

It doesn't need to be a binary for any specific architecture. It might be some other file with binary data with big endian strings. Since we don't know what people might throw at it on what machine, we at least want some support.
One plausible scenario could be some malware which encodes its strings in the opposite endianness of the target to evade detection for example.

So the only thing you're trying to optimize is: if there's a match early on, and PCRE2 matching wouldn't scan the whole input, you would get a speed-up if you didn't have to byte-swap it all.

The requirement I have is basically the following:

INPUT: a buffer of size x + the string encoding like (UTF-8/16-le/16-be/32-le/32-be, EBCDIC, IBM037 and others) + the regex pattern

OUTPUT: Any string matching the regex of length <= x, overlapping matches are optionally allowed, optionally the strings have to be aligned.

We do not have any additional information about the buffer at hand. Because our tool (Rizin) also exposes the search as a library. So people can really throw any data into it.

For endianness insensitive encodings this is no problem. But I'd like to reduce the additional memory and runtime increase for the foreign endianness encodings (which are only UTF-16/32 at the time).

Doing partial matching would prevent twice as much memory usage at a given time.
Which is of course only required for large files. But as said, if people use it as a library it maybe gets buffers of this size.

[NWilson]

The requirement for nonaligned strings is also quite tricky. Our UTF 16/32 support requires aligned input, so you'd have to copy the whole input at each offset as well. Ouch.

[Rot127]

Our UTF 16/32 support requires aligned input, so you'd have to copy the whole input at each offset as well. Ouch.

Yeah, I've just ran into this as well :D
For utf32/16 I would ignore this requirement for now and only allow for alignments from 2 or 4 upwards.

[zherczeg]

I remember we started with only 8bit byte and utf8 support, and eventually added more modes including invalid utf support. The request is reasonable, so it might happen at some point. But we are a small voluntary team, with limited resources.

[Rot127]

including invalid utf support.

This one is invaluable to me as well. Thanks for adding it!

But we are a small voluntary team, with limited resources.

Part of two open source projects myself. I can relate to this a lot :)
With all the discussion here I got already more advice than I expected.

Besides that, congrats to your folks. PCRE2 is really an excellent library. Everything is so much on point, I love it!