fix(tokenizer): preserve is_escaped across the SIMD string scan in streaming mode

findStringEnd's SIMD scanners (AVX2/SSE2/NEON) take is_escaped by
reference and mutate it across the whole remaining buffer. When they do
not find the closing quote (consumed == remaining) the function falls
through to the scalar scan, which restarts from cursor_index -- but
is_escaped now held the SIMD end-of-buffer state, so re-running the escape
state machine flipped escape parity for a string crossing a streaming
buffer boundary. A string whose escaped quote (\") straddled a buffer
split then had the escaped quote treated as the terminator (wrong value /
ExpectedDelimiter).

Snapshot is_escaped before the SIMD scan and restore it before the scalar
fallback, so the fallback re-derives escape state from the correct
starting value. The snapshot/restore is compiled only when a SIMD path
exists (no unused-variable warning under -Werror), and the common path
(quote found -> early return) is unaffected. Rejected the originally
proposed "return NeedMoreData" fix, which would wrongly report a complete
string whose closing quote is the final buffer byte as incomplete.

Added a streaming regression test (verified: fails without the fix with
ExpectedDelimiter, passes with it). All 1648 unit-test assertions pass.

Author: jorgen

include/json_struct/json_struct.h

@@ -1922,6 +1922,14 @@ JSON_STRUCT_FORCE_INLINE void Tokenizer::resetForNewValue()
 
 JSON_STRUCT_FORCE_INLINE Error Tokenizer::findStringEnd(const DataRef &json_data, size_t *chars_ahead)
 {
+#if defined(JSON_STRUCT_HAS_AVX2) || defined(JSON_STRUCT_HAS_NEON) || defined(JSON_STRUCT_HAS_SSE2)
+  // The SIMD scan below mutates is_escaped across the whole remaining buffer. When
+  // it does not find the closing quote we fall through to the scalar scan starting
+  // again from cursor_index, which must re-derive escape state from the ORIGINAL
+  // value -- re-running the state machine from the SIMD end-state would flip escape
+  // parity for a string crossing a streaming buffer boundary. Snapshot and restore.
+  const bool escaped_before_simd = is_escaped;
+#endif
 #ifdef JSON_STRUCT_HAS_AVX2
   if (JSON_STRUCT_LIKELY(json_data.size - cursor_index >= 32))
   {
@@ -1958,6 +1966,12 @@ JSON_STRUCT_FORCE_INLINE Error Tokenizer::findStringEnd(const DataRef &json_data
   }
 #endif
 
+#if defined(JSON_STRUCT_HAS_AVX2) || defined(JSON_STRUCT_HAS_NEON) || defined(JSON_STRUCT_HAS_SSE2)
+  // Restore the escape state to what it was before the SIMD scan re-mutated it, so
+  // the scalar fallback re-derives it correctly from cursor_index.
+  is_escaped = escaped_before_simd;
+#endif
+
   size_t end = cursor_index;
   JSON_STRUCT_PREFETCH(json_data.data + end + 64);
 

tests/json-struct-optimization-fixes.cpp

@@ -134,6 +134,42 @@ struct MultiInt
   JS_OBJ(a, b, c);
 };
 
+TEST_CASE("streaming_escape_parity_across_buffer_boundary", "[tokenizer]")
+{
+  // A string value whose escaped quote (\") straddles a buffer boundary: the
+  // first buffer ends on the backslash (is_escaped carried over), the second
+  // buffer starts with the escaped quote and is long enough (>=16 bytes) to hit
+  // the SIMD scan with no real closing quote. Previously the SIMD scan left
+  // is_escaped in its end-of-buffer state and the scalar fallback re-scanned from
+  // the start with the wrong parity, treating the escaped quote as the terminator.
+  const char BS = static_cast<char>(0x5c); // backslash
+
+  std::string b1 = std::string("[\"aaaaaaaa") + BS; // '[' '"' 8*a '\'
+  // >= 32 bytes so both the AVX2 (>=32) and SSE2 (>=16) SIMD scans engage.
+  std::string b2 = std::string("\"") + std::string(40, 'b'); // escaped quote + 40*b, no real quote
+  std::string b3 = std::string("\"]");                       // real closing quote + ']'
+
+  JS::Tokenizer tok;
+  tok.addData(b1.data(), b1.size());
+  tok.addData(b2.data(), b2.size());
+  tok.addData(b3.data(), b3.size());
+
+  JS::Token token;
+  REQUIRE(tok.nextToken(token) == JS::Error::NoError);
+  REQUIRE(token.value_type == JS::Type::ArrayStart);
+
+  JS::Error e = tok.nextToken(token);
+  REQUIRE(e == JS::Error::NoError);
+  REQUIRE(token.value_type == JS::Type::String);
+  // The raw token value keeps the escape sequences; the escaped quote must be
+  // part of the string, not a premature terminator.
+  std::string expected = std::string("aaaaaaaa") + BS + '"' + std::string(40, 'b');
+  REQUIRE(std::string(token.value.data, token.value.size) == expected);
+
+  REQUIRE(tok.nextToken(token) == JS::Error::NoError);
+  REQUIRE(token.value_type == JS::Type::ArrayEnd);
+}
+
 struct NestChild
 {
   int a = 0;