tokener: clear pending surrogate on reset

Discard the previous document's high surrogate when resetting the tokener.
Otherwise the next Unicode escape can add a replacement character or
combine with a surrogate from the abandoned document.

Extend the existing incremental parse tests for incomplete input and a
fatal error, using independent next-document string expectations.

Fixes #984.
Reported-by: GeekQAQ (GitHub)
Assisted-by: OpenAI Codex (AI)
Reviewed-by: GPT-6 Luna (AI)
This commit is contained in:
qeqx committed 2026-10-10 20:06:11 +09:00
1 parent 9e99c1cfe7
commit 7662139501
3 files changed
+16 -1

No files matched your search

+8
View File
@@ -335,6 +335,14 @@ struct incremental_step
{"{ \"foo", -1, -1, json_tokener_continue, 1, 0},
{": \"bar\"}", -1, 0, json_tokener_error_parse_unexpected, 1, 0},
/* Reset must discard a pending high surrogate from the previous document. */
{"\"\\uD834", -1, -1, json_tokener_continue, 1, 0},
{"\"\\u0041\"", -1, -1, json_tokener_success, 1, 0},
{"\"\\uD834", -1, -1, json_tokener_continue, 1, 0},
{"\"\\uDD1E\"", -1, -1, json_tokener_success, 1, 0},
{"\"\\uD834\\uZZZZ\"", -1, 9, json_tokener_error_parse_string, 1, 0},
{"\"\\u0041\"", -1, -1, json_tokener_success, 1, 0},
/* Check a supplemental code point that looks like a high surrogate */
{"\"\\uD836", -1, -1, json_tokener_continue, 0, 0},
{"\\uDE87", -1, -1, json_tokener_continue, 0, 0},