tokener: clear pending surrogate on reset

Discard the previous document's high surrogate when resetting the tokener.
Otherwise the next Unicode escape can add a replacement character or
combine with a surrogate from the abandoned document.

Extend the existing incremental parse tests for incomplete input and a
fatal error, using independent next-document string expectations.

Fixes #984.
Reported-by: GeekQAQ (GitHub)
Assisted-by: OpenAI Codex (AI)
Reviewed-by: GPT-6 Luna (AI)
This commit is contained in:
qeqx committed 2026-10-10 20:06:11 +09:00
1 parent 9e99c1cfe7
commit 7662139501
3 files changed
+16 -1

No files matched your search

+7 -1
View File
@@ -140,6 +140,12 @@ json_tokener_parse_ex(tok, "ä" , 4) ... OK: got object of type [string
json_tokener_parse_ex(tok, "ä" , 4) ... OK: got object of type [string]: "ä"
json_tokener_parse_ex(tok, { "foo , 6) ... OK: got correct error: continue
json_tokener_parse_ex(tok, : "bar"} , 8) ... OK: got correct error: unexpected character
json_tokener_parse_ex(tok, "\uD834 , 7) ... OK: got correct error: continue
json_tokener_parse_ex(tok, "\u0041" , 8) ... OK: got object of type [string]: "A"
json_tokener_parse_ex(tok, "\uD834 , 7) ... OK: got correct error: continue
json_tokener_parse_ex(tok, "\uDD1E" , 8) ... OK: got object of type [string]: "�"
json_tokener_parse_ex(tok, "\uD834\uZZZZ", 14) ... OK: got correct error: invalid string sequence
json_tokener_parse_ex(tok, "\u0041" , 8) ... OK: got object of type [string]: "A"
json_tokener_parse_ex(tok, "\uD836 , 7) ... OK: got correct error: continue
json_tokener_parse_ex(tok, \uDE87 , 6) ... OK: got correct error: continue
json_tokener_parse_ex(tok, " , 1) ... OK: got object of type [string]: "�"
@@ -388,5 +394,5 @@ json_tokener_parse_ex(tok, {"":1} , 7) ... OK: got correct error: invalid
json_tokener_parse_ex(tok, {"":1} , 7) ... OK: got correct error: invalid string sequence
json_tokener_parse_ex(tok, {"":1} , 7) ... OK: got correct error: invalid string sequence
json_tokener_parse_ex(tok, {"":1} , 7) ... OK: got correct error: invalid string sequence
End Incremental Tests OK=292 ERROR=0
End Incremental Tests OK=298 ERROR=0
==================================