Discard the previous document's high surrogate when resetting the tokener.
Otherwise the next Unicode escape can add a replacement character or
combine with a surrogate from the abandoned document.
Extend the existing incremental parse tests for incomplete input and a
fatal error, using independent next-document string expectations.
Fixes#984.
Reported-by: GeekQAQ (GitHub)
Assisted-by: OpenAI Codex (AI)
Reviewed-by: GPT-6 Luna (AI)
Strict mode rejected leading zeros but nothing else about the shape of a
number, so several malformed numbers parsed and were then written back out
verbatim, producing JSON that other parsers reject:
$ echo '[2.e3]' | ./json_parse -s -
[ 2.e3 ]
RFC 8259 requires a mandatory integer part and at least one digit in both the
fraction and the exponent:
number = [ minus ] int [ frac ] [ exp ]
int = zero / ( digit1-9 *DIGIT )
frac = decimal-point 1*DIGIT
exp = e [ minus / plus ] 1*DIGIT
so "1.", "-2.", "2.e3", "2.e+3", "0.e1", "-.123", "1e" and "1e+" are all
invalid. Walk the accumulated text against that grammar instead. The leading
zero rule from the previous check is part of the same walk rather than a
separate test, and its cases keep their coverage.
Only JSON_TOKENER_STRICT is affected; the default tokener stays as lenient as
it was. Measured against JSONTestSuite (318 files): strict mode went from 44
to 37 files accepted that the suite says must be rejected, default mode stayed
at 71, and nothing that must be accepted regressed in either mode.
This causes previously failing strings like "123-456" to return a valid json_object with the appropriate value. If you care about the trailing content, call json_tokener_parse_ex() and check the parse end point with json_tokener_get_parse_end().
Add an optional way to use an incremental chunk size ($TEST_PARSE_CHUNKSIZE)
for all of the single_basic_parse tests, in additional to the regular way.
Some badly formated "numbers" could get partly parsed,
resulting in truncated results instead of raising an
error.
Examples :
'1.2.3' -> (double)1.2
'2015-01-15' -> (int)2015
This patch is not perfect (ex: input can still end with a 'E', which
is forbidden by json.org doc) but should avoid non-sensically
formated input.
Tests added.
qeqx
Denis Gregor
Javid Khan
Shane F. Carr
Eric Hawicz
dota17
Jan Gerhards
chl
Iskren Chernev