Repository navigation
Add public on-demand parsing, allocation-free JsonValue navigation and optional UTF-8 validation; fix #70 - #81
Open
morrismeyer wants to merge 9 commits into
Open
morrismeyer wants to merge 9 commits into
morrismeyer wants to merge 9 commits into
Conversation
added 9 commits
November 21, 2026 20:40
Gradle 9 no longer sets testClassesDirs and classpath for Test tasks registered by the build, so test256 and test512 reported NO-SOURCE and ran no tests at all. Point both tasks at the test source set, as the built-in test task is.
Parsing an integer literal such as 5 into a float, Float, double or Double failed with "Invalid floating-point number. Fraction or exponent part is missing.", although it is a valid JSON number and real documents mix 5 and 5.0 for the same field. NumberParser.parseFloat/parseDouble now pass integer literals to the existing FloatParser/DoubleParser, which already round them exactly (fast path, Eisel-Lemire, and the slow path for more than 19 significant digits). All other validation is unchanged. The test integralNumberAsFloatingPointNumber asserted the old rejection; it is replaced by tests that expect Float.parseFloat/Double.parseDouble results at the root, in object fields, in arrays and in lists, plus tests that invalid integer literals are still rejected. Fixes simdjson#70.
parseOnDemand runs stage 1 only and returns the parser's OnDemandJsonIterator, a forward-only cursor that decodes values when the caller asks for them and skips everything else, without building a tape. This lets callers that need a few fields of a large document read them without allocating per value. The cursor is the iterator that schema-based parsing already uses. Its navigation methods and its string, number and boolean getters become public, and it gains: - peekType(), the JSON type of the next value, taken from its first byte - peekOffset() and lastTokenOffset(), the byte span of a value - getRawNumber(byte[]), the verbatim number literal - STRING_BUFFER_PADDING, the room getString and getFieldName may write past the decoded string - a check that nothing follows the root object or array once it has been read to its end The class documentation describes the lifecycle, the string buffer contract and how getDepth() and skipChild(int) skip the rest of a container.
Extracting a few fields from many values of a large document with
get(String) and the iterators allocates a JsonValue per value, an
iterator per array and a byte[] per key. JsonValue gains:
- get(byte[]), get(String) with a pre-encoded name
- index(), field, firstElement, advance and scopeEnd, which navigate the
tape of the whole document with int indices
- is{Object,Array,String,Long,Double,Boolean,Null}At type tests
- stringAt, longAt, doubleAt, numberAt and booleanAt getters
- stringBytes, byteOffset, byteLength and stringEquals, which read
decoded strings straight from the parser's string buffer
Methods that need a value of a particular type throw an
IllegalArgumentException for an index of another type, so a wrong
index cannot read unrelated tape entries.
Every parse validates its input as UTF-8 before indexing it. When the input is known to be valid, for example because it was produced or already checked by a trusted component, that pass is redundant. The new constructor SimdJsonParser(capacity, maxDepth, validateUtf8) creates a parser that skips it when validateUtf8 is false; invalid UTF-8 then goes undetected. The existing constructors keep validating.
SyntheaBundleBenchmark extracts a fixed set of fields from a corpus of Synthea FHIR bundles with each way of reading a document: domGetString (get(String) and iterators), domGetBytes (get(byte[])), domIndex (index-based JsonValue navigation), onDemand (parseOnDemand), schemaBased (records), and domParseOnly as the parsing baseline. Every extraction mode folds the same values into a checksum, and the trial setup refuses to measure if the modes disagree on the first files. SyntheaBundleVerifier runs all extraction modes over the whole corpus and reports whether they agree. No data is shipped: the corpus is a directory of generated bundles selected with -Dsynthea.dir, and -Dsynthea.validateUtf8=false selects parsers that skip UTF-8 validation.
Add README sections with short examples for parseOnDemand, the index-based JsonValue methods and SimdJsonParser(capacity, maxDepth, validateUtf8), and describe how to run the Synthea benchmark and verifier against a locally generated corpus.
skipChild looked up the depth change of every skipped token in a 127-entry table indexed with the token's first byte as a signed byte. A skipped object or array containing a malformed value that starts with 0x7F or a non-ASCII byte, such as an unquoted letter with an accent or, with UTF-8 validation off, an invalid UTF-8 byte, made it throw ArrayIndexOutOfBoundsException. The table now has 256 entries and is indexed with the unsigned byte, so such values are skipped like any other malformed scalar inside a skipped value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request makes simdjson-java practical for selective extraction from large real-world documents. It grew out of loading 1.1 million Synthea FHIR patient bundles (JSON files of 70 KB to 66 MB), where the cost was dominated by building the tape and by per-node allocation for fields that were never read.
It contains three additions, three fixes found on the way, and a benchmark:
SimdJsonParser.parseOnDemand): stage 1 only, no tape, a forward-only cursor that decodes the values the caller asks for and skips the rest. This is the iterator the schema-based parser already uses internally, made public with a documented contract. Relates to Lazy Parsing #33 and API #7.JsonValue:get(byte[])with a pre-encoded key, and index-based access (field,firstElement,advance, typed getters) that walks the tape without creating aJsonValueor an iterator per node. This overlaps in intent with JsonValue to support gc free parsing and adding benchmark #76; I am happy to rework it on top of that PR or drop it if JsonValue to support gc free parsing and adding benchmark #76 is preferred.validateUtf8, for input that is already known to be valid. Validation stays on by default.float/double. Real documents mix5and5.0in the same field; FHIRvalueQuantity.valuedoes, so schema-based parsing failed on the first Synthea bundle.skipChildindexed a 127-entry table with a signed byte, so a skipped value starting with a byte above 0x7E threwArrayIndexOutOfBoundsException. This is reachable onmainthrough schema-based parsing of an unknown field.jmhJardid not build on JDK 25, andtest256/test512ran zero tests on Gradle 9 (NO-SOURCE).Each of these is its own commit and can be reviewed or merged alone. If you would rather have separate pull requests, say which split you prefer and I will open them.
Results
256 Synthea FHIR R4 bundles, 954,637,091 bytes, held in memory; one benchmark operation is one pass over all of them, extracting the same fields from every resource in every mode (
resourceType,id, and a few fields per Patient, Encounter, Observation and Condition). AMD EPYC 7313 (AVX2, 256-bit vectors), Ubuntu OpenJDK 25.0.5.1, JMH with 2 forks, 3 warm-up and 5 measurement iterations,-prof gc. "Baseline" ismain(a4044aa) with only the benchmark added.Single thread:
main: DOM +get(String)get(byte[])validateUtf8 = falseget(byte[])alone nearly halves allocation, with a time difference inside the error bars.mainand 924.9 ms on this branch.Sixteen threads on one socket (each thread makes its own pass; per-thread throughput). These runs are noisy on this machine (run-to-run spread of 13 to 30%), so read them as indicative:
main: DOM +get(String)The advantage of on-demand grows under memory pressure (about 2.9x here), which matches what the loader showed in production: the work is bound by memory traffic, and not building a tape removes most of it.
Correctness of the comparison
SyntheaBundleVerifierruns every mode over the corpus and fails unless all of them produce the same checksum and the same per-type resource counts; the benchmark's trial setup refuses to measure if they disagree. All modes agree on 899 bundles (3.69 GB, including a 66 MB document), with validation on and off.No data is added to the repository. The corpus directory is given at run time:
Behaviour changes to review
FloatingPointNumberSchemaBasedParsingTest.integralNumberAsFloatingPointNumberasserted that123is rejected for floating-point targets. It is replaced by tests that expect theFloat.parseFloat/Double.parseDoublevalue for integers (including -0, 2^53 + 1,Long.MAX_VALUEand a 30-digit integer) and that invalid literals are still rejected. The DOM path is untouched.{"a":1}{"b":2}was accepted by the on-demand iterator once the root closed; it now fails with the existing "More than one JSON value at the root" error. Schema-based parsing already made this check.skipChildnever validated the scalars it skips ([x]is skipped without error onmain). The fix keeps that behaviour and only removes the array-index exception for bytes above 0x7E; such a value is still rejected when it is read. Rejecting at skip time is possible without slowing the normal case if you prefer it.JsonValuemethods throwsIllegalArgumentException(the document parsed correctly; the index argument is wrong).Contracts the new API documents and tests
getString/getFieldNamewrite whole SIMD vectors, so the destination needsOnDemandJsonIterator.STRING_BUFFER_PADDINGbytes beyond the decoded length. A test pins both sides of that bound.getDepth()/skipChild(int)(used to leave the object or array the caller is inside) is documented with an example and pinned for objects and arrays.Testing
./gradlew clean test test256: 2338 tests in each task, no failures (1961 before this branch; 12 removed by the On-demand double parsing breaks on integer values #70 change, 389 added).test512passes on an AVX2-only CPU through the non-intrinsic fallback; I have no AVX-512 hardware, so the 512-bit paths have not been measured.Commits