[BREAKING] Package rewrite for a 1.0 release

[quinnj]

This is a proposal to overhaul the package internals, providing mostly the same high-level interfaces (JSON.parse, JSON.json, etc) while also adding a number of enhancements (lazy parsing support, typed materialization similar to JSON3).

My goal is to deprecate JSON3.jl in favor of the simpler, faster, and overall better functionality as implemented in this PR.

I've reviewed all existing open issues/PRs to both this package and JSON3.jl, marking all that will be fixed or resolved with the functionality as proposed (over 70 issues resolved in total!).

I've tried to provide extensive new documentation (in the new docs/ directory) on all proposed functionality, including a migrate.md file that provides a detailed migration guide for both JSON.jl pre-1.0 users and JSON3.jl users.

My hope and aim is that the net upgrade for the vast majority of JSON pre-1.0 users will simply be adding JSON = "1" in their compat files. I've tried to support as much as possible in what I could tell was part of the JSON.jl public API. A few things were deliberately left out as my impression is they were either part of an ancient API that can be better served by more modern Julia mechanisms, or just seem not very useful. If there are things we discover feel too breaking, I'm definitely open to try and find ways to add backwards compatibility.

Another part of my commitment in this proposal, if/when merged and released, is to take time across the ecosystem to help upgrade packages. I've done this for other packages (CSV.jl, DataFrames.jl, HTTP.jl, etc.) with big releases, and I feel it helps with a new release "momentum" to get a bunch of key packages upgraded on the new release.

This proposal does add another dependency on the newish StructUtils.jl package and I'll try to provide a little context/justification. I've actually been working on the core refactor of this code and StructUtils.jl for around 2 years. The ideas and redesign came from architectural pains/complexities in JSON3.jl and StructTypes.jl, and a desire to find an internal framework that was at least as powerful/flexible as what ST + J3 provided, but in a vastly simpler way (the JSON3.jl code kept growing in complexity and became hard to work with/maintain/improve). I believe the design of StructUtils.jl to be much less invasive and natural when working with Julia structs, while the power of the core functional parsing methods provide a clean integration for handling 1) default materialization from JSON, 2) typed materialization, and 3) simple serialization from structs and a variety of other types. What's more, is the StructUtils.jl machinery generalizes easily beyond just JSON, and I already have other unreleased packages that use it for database model interactions, struct diffing, and a number of other use-cases.

As for review, I recognize that huge PRs like this are very hard to break down and see the exact net changes. In this case, the entire package is pretty much rewritten, so I'd recommend checking out the PR locally, testing out code, and reviewing the entire package/files in that way (i.e. trying to look at any kind of "diff" on github isn't probably that useful).

I'd also like to propose a public "community review" time where I would host a google meet call that would be open to anyone interested and I would take time to go over package internals/architecture and help answer questions/concerns in person. If folks are interested, please ping me in the public Julia slack or comment here and I'll arrange a time that works for those interested.

[codecov]

Codecov Report

❌ Patch coverage is 88.77863% with 147 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.78%. Comparing base (55a1bca) to head (7e1bf81).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
src/parse.jl	71.54%	70 Missing ⚠️
src/write.jl	89.34%	34 Missing ⚠️
src/lazy.jl	91.75%	32 Missing ⚠️
ext/JSONArrowExt.jl	86.79%	7 Missing ⚠️
src/JSON.jl	93.93%	2 Missing ⚠️
src/utils.jl	98.65%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #374       +/-   ##
===========================================
- Coverage   99.35%   88.78%   -10.57%     
===========================================
  Files           7        7               
  Lines         466     1311      +845     
===========================================
+ Hits          463     1164      +701     
- Misses          3      147      +144     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[oscardssmith]

possibly inf_string? Also do we need ninf or can it just print as a - before inf?

[oscardssmith]

also, should this be allownonfinite?

[quinnj]

JSON3.jl had allow_inf, but JSON.jl already had allownan, so I opted to keep that for backwards compat.

[tecosaur]

I do get the backwards compat, but as a potential consumer of the API I will say that allownan makes me think that the keywords is only for NaN, not Inf.

[tecosaur]

Could it be worth checking how many packages depend on the current JSON.jl to estimate how breaking any "breaking changes" might be in practice?

[ericphanson]

Tiny detail but these methods don’t close the file handle:

JSON.jl/src/parse.jl

Line 172 in c9d0b1b

parsefile(file; jsonlines::Union{Bool, Nothing}=nothing, kw...) = parse(open(file); jsonlines=(jsonlines === nothing ? isjsonl(file) : jsonlines), kw...)

[quinnj]

Interesting situation w/ the Documentation job failure here: Documenter depends on JSON, but doesn't have compat w/ 1.0, so it can't install. Would we need to have Documenter.jl pre-emptively support 1.0? Other solutions?

[KristofferC]

JSON ends up being a dependency in most environments. Is there any significant precompile / load times changes with this PR?

[quinnj]

JSON ends up being a dependency in most environments. Is there any significant precompile / load times changes with this PR?

# 1.0 precompile:
julia> @time using JSON
Precompiling JSON...
  1 dependency successfully precompiled in 3 seconds. 11 already precompiled.
  3.074514 seconds (243.41 k allocations: 17.114 MiB, 1.44% compilation time)


# 1.0 load:

julia> @time using JSON
  0.067531 seconds (119.51 k allocations: 7.854 MiB)


# Pre-1.0 load:
julia> @time using JSON
  0.181937 seconds (54.85 k allocations: 3.523 MiB)



# pre-1.0 precompile:
julia> @time using JSONold
Precompiling JSONold...
  1 dependency successfully precompiled in 1 seconds. 9 already precompiled.
  1.039715 seconds (95.82 k allocations: 7.128 MiB, 1.91% compilation time)

[quinnj]

Tiny detail but these methods don’t close the file handle:

JSON.jl/src/parse.jl

Line 172 in c9d0b1b

parsefile(file; jsonlines::Union{Bool, Nothing}=nothing, kw...) = parse(open(file); jsonlines=(jsonlines === nothing ? isjsonl(file) : jsonlines), kw...)

Just pushed a fix; thanks for mentioning.

[KristofferC]

Having to uncompress all the test files for all package updates feels like it could be slightly slow (at least on Windows). In TOML (which has a similar test suite) I get them during testing from an archive (https://github.com/JuliaLang/TOML.jl/blob/44aab3c023323587680ab8d8c7ef478bf78c4c0c/test/utils/utils.jl#L8) but I guess an archive could also be checked in. Alternatively, all the content of the files could be put into one file with a sentinel "file separator" and get constructed on the fly during testing.

[quinnj]

Having to uncompress all the test files for all package updates feels like it could be slightly slow (at least on Windows). In TOML (which has a similar test suite) I get them during testing from an archive (https://github.com/JuliaLang/TOML.jl/blob/44aab3c023323587680ab8d8c7ef478bf78c4c0c/test/utils/utils.jl#L8) but I guess an archive could also be checked in. Alternatively, all the content of the files could be put into one file with a sentinel "file separator" and get constructed on the fly during testing.

Great callout; I refactored so that we have tar files of both jsonchecker and JSONTestSuite test directories that get untarred when tests are run. I think that it's setup fine? Not sure if there are other considerations or if it's bad practice to untar in the test directory?

[tecosaur]

that get untarred when tests are run

What about just getting each file from the tarball directly (no on-filesystem untaring) using Tar.jl?

[quinnj]

that get untarred when tests are run

What about just getting each file from the tarball directly (no on-filesystem untaring) using Tar.jl?

Ah, can Tar.jl do that? I was going off the README docs and I didnt' see anything about that, but that would indeed by nice.

[quinnj]

that get untarred when tests are run

What about just getting each file from the tarball directly (no on-filesystem untaring) using Tar.jl?

Ok, I looked into Tar.extract with the predicate, but it seems pretty inefficient (checking each file in tarball for each extraction) and you still have to "extract" the single file to a directory. If there was a way to extract to an IOBuffer, I feel like that would make sense, but otherwise, I'm thinking of keeping it as-is.

[tecosaur]

You can create a Dict{String, Vector{UInt8}} for each file like so: https://github.com/tecosaur/DataToolkit.jl/blob/main/Common/ext/TarExt.jl#L22-L34

[nhz2]

I agree that API breakages are fine and expected. Since this is a pre-1.0 package, people depending on this package should expect an unstable API.

My main issue with this PR is that it currently relies on a large number of undocumented features from Julia, which makes reviewing the code almost impossible for someone like me who isn't familiar with how Julia is implemented, or how it will change in the future.

I also don't think mmap should be used in this package. Many commonly used systems do not have mmap available, or working well, and at least on my machine, read is much faster for small files. Ref: https://docs.nersc.gov/performance/io/dvs/#do-not-use-memory-mapping-mmap
JuliaLang/julia#58736

julia> @be _ Mmap.mmap("foo.json") x->GC.gc() seconds=100.0
Benchmark: 710 samples with 1 evaluation
 min    206.200 μs (22 allocs: 1.273 KiB)
 median 268.750 μs (22 allocs: 1.273 KiB)
 mean   298.476 μs (22 allocs: 1.273 KiB)
 max    691.200 μs (22 allocs: 1.273 KiB)

julia> @be _ read("foo.json") x->GC.gc() seconds=100.0
Benchmark: 798 samples with 1 evaluation
 min    158.300 μs (16 allocs: 800 bytes)
 median 203.700 μs (16 allocs: 800 bytes)
 mean   205.402 μs (16 allocs: 800 bytes)
 max    317.600 μs (16 allocs: 800 bytes)

julia> filesize("foo.json")
5

julia> versioninfo()
Julia Version 1.11.6
Commit 9615af0f26 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × 12th Gen Intel(R) Core(TM) i5-1240P
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, alderlake)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

[quinnj]

I'm concerned users like VSCodeServer will struggle with this change because they need to vendor all of their dependencies and this new version has more dependencies. It makes sense to me to have JSON.jl be the "best JSON package for most users" but it seems that JSON.jl is currently "the best JSON package for dependency-sensitive, performance-insensitive users" and such users should be provided a transition plan. Maybe pre-v1 JSON.jl can be spun out into a new SimpleJSON.jl or something?

cc @davidanthoff

I've added a new vendor/ directory that includes a single file/module JSONX that provides simplified parse/json functionality with true no-dependency. It's meant to be copied into a project/application and included manually. There are tests specific to the vendor code which will be triggered by a new action workflow that specifically tests the vendored code implementation. cc: @davidanthoff

[quinnj]

I agree that API breakages are fine and expected. Since this is a pre-1.0 package, people depending on this package should expect an unstable API.

My main issue with this PR is that it currently relies on a large number of undocumented features from Julia, which makes reviewing the code almost impossible for someone like me who isn't familiar with how Julia is implemented, or how it will change in the future.

I also don't think mmap should be used in this package. Many commonly used systems do not have mmap available, or working well, and at least on my machine, read is much faster for small files. Ref: https://docs.nersc.gov/performance/io/dvs/#do-not-use-memory-mapping-mmap JuliaLang/julia#58736

julia> @be _ Mmap.mmap("foo.json") x->GC.gc() seconds=100.0
Benchmark: 710 samples with 1 evaluation
 min    206.200 μs (22 allocs: 1.273 KiB)
 median 268.750 μs (22 allocs: 1.273 KiB)
 mean   298.476 μs (22 allocs: 1.273 KiB)
 max    691.200 μs (22 allocs: 1.273 KiB)

julia> @be _ read("foo.json") x->GC.gc() seconds=100.0
Benchmark: 798 samples with 1 evaluation
 min    158.300 μs (16 allocs: 800 bytes)
 median 203.700 μs (16 allocs: 800 bytes)
 mean   205.402 μs (16 allocs: 800 bytes)
 max    317.600 μs (16 allocs: 800 bytes)

julia> filesize("foo.json")
5

julia> versioninfo()
Julia Version 1.11.6
Commit 9615af0f26 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × 12th Gen Intel(R) Core(TM) i5-1240P
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, alderlake)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

I've appreciated your thorough review of this PR so far. I've tried to address all of your concerns of "julia internals" usage by putting in proper guards where appropriate and raising issues in Julia proper to try and get better public APIs for things.

I think I can also get behind your thoughts on Mmap. It's been a pattern of mine for years now to reach for it in file-byte-parsing scenarios, but I think overall read performance has improved such that it's not a clear win in all cases anymore. I'll do some benchmarking as well, but I'd be fine to remove that support, which also further simplifies things.

[quinnj]

Usage of and dependency on Mmap has now been removed.

[nsajko]

Using primitive type instead of just struct here seems ugly and unnecessary.

[nhz2]

I think the JSON.json function should keep its old API for backwards compatibility, as I don't see why a new API can't be given a new function name. For example, JSON.writefile, like in Parquet2.jl

For example, JSON.json could be defined as:

function json(a, indent=nothing)
  if isnothing(indent)
    writefile(String, a)
  else
     String(push!(writefile(Vector{UInt8}, a; pretty=convert(Int, indent)), UInt8('\n')))
end

The JSON.json currently in this PR is a bit hard to map onto the existing JSON.json.
For example, the following works on the existing version but is ambiguous in this PR.

julia> @test JSON.json("hello world", 2) == "\"hello world\"\n"
Error During Test at REPL[12]:1
  Test threw exception
  Expression: JSON.json("hello world", 2) == "\"hello world\"\n"
  MethodError: json(::String, ::Int64) is ambiguous.
  
  Candidates:
    json(fname::String, obj; kw...)
      @ JSON ~/github/JSON.jl/src/write.jl:428
    json(a, indent::Integer)
      @ JSON ~/github/JSON.jl/src/JSON.jl:107
  
  Possible fix, define
    json(::String, ::Integer)
  
  Stacktrace:
   [1] macro expansion
     @ ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Test/src/Test.jl:677 [inlined]
   [2] top-level scope
     @ REPL[12]:1
ERROR: There was an error during testing

[nhz2]

Several packages are currently using the show_json interface #377
It seems like less work to deprecate that interface, instead of completely removing it.

Also, it would be good to include in the migration guide an example of migrating code like https://github.com/JuliaComputing/TableView.jl/blob/a3a3172fed89260b779d79f22c901ed74d96b90d/src/TableView.jl#L293-L319 where show_json is used to avoid allocating a temporary dict.

I'm guessing this would involve defining a custom StructUtils.applyeach but it wasn't clear to me from https://juliaservices.github.io/StructUtils.jl/dev/#How-make-Works how applyeach fits in exactly.

[quinnj]

Quick update: I've been diving deep into a few performance characteristics of the overall code proposed here vs. pre-1.0 JSON and JSON3. I've found a few places particularly where we can improve by avoiding more allocations and it's been a good dive overall in terms of my own understanding of the core compiler's allocation elision/escape analysis limitations.

I'm hoping to get those performance improvements in this week, which should help tuple parsing (as reported here) perf be more in line.

I appreciate those looking into ecosystem ripple effects and I'll catch up on those comments soon.

[asprionj]

Hi, I'm coming from trying to juliac --trim - compile an MWE that just parses a JSON string using this branch of JSON.jl. (If interested, here's the actual mention in the discourse thread.) Thought I use you're future "complete re-write" directly.

Now, my question: do you intend to make (the rewrite of) JSON.jl trimmable? That is, have everything type-stable etc. to allow trimming and thus small-binary generation work? I think parsing and serialising JSON is one of the fundamental things to do in today's mostly web-based and micro-servic'ing software landscape. So, having this trimmable would definitely bring julia forward as more-generally-usable language.

[quinnj]

Update: I believe the performance issue w/ tuples reported in #376 is now fixed in this PR, so I'm tentatively marking this to be merged soon + doing the 1.0 release. I'm going to investigate #377 (thank you @nhz2 for the research!) and may delay the merge or provide more compat/migration guides as appropriate.

[quinnj]

Ok, pushed a couple of tweaks:

• Added compat for the JSON.json(str, indent) case @nhz2 mentioned; no more ambiguous case
• Added a Tables.jl extension for StructUtils.jl which should make a few dependents drastically simplify their integration code
• JSON.jl 1.0 will be trimmable! I tested some basic serialize/deserialize cases on Julia 1.12-rc3 and it works w/ a few tweaks I made!
• Reviewed the rest of the packages in Packages that need changes for JSON 1.0 #377 and they are all very easy updates that will end up with much simpler integration code (most of those packages are pretty dated though to be fair; many years since any updates from what I can tell)