How to Diff Large JSON Files on Linux — Even Bigger Than RAM

Two NDJSON exports from the same pipeline, a day apart. A database dump before and after a migration. An API snapshot that should be identical to yesterday's — but is it? At small sizes, any JSON diff tool answers this question. Below 100 MB, some of them already stop answering — and by a gigabyte, most have.

This guide covers why comparing two large JSON files on Linux is harder than it looks, what actually happened when we ran the common command-line tools against the same large file pairs on the same machine, and how gjxdiff — a JSON diff CLI for Linux built for exactly this case — compares files bigger than RAM. Every number here is a real measurement, and every command is copy-paste ready.

Why Diffing Large JSON Files Breaks the Usual Tools

Plain diff is the wrong instrument for JSON. It compares text lines, and JSON does not keep its meaning in lines: two files can differ in indentation, key order, or record order while containing identical data, and a line diff reports thousands of changes anyway. A minified multi-gigabyte file is often a single line, at which point a line diff has exactly one unit to work with.

Structural JSON diff tools solve that by parsing both documents and comparing keys and values. Almost all of them parse both documents fully into memory first — and that is where large files kill them. Depending on the runtime, the failure looks like one of three things:

• "JavaScript heap out of memory" — a Node.js process exhausts its heap building the parsed tree, which is many times larger than the file on disk.
• An OOM kill — the kernel terminates the process when the machine or the container runs out of memory.
• ERR_STRING_TOO_LONG — V8 caps a single string at roughly 512 MiB, so readFileSync(path, "utf8") cannot even read a larger file into memory. This one is independent of RAM: a 128 GB workstation fails exactly like a laptop.

None of this is bad engineering by the tools in question. They were built for configuration files and API responses, and at those sizes they work well. The problem is purely one of scale.

What Actually Happens at Scale, Measured

All measurements below were taken on August 4, 2026 on one machine: a Linux container with 8 GiB of RAM and 4 cores, a SATA SSD, cold page cache, single first runs, a 900-second timeout, and a 6 GB memory cap per tool. Same files, same box, for every tool. Three of the test pairs — 83 MB, 165 MB, and 837 MB per side — carried 20 planted differences each; the change-dense 156 MB pair was verified by exact record accounting instead.

Tool	2 MB pair	83 MB per side	837 MB per side
jd 2.5.0	correct, 14.59 s	exceeded the 6 GB memory cap	exceeded the 6 GB memory cap
json-diff 1.0.6	correct, 2 min 45 s	exceeded the 15-minute timeout	refused the input in 1.89 s (V8 string cap)
jsondiffpatch 0.7.6	1.86 s	2.5 s, 1.3 GB of RAM	refused the input in 0.94 s (V8 string cap)
jq 1.7 + sort + diff	0.23 s	5.0 s, 3.4 MB of RAM	56 s, text output only
gjxdiff 0.8.0	0.11 s	2.8 s	26 s

Two notes on reading the table. First, jd, json-diff, and jsondiffpatch cannot read NDJSON at all, so on the NDJSON pairs (83 MB and 837 MB per side) those three were measured on a JSON-array twin holding the identical records at the same size; gjxdiff and the jq pipeline ran the NDJSON form. Second, no competitor was run at 1.36 GB per side — that rung is gjxdiff-only and appears in the table below.

jd 2.5.0 is a true structural diff in a single Go binary, and on small files on this machine it did its job. Its memory use grows quadratically with array length, though, and on these files it exceeded the 6 GB cap from the 83 MB inputs upward.

json-diff 1.0.6 (npm) produced correct output on small files but slowly: 2 min 45 s on the 2 MB pair. On the 83-165 MB pairs it exceeded the 15-minute timeout; the 837 MB input it refused outright in 1.89 seconds with ERR_STRING_TOO_LONG — the same V8 string cap that stops jsondiffpatch. It parses both documents fully into the Node heap and compares the trees; on the mid-size inputs that approach simply runs out of time before it runs out of memory.

jsondiffpatch 0.7.6 (npm, with objectHash configured) was the strongest of the Node tools on this machine: fast up to the mid-100s of megabytes per side, and correct on the two pairs where we checked its output against ground truth (83 MB and 165 MB) — 165 MB was the largest pair it completed here — at a cost of 1.3-3.9 GB of RAM. But like every tool that reads whole files with readFileSync in utf8, it cannot open files above roughly 512 MiB at all — that is V8's string cap, and more RAM does not change it.

The jq recipe — normalize both files with jq -S, sort, and run ordinary diff — deserves respect. For keyed NDJSON it is genuinely useful, and on NDJSON input its memory stays flat at every size we tried (about 3.5 MB even on the 837 MB pair). On a single-document JSON file jq must hold the whole document — 2.3 GB on the 165 MB GeoJSON here. It has two structural limits: the output is raw text lines, not a structural report with paths and operations; and jq preserves number literals, so representation-only differences such as 1.0 versus 1.00 are reported as changes even though the values are equal.

gjxdiff on the Same Files

gjxdiff 0.8.0, cold runs on the same box — without the 6 GB cgroup cap used for the comparison table above, because capping these runs would mask the memory-budget behavior being measured:

Input	Time	Peak RAM
2 MB pair	0.11 s	14 MB
83 MB per side, NDJSON, 310,000 records	2.8 s	478 MB
165 MB per side, real-world GeoJSON	5.0 s	920 MB
156 MB per side, 30% of records changed	7.5 s	924 MB
837 MB per side, NDJSON, 3.1 million records	26 s	3.9 GB
1.36 GB per side	61 s	4.7 GB (1.4 GB of it heap; the rest is reclaimable file-backed mmap pages the kernel counts)

Correctness on the same runs: on the three pairs with planted differences (83 MB, 165 MB, and 837 MB per side), gjxdiff found all 20 planted differences with zero false positives. On the change-dense 156 MB pair, its output was verified by exact record accounting — 348,685 report records for 174,342 changed records, with no over- or under-reporting. The 1.36 GB pair is a scale run with no planted ground truth.

How gjxdiff Works

gjxdiff is built around one rule: input size must not be bounded by RAM.

• Memory-mapped inputs. The files are never loaded into memory; the kernel pages bytes in as they are compared.
• Working state on disk. Intermediate structures live in temporary files under a memory budget — a fixed 4 GiB by default, adjustable with --memory-limit.
• Key-based record pairing. NDJSON records are matched by an identity field (autodetected, or forced with --key), so reordered records are not reported as churn.
• Three outputs. An NDJSON machine report when stdout is piped, a color-coded human view on a terminal, and an RFC 6902 JSON Patch export with --patch.
• Deterministic. The same inputs and flags produce a byte-identical report on any machine — the default budget is a fixed number, not a share of free RAM, so nothing in the report depends on the machine's momentary state.
• Single static musl binary. No runtime, no dependencies; drop it on your PATH and it runs on any x86-64 Linux distribution.
• Single-threaded by design. The times in the tables above are what one core does.

Honest Limitations

Same policy as the rest of this site: boundaries stated up front.

• Linux x86-64 only. No Windows or macOS build.
• Single-threaded. It will not use your other cores.
• Very long arrays coarsen. Arrays whose direct children exceed an alignment cap are reported as coarse changed regions rather than item by item. The cap scales with the memory budget; raising --memory-limit raises it.
• Move detection has a floor. Moves are detected only for whole objects or arrays of at least 64 bytes; smaller displaced values report as an addition plus a removal.
• Temp disk use. Temporary files can reach about 2x the combined input size; the driver is how many records and nodes each input byte carries — many tiny records cost the most — with difference density a secondary factor. --temp picks the directory and --max-temp caps the total.
• Pre-1.0 report format. The report's meta line carries "stability":"draft"; the format may still change between minor releases.

Practical How-To

Install

Download the tarball from the repository's dist/0.8.0/ directory, verify its SHA-256 against the SHA256SUMS file published next to it, and put the binary on your PATH — full steps are in INSTALL.md:

sha256sum gjxdiff-0.8.0-x86_64-linux-musl.tar.gz   # compare against SHA256SUMS in dist/0.8.0/
tar xzf gjxdiff-0.8.0-x86_64-linux-musl.tar.gz     # extracts into gjxdiff-0.8.0-x86_64-linux-musl/
sudo install -m 0755 gjxdiff-0.8.0-x86_64-linux-musl/gjxdiff /usr/local/bin/gjxdiff
gjxdiff --version

Basic Diff

gjxdiff a.json b.json                    # human view on a terminal
gjxdiff a.json b.json > report.ndjson    # machine NDJSON report when piped
gjxdiff --stat a.json b.json             # totals only

JSON and NDJSON are detected per file, and a JSON file can be compared against an NDJSON file. Formatting, key order, and number representation (1.0 versus 1.00) do not count as differences.

CI Gate With Exit Codes

The exit code always reflects the full comparison: 0 identical, 1 differences found, 2 error, 130 interrupted.

gjxdiff --profile ci expected.ndjson actual.ndjson > report.ndjson
case $? in
  0) echo "identical" ;;
  1) echo "differences found, see report.ndjson" ;;
  *) echo "gjxdiff error" >&2; exit 2 ;;
esac

--profile ci bundles --ndjson --color=never --progress=never. For a pure yes/no gate, gjxdiff --quiet a.json b.json prints nothing and answers with the exit code alone.

Pair NDJSON Records by Key

When record order changed between the two files, pair records by identity instead of position:

gjxdiff --key id orders-a.ndjson orders-b.ndjson
gjxdiff --key region,id events-a.ndjson events-b.ndjson   # compound key

By default gjxdiff autodetects a key and falls back to order-based alignment where it is not confident; --key none disables keyed matching entirely.

Ignore Volatile Fields

gjxdiff --ignore ts,updated_at a.ndjson b.ndjson
gjxdiff --ignore '$.meta.generated_at' a.json b.json

A bare name is ignored at any depth; the wildcard form $.arr[*].ts ignores the field at any array position, while items[0].v pins one exact position. Ignored content is excluded from the comparison, and a pair identical apart from ignored fields exits 0.

Export an RFC 6902 Patch

gjxdiff --patch changes.json config-a.json config-b.json

The patch transforms A into B and is built under the same memory budget as the diff, so generating an RFC 6902 patch from large files works where in-memory tools cannot open the inputs. When a correct patch cannot be proven — for example, an operation with no concrete RFC 6901 address — gjxdiff refuses with exit 2 and writes no file, never a silently partial patch.

Diff an API Response From stdin

curl -s https://api.example.com/items | gjxdiff - snapshot.json

- reads one side from standard input, which makes drift checks against a saved snapshot a one-liner.

Where gjxdiff Comes From

gjxdiff grew out of the diff feature of GiantJSON Viewer+, our Android viewer for multi-gigabyte JSON — the same problem, solved with the same conviction that file size must not be bounded by RAM. If you need this comparison on a phone rather than a Linux box, see How to Compare Two JSON Files on Android.

gjxdiff is free for individuals and for organizations under 100 people, including their own CI; the full terms are in the LICENSE in the repository. Commercial licensing and questions: support@giantjson.com.

Frequently Asked Questions

How do I diff two JSON files that are too big to fit in memory?

Use a diff tool that does not parse the inputs into RAM. gjxdiff memory-maps both files and keeps its working state in temporary files under a fixed memory budget, so input size is bounded by disk, not RAM. On an 8 GiB Linux container it compared a 1.36 GB-per-side pair in 61 seconds, and with --memory-limit 512M the same multi-gigabyte pair still completed in 28 seconds using 42 MB of heap, with the coarser result clearly disclosed.

Why does json-diff time out or refuse large files?

json-diff parses both documents fully into the Node.js heap and then compares the object trees. A parsed tree is many times larger than the file on disk, so memory and time grow with input size. On our test machine, json-diff 1.0.6 was correct on small files but took 2 min 45 s on a 2 MB pair, exceeded a 15-minute timeout on the 83-165 MB pairs, and refused an 837 MB input outright in 1.89 seconds with ERR_STRING_TOO_LONG — the same V8 string cap that stops jsondiffpatch. Scale, not correctness, is the problem.

What causes ERR_STRING_TOO_LONG when reading a big JSON file in Node.js?

V8, the JavaScript engine in Node.js, caps a single string at roughly 512 MiB. readFileSync(path, "utf8") must return the whole file as one string, so it fails with ERR_STRING_TOO_LONG for files above that size no matter how much RAM the machine has. Any Node-based JSON diff tool that reads whole files this way inherits the limit; adding RAM does not help.

How can I compare NDJSON files where the record order changed?

Pair records by an identity field instead of by position. gjxdiff autodetects a key field on its own, or you can force one with --key id; compound keys such as --key region,id are supported. With keyed pairing, reordered records are not reported as churn: an order-only difference produces zero change records and exit code 1.

Can I generate an RFC 6902 patch from very large JSON files?

Yes. gjxdiff --patch changes.json a.json b.json writes an RFC 6902 JSON Patch that transforms A into B, built under the same memory budget as the diff itself, so it works on files bigger than RAM. If a correct patch cannot be proven — for example when an operation has no concrete RFC 6901 address — gjxdiff refuses with exit code 2 and writes no file rather than emitting a silently partial patch.

Does gjxdiff work on Windows or macOS?

No. gjxdiff is Linux x86-64 only. It ships as a single static musl binary, so it runs on any x86-64 Linux distribution without dependencies, but there is no Windows or macOS build.

Is gjxdiff open source?

No. gjxdiff is distributed as a prebuilt binary. It is free for individuals and for organizations with fewer than 100 people, including their own CI and automation; automated use in larger organizations and embedding in commercial products require a commercial license. Free use requires the attribution line gjxdiff by Tibor Kovacs (Kotysoft). The full terms are in the LICENSE file in the GitHub repository.

Is the jq normalize-and-diff recipe good enough for comparing JSON files?

Often, yes. Normalizing both files with jq -S, sorting, and running diff is genuinely useful, and on NDJSON input its memory stays flat; a single-document JSON must be held whole in memory (2.3 GB for a 165 MB document in our tests). Its two limits: the output is raw text lines rather than a structural report with paths, and jq preserves number literals, so representation-only differences such as 1.0 versus 1.00 are reported as changes even though the values are equal.

Conclusion

Below a few hundred megabytes, you have choices, and several of them are good. Above that, on the machine we measured, the field thins out fast: in-memory structural tools hit heap limits, timeouts, or V8's string cap, and the jq recipe — solid as it is — answers with text lines rather than structure.

gjxdiff exists for that upper range: a structural JSON and NDJSON diff for the command line that memory-maps its inputs, works inside a fixed memory budget, pairs reordered records by key, exports RFC 6902 patches, and tells you plainly whenever a limit shaped the answer. The binary and the full manual are at github.com/kotysoft/gjxdiff.