JSON Diff Out of Memory? Why It Happens and What Works

The diff was working fine on last month's exports. Today's files are bigger, and now the run ends with this instead of a report:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Or, on even larger files, before any comparison starts at all:

Error: Cannot create a string longer than 0x1fffffe8 characters
    code: 'ERR_STRING_TOO_LONG'

Both errors come from the same root cause, but they need different fixes — and one of them cannot be fixed with more memory at all. This guide explains why JSON diff tools crash on large files, then walks an escalation ladder from the quick fix to the approach that works at any size, with measured numbers at every step.

Why JSON Diff Tools Run Out of Memory

Almost every structural JSON diff tool works the same way: parse file A into an object tree, parse file B into another, then walk both trees comparing them. The design is correct and, at config-file sizes, fast. The cost is hidden in the word parse.

A parsed tree is many times larger than the file on disk. Every object becomes a heap allocation with headers and pointers, every short string becomes a string object, every number gets boxed into the runtime's representation. A diff holds two of these trees at once, plus its own bookkeeping for the comparison. So a "300 MB diff" is actually a multi-gigabyte memory workload — and when it exceeds the runtime's heap ceiling, the process dies with the heap error above.

This is not bad engineering by the tools. They were built for API responses and configuration files, and at those sizes the in-memory design is the right one. The problem is purely one of scale.

The Escalation Ladder

Four steps, cheapest first. The measurements referenced below were taken on August 4, 2026, on one machine: a Linux container with 8 GiB of RAM and 4 cores, a SATA SSD, cold page cache, single first runs, a 900-second timeout, and a 6 GB memory cap per tool.

Step 1: Raise the Node.js Heap

Node's default heap ceiling is far below what the machine could give it. --max-old-space-size raises it (the value is in megabytes):

node --max-old-space-size=6144 diff-script.js a.json b.json
NODE_OPTIONS="--max-old-space-size=6144" json-diff a.json b.json

This is the right first move, and it genuinely works — inside a window. What we measured at the edges of that window:

• jsondiffpatch 0.7.6 (with objectHash configured) is the tool the bigger heap helps most: 2.5 s on 83 MB per side at 1.3 GB of RAM, 8 s on 165 MB per side at 3.9 GB. Fast and correct — as long as the heap keeps up and the files stay under the string cap described below.
• json-diff 1.0.6 has a different bottleneck: time. It took 2 min 45 s on a 2 MB pair (at just 193 MB of RAM) and exceeded a 15-minute timeout on the 83-165 MB pairs. At 837 MB it never got that far: it refused the input in 1.89 seconds with ERR_STRING_TOO_LONG — the same V8 string cap as jsondiffpatch. A bigger heap does not make it finish sooner.
• jd 2.5.0 is a Go binary, so the Node flag does not apply — and its limit is structural: memory grows quadratically with array length. Growing one array from 500 to 8,000 elements took it from 0.03 s and 12 MB to 13.4 s and 1.1 GB, and on our test files it exceeded the 6 GB cap from 83 MB inputs upward. Against a quadratic curve, headroom buys very little.

One test-setup note: the NDJSON benchmark pairs were supplied to jd, json-diff, and jsondiffpatch as JSON-array twins holding the identical records at the same size, because none of the three can read NDJSON at all; gjxdiff and jq ran the NDJSON form.

And no heap size helps a Node tool that cannot read the file in the first place — that is the string cap, covered in its own section.

Step 2: Split or Narrow the Comparison

If only part of the file matters, do not diff the rest. Extract the interesting subtree and compare the slices:

jq -c '.data.items' big-a.json > items-a.json
jq -c '.data.items' big-b.json > items-b.json
json-diff items-a.json items-b.json

The extraction still parses the whole document, but jq's internal representation is far leaner than a JavaScript object tree, and the diff tool only ever sees the slice. (gjxdiff has this built in: --path '$.data.items' compares one subtree with no extraction step.)

For NDJSON, the file splits naturally at record boundaries — split -l makes chunks any tool can handle. The caveat: chunking by line number assumes record N is still record N in both files. One inserted or reordered record misaligns every chunk after it. If your records have moved, compare NDJSON files by key instead of by position.

Step 3: The jq Recipe for Keyed Data

When the inputs are NDJSON (or can be turned into it) and the records carry identity, normalize-sort-diff answers most questions in almost no memory:

jq -S -c . a.ndjson | sort > a.norm
jq -S -c . b.ndjson | sort > b.norm
diff a.norm b.norm

Measured: 5.0 s at 3.4 MB of RAM on the 83 MB NDJSON pair, 56 s at 3.5 MB on the 837 MB NDJSON pair. On NDJSON/JSONL input, memory is flat because jq streams one record at a time — there, this recipe simply does not have the out-of-memory problem. On a single-document JSON file, jq must hold the whole parsed document: 2.3 GB on our 165 MB GeoJSON and 1.6 GB on a 156 MB document, so the flat-memory property is specific to newline-delimited input. Its limits are of a different kind: the output is text lines rather than a structural report, and jq preserves number literals, so 1.0 versus 1.00 is flagged as a change. The full treatment, including when the recipe is all you need, is in How to Compare NDJSON Files When the Record Order Changed.

Step 4: A Disk-Backed Structural Diff

When you need a real structural report — paths, operations, move detection — or an RFC 6902 patch, and the files are past what in-memory tools survive, the remaining option is a diff that never builds the trees in RAM. gjxdiff memory-maps both inputs and keeps its working state in temporary files under a fixed memory budget, so input size is bounded by disk:

gjxdiff a.json b.json > report.ndjson
gjxdiff --memory-limit 512M a.json b.json    # fit a small container

The same benchmark files, gjxdiff 0.8.0, cold runs — the 1.36 GB row ran without the 6 GB per-tool cap used above, because capping it would mask the memory-budget behavior being measured:

Input	Time	Peak RAM
2 MB pair	0.11 s	14 MB
83 MB per side, NDJSON, 310,000 records	2.8 s	478 MB
165 MB per side, real-world GeoJSON	5.0 s	920 MB
837 MB per side, NDJSON, 3.1 million records	26 s	3.9 GB
1.36 GB per side	61 s	4.7 GB (1.4 GB of it heap; the rest is reclaimable file-backed mmap pages the kernel counts)

Correctness on the same runs: on the three pairs with planted differences (83 MB, 165 MB, and 837 MB per side), gjxdiff found all 20 planted differences with zero false positives; a change-dense 156 MB pair in the same benchmark was verified by exact record accounting (348,685 report records for 174,342 changed records), and the 1.36 GB pair is a scale run with no planted ground truth.

The full tool-by-tool comparison — same files, same machine, all five tools — is in How to Diff Large JSON Files on Linux.

The 512 MiB Wall More RAM Cannot Fix

One failure in this space is routinely misdiagnosed as an out-of-memory error, and it is not one. V8 — the JavaScript engine inside Node.js — caps a single string at 0x1fffffe8 characters, roughly 512 MiB. readFileSync(path, "utf8") must return the entire file as one string, so for any file above the cap it throws ERR_STRING_TOO_LONG immediately, before parsing, before diffing, before memory pressure even begins.

We measured exactly this: on the 837 MB file, jsondiffpatch — the strongest Node tool in our tests — failed in under a second, and json-diff refused the same file in 1.89 seconds with the same error. Not slowly, not after swapping: instantly, because the file could not be read into a string. A 128 GB workstation fails identically to a laptop, and --max-old-space-size is irrelevant. Every Node-based diff tool that reads whole files in utf8 inherits this wall; the only ways past it are chunked reading (which in-memory diff tools do not do) or a tool outside the JavaScript runtime.

So the rule of thumb: heap errors are negotiable, ERR_STRING_TOO_LONG is not. If you see the string error, skip straight to steps 2-4 of the ladder.

When the File Is on a Phone

A related situation with the same root cause: a multi-gigabyte JSON file lands on your Android device, and every viewer app dies trying to load it into memory — the mobile version of the heap error. GiantJSON Viewer+, the app behind this site, opens multi-gigabyte files on a phone through a streaming architecture that never loads the file into the Java heap; how that works is covered in How to View Large JSON Files on Android.

Frequently Asked Questions

What does "JavaScript heap out of memory" mean during a JSON diff?

The full message is FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory. It means the Node.js process hit its heap ceiling while building the parsed object trees and diff structures. A parsed JSON tree is many times larger than the file on disk, and a diff needs both trees at once. The ceiling can be raised with node --max-old-space-size, which helps until the workload outgrows physical RAM or hits V8's string cap.

Does raising --max-old-space-size fix JSON diff out-of-memory errors?

Sometimes, and only within a window. On our test machine it is what kept jsondiffpatch working at 83-165 MB per side (at 1.3-3.9 GB of RAM). It does nothing for tools that run out of time first — json-diff exceeded a 15-minute timeout on the 83-165 MB pairs — and nothing against V8's roughly 512 MiB string cap, which stops Node tools from even reading larger files regardless of heap size; at 837 MB json-diff refused the input in 1.89 seconds for exactly that reason.

What causes ERR_STRING_TOO_LONG ("Cannot create a string longer than 0x1fffffe8 characters")?

V8 caps a single JavaScript string at 0x1fffffe8 characters, roughly 512 MiB. readFileSync(path, "utf8") must return the whole file as one string, so any Node tool that reads whole files this way fails with Error: Cannot create a string longer than 0x1fffffe8 characters (code ERR_STRING_TOO_LONG) on larger files. It is an engine limit, not a memory limit: adding RAM or raising the heap does not change it.

How do I diff JSON files that are bigger than my RAM?

Use a diff that does not parse the inputs into memory. gjxdiff memory-maps both files and keeps its working state in temporary files under a fixed memory budget, so input size is bounded by disk. Measured on an 8 GiB Linux container: 837 MB per side in 26 seconds, 1.36 GB per side in 61 seconds, and the same multi-gigabyte pair under --memory-limit 512M in 28 seconds using 42 MB of heap, with the coarser result disclosed.

Why does a JSON diff crash on one file but work on another of the same size?

Structure matters as much as byte count. Some tools scale with the shape of the data, not just its size: jd's memory grows quadratically with array length in our measurements — growing one array from 500 to 8,000 elements took it from 0.03 seconds and 12 MB to 13.4 seconds and 1.1 GB. A file dominated by one huge array can therefore fail where a larger but flatter file succeeds.

Can I put a hard memory limit on a JSON diff, for CI containers?

gjxdiff runs under a fixed memory budget of 4 GiB by default, the same on every machine, and --memory-limit sets it explicitly — 512M for a small CI container, more to spend a big machine. The limit is honored as given, and if a tighter budget forces a coarser comparison, every output mode says so; nothing degrades silently. Measured: a multi-gigabyte pair under --memory-limit 512M completed in 28 seconds using 42 MB of heap.

Conclusion

"JSON diff out of memory" is two different problems wearing one error message. The heap variant is negotiable: raise the ceiling, narrow the comparison, or switch to the jq recipe, and you can get a long way — the measured windows above show exactly how far. The string-cap variant is not negotiable inside Node at all.

Past both walls sits the disk-backed approach: gjxdiff compares JSON and NDJSON files bigger than RAM inside a fixed, explicit memory budget, and tells you plainly whenever a limit shaped the answer. The binary and the full manual are at github.com/kotysoft/gjxdiff.