How to Compare NDJSON Files When the Record Order Changed

You export the same table twice, a day apart. Same rows, same data — and diff reports that nearly every line changed. Anyone who compares NDJSON or JSONL files from databases, event pipelines, or log shippers hits this eventually: the records are fine, the order moved.

This guide covers why record order changes between dumps, why plain diff and naive sort tricks mislead, an honest jq recipe that solves most of the problem with copy-paste commands, and key-based record pairing with gjxdiff for the cases the recipe cannot cover. Every number here is a real measurement.

Why Record Order Changes Between Dumps

NDJSON writers rarely promise an order. A SELECT without an ORDER BY returns rows in whatever order the storage engine finds convenient, and that order shifts after inserts, deletes, or a vacuum. Export jobs that run in parallel interleave their workers' output differently on every run. Log shippers and event queues deliver by arrival, not by timestamp. None of this changes the data — two dumps can be record-for-record identical and still list those records in a completely different sequence.

For a line-based tool, that distinction does not exist. A record that moved from line 12 to line 47,000 is one deletion plus one unrelated addition, and a reordered dump of a million records produces a diff with millions of phantom changes wrapped around the three edits you actually care about.

Why Plain diff and Naive sort Mislead

The first instinct is to sort both files and diff the results:

sort a.ndjson > a.sorted
sort b.ndjson > b.sorted
diff a.sorted b.sorted        # misleading

This only works when both files serialize every record byte-identically. In practice they often do not: one exporter writes {"id":7,"name":"x"} and the other {"name": "x", "id": 7}, or one writes 1.0 where the other writes 1. Same data, different bytes — so identical records sort to different positions and diff reports them as changed. The output mixes real edits with serialization noise, and there is no way to tell them apart by looking.

The fix is to normalize before sorting, and jq does that well.

The jq Recipe: Normalize, Sort, Diff

This is the honest first answer, and for many jobs it is also the last one needed:

jq -S -c . a.ndjson | sort > a.norm
jq -S -c . b.ndjson | sort > b.norm
diff a.norm b.norm

-S sorts every object's keys, -c prints each record minified on one line. After that, byte equality means data equality for most records, sorting makes the order irrelevant, and plain diff shows which records exist only in one file.

It is also genuinely cheap. Measured on August 4, 2026 — all numbers in this article come from the same machine: a Linux container with 8 GiB of RAM and 4 cores, a SATA SSD, cold page cache, single first runs — the jq 1.7 recipe processed an 83 MB-per-side keyed NDJSON pair in 5.0 seconds using 3.4 MB of RAM, and an 837 MB-per-side pair in 56 seconds at 3.5 MB. Because jq streams NDJSON one record at a time, memory stays flat no matter how large the NDJSON file grows — a property of newline-delimited input specifically. On a single-document JSON file, jq must hold the whole parsed document: 2.3 GB on a 165 MB document in the same benchmark.

When This Is All You Need

• You want a yes/no answer: did anything change besides the order?
• Records are small enough that reading a whole changed record is fine.
• You are in CI or on a server where jq is already installed and adding a tool is friction.

Where It Stops Being Enough

The recipe has two structural limits, and both follow from what it is:

• The output is text lines, not a report. A record with one changed field appears as one full record removed and one full record added. On wide records, finding which field changed is manual work, and there are no paths, no operations, and nothing machine-readable to feed a pipeline.
• Number literals count as differences. jq preserves the literal it read, so 1.0 versus 1.00 — the same number — survives normalization and shows up in the diff as a change. On files re-serialized by different writers, this produces false positives that no amount of sorting removes.

When either limit bites — or when the files outgrow what you want to eyeball — the next step is a tool that pairs records by identity and compares them structurally.

Pairing Records by Key with gjxdiff

gjxdiff is a structural JSON and NDJSON diff CLI for Linux x86-64, built for files bigger than RAM. Its answer to the reordering problem is record identity: instead of comparing line 12 with line 12, it works out which record in B is the record from A, and compares those two — wherever they moved.

gjxdiff a.ndjson b.ndjson                    # key autodetected
gjxdiff --key id a.ndjson b.ndjson           # force the identity field
gjxdiff --key region,id a.ndjson b.ndjson    # compound key
gjxdiff --key meta.id a.ndjson b.ndjson      # field one level down
gjxdiff --key none a.ndjson b.ndjson         # positions are the identity

By default gjxdiff autodetects a key field on its own and falls back to order-based alignment where it is not confident. --key forces the choice: compound keys combine up to 16 comma-separated fields, meta.id reaches one level of nesting, and \, escapes a literal comma in a field name. --key none disables keyed matching entirely — the right call when position itself is the meaning, as in a time series.

What you get instead of text lines is a field-level report: each difference carries its path (for example $.users[*].email), its operation, and byte offsets into both files. On a terminal it renders as a color-coded human view; piped, it becomes an NDJSON machine report. And unlike jq, gjxdiff compares values, not literals — 1.0 versus 1.00 is not a difference.

Ignore the Fields That Always Change

Exports love to re-stamp timestamps. Exclude them from the comparison instead of reading past them:

gjxdiff --key id --ignore updated_at,ts a.ndjson b.ndjson

A bare name is ignored at any depth; the wildcard form $.arr[*].ts ignores the field at any array position, while items[0].v pins one exact position. A pair identical apart from ignored fields exits 0.

JSON on One Side, NDJSON on the Other

Formats are detected per file, and a JSON document can be compared against an NDJSON file. When the API snapshot is a JSON array and the database dump of the same data is newline-delimited, that is one command, not a conversion step.

Measured on Real Files

Same machine and conditions as the jq numbers above. On the benchmark's three pairs with planted differences (83 MB, 165 MB, and 837 MB per side), gjxdiff 0.8.0 found all 20 planted differences with zero false positives; the change-dense 156 MB pair was verified by exact record accounting (348,685 report records for 174,342 changed records), and the 1.36 GB pair is a scale run with no planted ground truth.

Input	Time	Peak RAM
2 MB pair	0.11 s	14 MB
83 MB per side, NDJSON, 310,000 records	2.8 s	478 MB
156 MB per side, 30% of records changed	7.5 s (52 MB report)	924 MB
837 MB per side, NDJSON, 3.1 million records	26 s	3.9 GB
1.36 GB per side	61 s	4.7 GB (1.4 GB of it heap; the rest is reclaimable file-backed mmap pages the kernel counts)

The 30%-changed row matters for real-world use: a change-dense pair does not blow up, it just writes a bigger report. For the full tool-by-tool comparison at these sizes — jd, json-diff, jsondiffpatch, and jq on the same box — see How to Diff Large JSON Files on Linux. If the tools you tried are crashing rather than merely misreporting, that failure mode has its own guide: JSON Diff Out of Memory? Why It Happens and What Works.

One more thing keyed pairing gives you: because reordered records are matched rather than re-listed, gjxdiff can also export the comparison as an RFC 6902 JSON Patch when you need a machine-applyable delta rather than a report.

The Same Problem on Your Phone

NDJSON dumps do not always arrive at a Linux box first. The Compare Files tool in GiantJSON Viewer+, our Android viewer for multi-gigabyte JSON, solves the same reordering problem the same way: list items are paired by an identity field — detected automatically, set by position, or named explicitly with a custom key — and the comparison streams from disk caches, so file size is not bound by the phone's RAM.

How the on-device keyed matching works, with measured numbers on 2 GB pairs, is covered in How to Compare Two JSON Files on Android.

Frequently Asked Questions

Are NDJSON and JSONL the same thing?

In practice, yes. NDJSON (newline-delimited JSON) and JSONL (JSON Lines) both mean a text file with one complete JSON value per line, almost always one object per line. The two names come from two independent specification efforts; the files themselves are interchangeable, and every technique in this guide applies to both.

How do I compare two NDJSON files if the record order changed?

Either make the order irrelevant or pair records by identity. The jq recipe — jq -S -c . on both files, then sort, then diff — makes order irrelevant and works with tiny memory, but reports whole records as text lines. gjxdiff pairs records by an identity field (autodetected, or forced with --key id), so reordered records are matched and compared field by field, and only real data differences are reported.

Is sort plus diff enough to compare NDJSON files?

Only after normalization. Sorting raw lines works only if both files serialize every record byte-identically; different key order or number formatting breaks it. Normalizing with jq -S -c first fixes key order and formatting, and the result is genuinely useful. Two limits remain: the output is text lines rather than a field-level report, and jq preserves number literals, so 1.0 versus 1.00 is flagged as a difference even though the values are equal.

How do I diff NDJSON records by more than one field?

Use a compound key. gjxdiff --key region,id pairs records by the combination of both fields; up to 16 fields are supported, a field one level down is written as meta.id, and a literal comma in a field name is escaped as \,.

What does gjxdiff report when only the record order changed?

With keyed pairing, an order-only difference produces zero change records and exit code 1: the files are not byte-identical, but no data changed. Reordered records are never reported as removals plus additions.

Can I compare a JSON array against an NDJSON file?

Yes. gjxdiff detects JSON and NDJSON per file, and one side can be a JSON document while the other is NDJSON — useful when an API snapshot is a JSON array and the database dump of the same data is newline-delimited.

How large can the compared NDJSON files be?

gjxdiff memory-maps its inputs and keeps working state in temporary files under a fixed memory budget, so input size is bounded by disk, not RAM. Measured on an 8 GiB Linux container: 83 MB per side (310,000 records) in 2.8 seconds, 837 MB per side (3.1 million records) in 26 seconds, and 1.36 GB per side in 61 seconds.

Conclusion

Reordered NDJSON is not a diff problem, it is an identity problem. The jq recipe solves it by erasing order — cheap, scriptable, and enough whenever a record-level text answer is enough. When you need to know which field changed, need 1.0 and 1.00 treated as equal, or need a machine-readable report at millions-of-records scale, keyed pairing is the tool for the job: gjxdiff --key id a.ndjson b.ndjson, and reorders stop being noise.