JSON Lines (NDJSON): Streaming JSON for Logs, Pipelines and LLMs

A log file grows by 100 MB a day. You need to filter it for errors, aggregate by user, and count by status code. A JSON array file requires loading all of it into memory to parse. An NDJSON file — where every line is a valid JSON object — requires nothing. Read line by line. Process line by line. Stop any time. That is the practical difference, and it is the reason every serious log pipeline, every LLM training data format, and every streaming API eventually reaches for newline-delimited JSON.

You can convert between JSON arrays and JSON Lines with the JSON Lines Converter — paste in either format and get the other out.

Why JSON Lines (also called NDJSON)

A standard JSON array file looks like this:

[
  { "user": "alice", "event": "login", "ts": 1719840000 },
  { "user": "bob", "event": "purchase", "ts": 1719840001 }
]

To parse it, a JSON parser must read the entire string, build the full tree in memory, and expose the array. For a 10 GB log file, that is 10 GB of memory. For a streaming scenario — a server sending events as they occur — JSON arrays require buffering until you close the array, which you cannot do until the stream ends.

NDJSON (Newline-Delimited JSON, the format documented at ndjson.org) solves this:

{ "user": "alice", "event": "login", "ts": 1719840000 }
{ "user": "bob", "event": "purchase", "ts": 1719840001 }
{ "user": "carol", "event": "logout", "ts": 1719840002 }

Every line is a valid JSON object. You can read the first line, process it, discard it from memory, and read the second. The parser never needs to see more than one line at a time. The file is also appendable: you can echo '{ "event": "error" }' >> log.ndjson and the file remains valid.

A note on naming: NDJSON / JSON Lines is a community format. It is not the same as RFC 7464 ("JSON Text Sequences"), which is a different streaming-JSON format that uses the ASCII Record Separator (\x1E) between records rather than newlines. The two are easy to confuse because both describe themselves as "streaming JSON"; they are not interchangeable.

Where it is used in production

Log aggregation. Tools like jq (for JSON extraction), Elasticsearch's Beats, and Loki all expect or produce NDJSON. A server that writes one JSON object per line can be piped directly into a log processor without buffering.

LLM training and evaluation data. Many fine-tuning pipelines use NDJSON for instruction datasets:

{ "instruction": "What is 2+2?", "input": "", "output": "4" }
{ "instruction": "Translate to French", "input": "Hello", "output": "Bonjour" }

Each sample is one line. Shuffling, filtering, and splitting the dataset is sort, head, and tail — no JSON parser needed.

Streaming APIs. The jq manual recommends NDJSON for piping JSON between tools. HTTP transfer with Transfer-Encoding: chunked works naturally with newline-delimited objects.

Terraform plan files. terraform show -json outputs NDJSON — one JSON object per line representing a resource change. This lets you filter and analyze plan output programmatically without loading the whole plan into a JSON parser.

Working with NDJSON in code

Most languages handle it directly:

# Reading NDJSON
with open("events.ndjson") as f:
    for line in f:
        event = json.loads(line)
        process(event)

// Node.js — reading
import { readFileSync } from "fs";
const lines = readFileSync("events.ndjson", "utf8").split("\n");
for (const line of lines) {
  if (line.trim()) process(JSON.parse(line));
}

// Node.js — writing (appendable)
import { appendFileSync } from "fs";
function logEvent(event) {
  appendFileSync("events.ndjson", JSON.stringify(event) + "\n");
}

# Unix tools work on NDJSON directly
grep '"error"' app.ndjson | jq ".user"   # extract users from error lines
jq -c "select(.status >= 400)" app.ndjson   # filter HTTP errors
jq -s "map(.ts)" app.ndjson    # collect all timestamps

The strict rules

Not every newline-separated file is valid NDJSON. The spec requires:

1. Each line must be a valid JSON value (object, array, string, number — but practically, all records are objects), and the file is one record per line
2. No trailing comma is allowed
3. The record separator is a newline — both \n (LF) and \r\n (CRLF) are accepted, since many tools and platforms emit one or the other. What matters is that each record stays on its own line and is itself a valid JSON value.
4. UTF-8 encoding is assumed

These rules are what make Unix tools like grep, wc, and cut reliable on NDJSON files. If your file has blank lines or malformed JSON on a line, a streaming parser will throw.

Converting to and from JSON arrays

For human inspection and API responses, a JSON array is more convenient. For processing and storage, NDJSON is more efficient. Converting between them is a matter of framing:

# JSON array to NDJSON
with open("data.json") as f:
    data = json.load(f)
with open("data.ndjson", "w") as f:
    for item in data:
        f.write(json.dumps(item) + "\n")

# NDJSON to JSON array
with open("data.ndjson") as f:
    data = [json.loads(line) for line in f if line.strip()]
with open("out.json", "w") as f:
    json.dump(data, f)

Use the JSON Lines Converter to handle this without writing code — paste a JSON array or NDJSON file and get the other format instantly.

When not to use NDJSON

JSON Lines is the wrong choice when:

• You need to validate the complete structure before processing anything (schema validation requires a full parse)
• The file is consumed by a tool that expects a JSON array (most REST APIs)
• You need nested arrays inside records — they are valid JSON, but a line like {"users": [{"id": 1}, {"id": 2}]} is fine; {"a": 1}, {"a": 2} (bare array elements) is not valid NDJSON
• You need random access (NDJSON has no index)

For structured data where you frequently look up a specific record, a JSON array or a database is the right tool. NDJSON earns its place in append-only, streaming, or large-file scenarios.

Try it

Paste any JSON array or NDJSON file into the JSON Lines Converter to convert between formats — all processing happens in your browser.