Format JSONL Files (JSON Lines)

JSONL emerged from data engineering and log processing workflows where streaming the data is more efficient than loading it into memory. A 100 gigabyte file written as one giant JSON array of records cannot be parsed in a streaming fashion by a strict RFC 8259 parser because the parser must wait for the closing bracket of the array before producing any structured output. The same data written as JSONL, with one record per line, can be processed record by record with constant memory regardless of total file size. This streaming property is the entire reason JSONL exists as a distinct format from standard JSON, and it is why every large-scale data system from BigQuery to Snowflake to Spark accepts JSONL as a first-class input format.

The format is intentionally simple. Each line of a JSONL file is a complete JSON document, typically an object but sometimes an array or even a primitive value. Lines are separated by single newline characters (LF), and the file may or may not end with a trailing newline. Blank lines are ignored. There is no global schema, no wrapper, and no metadata at the file level. This simplicity is the source of the format streaming power: a JSONL file is a concatenation of independent JSON documents, so it can be split, merged, sorted by record, and processed in parallel with bog-standard line-oriented Unix tools such as head, tail, grep, awk, and split. None of these operations work cleanly on a single giant JSON array.

Common uses for JSONL include application log aggregation (every log line is one JSON record with timestamp, level, message, and metadata fields), machine learning training data (each training example is one JSONL record with features and labels), webhook event archives (each delivered event is one JSONL record with the full payload and delivery metadata), and analytical data warehouse staging (raw events land as JSONL files in object storage before transformation into columnar table formats). The common thread is that the records are independent and can be processed without reference to each other, which is exactly the property JSONL is designed to support.

When to use JSONL versus standard JSON depends on the consumer. Use JSONL when records are independent, the file may be large, the consumer streams records one at a time, or you need to combine with line-oriented Unix tools. Use standard JSON when the data has a meaningful global structure, the file is small enough to load fully into memory, or the consumer is a tool that does not understand JSONL. For interchange between systems that support both, JSONL is often the better default because it scales gracefully without sacrificing functionality on small datasets. Tools that produce standard JSON can convert to JSONL with jq -c .[] input.json > output.jsonl in one step.

JSONL also interacts well with compression. Gzip and brotli build dictionaries of repeated substrings, and JSONL files with similar records benefit from dictionary reuse across records. A 1 gigabyte JSONL log file may compress to 100 megabytes or less, with the same compression ratio that a single giant JSON array would achieve. The advantage of JSONL is that the compressed file can still be decompressed and streamed line by line without loading the full content, because gzip and brotli support streaming decompression. Combining JSONL with gzip is the standard format for large log archives, training datasets, and event stores in modern data infrastructure.