What Is a .PARQUET File & How to Open It

Apache Parquet (.parquet)

Overview

Apache Parquet is a columnar storage format for analytical data. Where CSV or JSON store records row by row, Parquet groups all the values of a single column together, which changes the economics of large-scale querying: a query touching 3 columns of a 200-column table reads only those 3 columns from disk.

Storing like values together also compresses far better than mixed row data, and lets each column use an encoding suited to its type. A dataset that occupies 100 GB as CSV commonly lands between 5 and 20 GB as Parquet with no loss of information.

Technical Specifications

Format Details

MIME Type: application/octet-stream (sometimes application/vnd.apache.parquet)
File Extension: .parquet, occasionally .parq
Category: Data
Magic bytes: 50 41 52 31 (PAR1) at both the start and the end of the file
Metadata encoding: Apache Thrift
Created: 2013, by Twitter and Cloudera
Governance: Apache Software Foundation

Identification

Parquet is unusually easy to identify with confidence:

head -c 4 data.parquet     # PAR1
tail -c 4 data.parquet     # PAR1

The trailing magic matters structurally: the footer holds the schema and the row-group index, so readers seek to the end first. A file missing its footer is unreadable even if the data blocks are intact - which is why interrupted writes produce unusable Parquet files.

File Structure

data.parquet
├── "PAR1"                          4-byte header magic
├── Row Group 0
│   ├── Column Chunk: user_id
│   │     └── Page 0, Page 1, ...   (dictionary page + data pages)
│   ├── Column Chunk: event_time
│   └── Column Chunk: country
├── Row Group 1
│   └── ...
├── Footer (Thrift-encoded FileMetaData)
│   ├── schema                      nested, typed column definitions
│   ├── num_rows
│   ├── row_groups[]                offsets, sizes, and per-column statistics
│   └── key_value_metadata          arbitrary application metadata
├── footer length (4 bytes, little-endian)
└── "PAR1"                          4-byte trailer magic

Row groups

A row group is a horizontal slice of the table - typically 128 MB or so - containing a chunk of every column. Row groups are the unit of parallelism: distributed engines assign different row groups to different workers.

Column statistics and predicate pushdown

Each column chunk records min, max, and null count. A query filtering WHERE country = 'FR' can read the footer, see that a row group's country range is AR to BR, and skip that entire row group without reading any data. This predicate pushdown is where most of Parquet's query speed comes from.

Encodings

Dictionary encoding: repeated values stored once, referenced by index. Very effective for low-cardinality columns.
Run-length encoding and bit-packing: for repeated values and small integers.
Delta encoding: for sorted or near-sorted numeric and timestamp columns.
Compression: Snappy (fast, the common default), Zstandard (better ratio, now widely preferred), Gzip, LZ4, Brotli.

Encoding is applied per column, so a table can mix a dictionary-encoded string column with a delta-encoded timestamp column in the same file.

History and Development

Parquet was created in 2013 by engineers at Twitter and Cloudera, drawing on the record shredding and assembly algorithm described in Google's Dremel paper. That algorithm is what allows Parquet to store deeply nested structures - lists, maps, structs - in a flat columnar layout using definition and repetition levels.

The format became a top-level Apache project in 2015 and is now the default storage format for essentially every analytical system: Spark, Hive, Presto/Trino, Snowflake, BigQuery (for external tables), DuckDB, Databricks, and Polars. The table formats Delta Lake, Apache Iceberg, and Apache Hudi are all layers of metadata over collections of Parquet files.

Common Use Cases

Data lakes: the standard on-disk format for S3, GCS, and ADLS-based analytics.
ETL pipelines: intermediate storage between processing stages.
Analytical warehouses: external tables and bulk import/export.
Machine learning datasets: feature stores and training data too large for CSV.
Log and event archives: highly compressible, cheap to query selectively.
Data interchange: moving large tabular datasets between systems with schema and types intact.

How to Open a Parquet File

Command line

# DuckDB: query a Parquet file directly, no import step
duckdb -c "SELECT * FROM 'data.parquet' LIMIT 10;"
duckdb -c "DESCRIBE SELECT * FROM 'data.parquet';"

# parquet-tools
parquet-tools schema data.parquet
parquet-tools head -n 5 data.parquet
parquet-tools rowcount data.parquet

Python

import pandas as pd

# Read everything
df = pd.read_parquet("data.parquet")

# Read only the columns you need - the whole point of columnar storage
df = pd.read_parquet("data.parquet", columns=["user_id", "country"])

Inspecting metadata without reading the data:

import pyarrow.parquet as pq

pf = pq.ParquetFile("data.parquet")
print(pf.metadata)                    # rows, row groups, created_by
print(pf.schema_arrow)                # typed schema

rg = pf.metadata.row_group(0)
for i in range(rg.num_columns):
    col = rg.column(i)
    print(col.path_in_schema, col.statistics.min, col.statistics.max)

Writing

import pandas as pd

df.to_parquet("out.parquet", compression="zstd", index=False)

# Partitioned output: one directory level per partition key
df.to_parquet("dataset/", partition_cols=["year", "country"])

GUI viewers

Tad: a cross-platform Parquet and CSV viewer.
VS Code: the parquet-viewer extension renders files inline.
JetBrains DataGrip and DBeaver: open Parquet via DuckDB.

Advantages

Column pruning: read only the columns a query needs.
Predicate pushdown: skip whole row groups using footer statistics.
Strong compression: homogeneous columns compress far better than mixed rows.
Typed schema: types, nullability, and nested structure are preserved, unlike CSV.
Nested data support: lists, maps, and structs are first-class.
Universal support: every major analytical engine reads and writes it.
Splittable: row groups enable parallel distributed reads.

Limitations

Not for row-level access: fetching one complete record is slower than in a row store.
Immutable: no in-place updates or deletes; that is what Delta Lake and Iceberg add on top.
Write overhead: data is buffered per row group, so writing is more expensive than appending to CSV.
Not human-readable: requires tooling to inspect.
Small-file problem: many tiny Parquet files perform badly; the footer overhead dominates.
Corruption sensitivity: a truncated file with no footer is unreadable.

CSV: the row-based text format Parquet usually replaces.
JSON and JSONL: flexible but far slower and larger for analytics.
H5: HDF5, the scientific-computing counterpart for array data.
NPY: NumPy's array format, for dense numeric arrays rather than tables.
SQLITE: a row-oriented embedded database for transactional access.

What Is a .PARQUET File?