bzip2 Compressed Data Format

Overview

bzip2 is a lossless data compression algorithm and file format developed by Julian Seward. It uses the Burrows-Wheeler transform and Huffman coding to achieve better compression ratios than gzip, though at the cost of slower compression and decompression speeds. The format is widely used in Unix-like systems for archiving and data compression.

Technical Details

File Extension: .bz2
MIME Type: application/x-bzip2
Compression Algorithm: Burrows-Wheeler transform + Huffman coding
Block Size: 100KB to 900KB (configurable)
Magic Number: BZ (0x425A)
Maximum File Size: Unlimited (stream-based)

bzip2 uses a multi-stage compression process:

1. Burrows-Wheeler Transform (BWT)
2. Move-to-Front Transform (MTF)
3. Run-Length Encoding (RLE)
4. Huffman Coding

Key Features

• High Compression Ratio: Better than gzip, competitive with modern algorithms
• Error Recovery: Block-based structure allows partial recovery
• Multi-threaded: Parallel compression/decompression support
• Open Source: Free implementation with no patents
• Cross-Platform: Available on all major operating systems
• Stream Processing: Can compress/decompress without loading entire file

Compression Performance

Comparison with Other Formats

Algorithm    Compression Ratio    Speed      Memory Usage
bzip2        High (85-90%)        Slow       Moderate
gzip         Medium (75-85%)      Fast       Low
xz/LZMA      Very High (90-95%)   Very Slow  High
LZ4          Low (60-70%)         Very Fast  Very Low

Block Size Impact

# Small blocks (100KB) - faster, less compression
bzip2 -1 file.txt

# Large blocks (900KB) - slower, better compression
bzip2 -9 file.txt

# Default block size (900KB)
bzip2 file.txt

Common Use Cases

1. File Archiving: Long-term storage with good compression
2. Software Distribution: Compressing source code and binaries
3. Backup Systems: Reducing backup storage requirements
4. Data Transfer: Minimizing bandwidth usage
5. Log File Compression: Compressing rotated log files
6. Scientific Data: Compressing large datasets

Command Line Usage

Basic Compression

# Compress file (replaces original)
bzip2 file.txt

# Compress keeping original
bzip2 -k file.txt

# Compress with specific compression level (1-9)
bzip2 -9 file.txt

# Compress to stdout
bzip2 -c file.txt > file.txt.bz2

# Compress multiple files
bzip2 file1.txt file2.txt file3.txt

Decompression

# Decompress file
bunzip2 file.txt.bz2

# Decompress keeping compressed file
bunzip2 -k file.txt.bz2

# Decompress to stdout
bunzip2 -c file.txt.bz2

# Test archive integrity
bunzip2 -t file.txt.bz2

# Force decompression
bunzip2 -f file.txt.bz2

Advanced Options

# Verbose output
bzip2 -v file.txt

# Very verbose (show compression statistics)
bzip2 -vv file.txt

# Small memory usage (slower)
bzip2 -s file.txt

# Parallel compression (pbzip2)
pbzip2 -p4 file.txt  # Use 4 processors

Programming APIs

C Library

#include <bzlib.h>

// Compression example
FILE *input = fopen("input.txt", "rb");
FILE *output = fopen("output.bz2", "wb");

BZFILE *bzfile = BZ2_bzWriteOpen(&bzerror, output, 9, 0, 0);

char buffer[1024];
int bytes_read;
while ((bytes_read = fread(buffer, 1, sizeof(buffer), input)) > 0) {
    BZ2_bzWrite(&bzerror, bzfile, buffer, bytes_read);
}

BZ2_bzWriteClose(&bzerror, bzfile, 0, NULL, NULL);
fclose(input);
fclose(output);

// Decompression example
FILE *compressed = fopen("input.bz2", "rb");
FILE *decompressed = fopen("output.txt", "wb");

BZFILE *bzfile = BZ2_bzReadOpen(&bzerror, compressed, 0, 0, NULL, 0);

char buffer[1024];
int bytes_read;
while ((bytes_read = BZ2_bzRead(&bzerror, bzfile, buffer, sizeof(buffer))) > 0) {
    fwrite(buffer, 1, bytes_read, decompressed);
}

BZ2_bzReadClose(&bzerror, bzfile);
fclose(compressed);
fclose(decompressed);

Python

import bz2

# Compression
with open('input.txt', 'rb') as input_file:
    with bz2.BZ2File('output.bz2', 'wb', compresslevel=9) as output_file:
        output_file.write(input_file.read())

# Decompression
with bz2.BZ2File('input.bz2', 'rb') as input_file:
    with open('output.txt', 'wb') as output_file:
        output_file.write(input_file.read())

# String compression
text = "Hello, World! " * 1000
compressed = bz2.compress(text.encode('utf-8'))
decompressed = bz2.decompress(compressed).decode('utf-8')

# Incremental compression
compressor = bz2.BZ2Compressor(compresslevel=9)
compressed_data = compressor.compress(b"First chunk")
compressed_data += compressor.compress(b"Second chunk")
compressed_data += compressor.flush()

Java

import org.apache.commons.compress.compressors.bzip2.*;

// Compression
FileInputStream input = new FileInputStream("input.txt");
FileOutputStream output = new FileOutputStream("output.bz2");
BZip2CompressorOutputStream bzOut = new BZip2CompressorOutputStream(output);

byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = input.read(buffer)) != -1) {
    bzOut.write(buffer, 0, bytesRead);
}

bzOut.close();
input.close();

// Decompression
FileInputStream compressed = new FileInputStream("input.bz2");
BZip2CompressorInputStream bzIn = new BZip2CompressorInputStream(compressed);
FileOutputStream decompressed = new FileOutputStream("output.txt");

byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = bzIn.read(buffer)) != -1) {
    decompressed.write(buffer, 0, bytesRead);
}

bzIn.close();
decompressed.close();

File Format Structure

Header Format

bzip2 File Structure:
├── Magic Number (2 bytes): "BZ"
├── Version (1 byte): 'h' for bzip2
├── Block Size (1 byte): '1'-'9'
└── Compressed Blocks
    ├── Block Header
    ├── Block Magic: 0x314159265359 (π)
    ├── Block CRC (4 bytes)
    ├── Randomized (1 bit)
    ├── Origptr (24 bits)
    ├── Huffman Tables
    └── Compressed Data

Block Structure

Each Block Contains:
├── Block magic number
├── Block CRC32
├── Randomization flag
├── Original pointer
├── Huffman mapping tables
└── Huffman-encoded data

Optimization Techniques

Compression Level Selection

# Fast compression (level 1)
bzip2 -1 file.txt  # ~50% compression, fastest

# Balanced (level 6, default)
bzip2 -6 file.txt  # ~75% compression, moderate speed

# Maximum compression (level 9)
bzip2 -9 file.txt  # ~85% compression, slowest

Memory Usage Control

# Reduce memory usage (slower)
bzip2 -s file.txt

# Monitor memory usage
bzip2 -v -s file.txt

Parallel Processing

# Using pbzip2 for multi-core compression
pbzip2 -p8 large_file.txt  # Use 8 CPU cores

# Parallel with memory limit
pbzip2 -p4 -m500 file.txt  # 4 cores, 500MB memory limit

Integration with Archives

tar + bzip2

# Create compressed tar archive
tar -cjf archive.tar.bz2 directory/

# Extract compressed tar archive
tar -xjf archive.tar.bz2

# List contents
tar -tjf archive.tar.bz2

# Add compression level
tar --bzip2 -cf archive.tar.bz2 directory/

Combined with Other Tools

# Pipe compression
cat large_file.txt | bzip2 -c > compressed.bz2

# Database dump compression
mysqldump database | bzip2 > backup.sql.bz2

# Log rotation with compression
logrotate --compress --compresscmd=/bin/bzip2

Error Handling and Recovery

Integrity Testing

# Test file integrity
bzip2 -t file.bz2

# Test and show progress
bzip2 -tv file.bz2

# Detailed testing
bunzip2 -t file.bz2 && echo "File is valid"

Recovery from Corruption

# Attempt recovery
bzip2recover damaged_file.bz2

# This creates rec00001damaged_file.bz2, rec00002damaged_file.bz2, etc.
# Try to decompress each recovered block
for file in rec*damaged_file.bz2; do
    bunzip2 -t "$file" && echo "$file is recoverable"
done

Performance Considerations

When to Use bzip2

• Good for: Archival storage, slow networks, storage-constrained systems
• Avoid for: Real-time compression, low-latency applications, frequent access

Alternatives Comparison

# Speed comparison for 100MB file
time gzip large_file.txt      # ~2 seconds, 70% compression
time bzip2 large_file.txt     # ~8 seconds, 85% compression
time xz large_file.txt        # ~15 seconds, 90% compression
time lz4 large_file.txt       # ~0.5 seconds, 60% compression

bzip2 continues to be a valuable compression tool, offering an excellent balance of compression ratio and widespread compatibility, making it ideal for archival purposes and scenarios where storage space is more critical than compression speed.