H5 (HDF5) File Format

Overview

The H5 format, also known as HDF5 (Hierarchical Data Format version 5), is a high-performance data management and storage format designed for storing and organizing large amounts of numerical data. Developed by the HDF Group, HDF5 is widely used in scientific computing, engineering, and data analysis applications for its ability to handle complex, heterogeneous data efficiently.

Technical Details

File Characteristics

• Extension: .h5, .hdf5
• MIME Type: application/x-hdf
• Category: Data
• Format Type: Binary hierarchical data format

Key Features

• Hierarchical Structure: Tree-like data organization
• Cross-Platform: Works across different operating systems and architectures
• Self-Describing: Metadata embedded within the file
• Extensible: Support for custom data types and attributes
• High Performance: Optimized for large-scale data operations

File Structure

Hierarchical Organization

HDF5 File Structure:
├── Root Group (/)
│   ├── Dataset1
│   ├── Dataset2
│   ├── Group1/
│   │   ├── SubDataset1
│   │   ├── SubDataset2
│   │   └── NestedGroup/
│   │       └── DeepDataset
│   ├── Group2/
│   │   ├── Array3D
│   │   └── TimeSeries
│   └── Attributes
│       ├── title: "Experimental Data"
│       ├── version: "1.0"
│       └── created: "2024-01-15"

Core Components

• Groups: Container objects that organize datasets
• Datasets: Multidimensional arrays of data
• Attributes: Metadata attached to groups or datasets
• Dataspaces: Define dimensionality and size of datasets
• Datatypes: Define data format and interpretation

Data Types and Structures

Native Data Types

import h5py
import numpy as np

# Create HDF5 file with various data types
with h5py.File('example.h5', 'w') as f:
    # Integer datasets
    f.create_dataset('integers', data=np.arange(100, dtype=np.int32))
    
    # Float datasets
    f.create_dataset('floats', data=np.random.random(1000).astype(np.float64))
    
    # String datasets
    string_data = np.array([b'hello', b'world', b'hdf5'], dtype='S10')
    f.create_dataset('strings', data=string_data)
    
    # Boolean datasets
    f.create_dataset('booleans', data=np.array([True, False, True]))
    
    # Complex numbers
    complex_data = np.array([1+2j, 3+4j, 5+6j])
    f.create_dataset('complex', data=complex_data)

Multidimensional Arrays

# 3D array example
with h5py.File('arrays.h5', 'w') as f:
    # Create 3D dataset
    data_3d = np.random.random((100, 50, 25))
    dataset = f.create_dataset('3d_array', data=data_3d)
    
    # Add attributes
    dataset.attrs['description'] = 'Random 3D array'
    dataset.attrs['units'] = 'meters'
    dataset.attrs['created_by'] = 'simulation_v1.0'
    
    # Create resizable dataset
    resizable = f.create_dataset('expandable', 
                                shape=(100, 100), 
                                maxshape=(None, 100),
                                dtype=np.float32)
    
    # Create chunked dataset for better performance
    chunked = f.create_dataset('chunked_data',
                              shape=(1000, 1000),
                              chunks=(100, 100),
                              dtype=np.float64)

Compound Data Types

# Create compound data type (like C struct)
dt = np.dtype([('name', 'S20'),
               ('age', 'i4'),
               ('weight', 'f8'),
               ('active', '?')])

# Sample data
people = np.array([('Alice', 25, 65.5, True),
                   ('Bob', 30, 75.0, False),
                   ('Charlie', 35, 80.2, True)], dtype=dt)

with h5py.File('compound.h5', 'w') as f:
    f.create_dataset('people', data=people)

Groups and Organization

Creating Hierarchical Structure

with h5py.File('organized.h5', 'w') as f:
    # Create groups
    experiment = f.create_group('experiment_001')
    raw_data = experiment.create_group('raw_data')
    processed = experiment.create_group('processed')
    
    # Add datasets to groups
    raw_data.create_dataset('temperature', data=np.random.random(1000))
    raw_data.create_dataset('pressure', data=np.random.random(1000))
    raw_data.create_dataset('timestamps', data=np.arange(1000))
    
    # Processed data
    processed.create_dataset('filtered_temp', data=np.random.random(800))
    processed.create_dataset('statistics', data=np.array([25.5, 1.2, 30.1]))
    
    # Add metadata
    experiment.attrs['date'] = '2024-01-15'
    experiment.attrs['researcher'] = 'Dr. Smith'
    experiment.attrs['equipment'] = 'Sensor Array v2.1'

Navigation and Access

# Reading hierarchical data
with h5py.File('organized.h5', 'r') as f:
    # Access by path
    temp_data = f['experiment_001/raw_data/temperature'][:]
    
    # Navigate using groups
    exp = f['experiment_001']
    raw = exp['raw_data']
    temperature = raw['temperature']
    
    # List contents
    print("Root contents:", list(f.keys()))
    print("Experiment contents:", list(exp.keys()))
    
    # Access attributes
    print("Date:", exp.attrs['date'])
    print("Researcher:", exp.attrs['researcher'])

Advanced Features

Compression and Filters

import h5py
import numpy as np

with h5py.File('compressed.h5', 'w') as f:
    data = np.random.random((1000, 1000))
    
    # GZIP compression
    f.create_dataset('gzip_data', data=data, 
                    compression='gzip', compression_opts=9)
    
    # SZIP compression
    f.create_dataset('szip_data', data=data,
                    compression='szip')
    
    # LZF compression (fast)
    f.create_dataset('lzf_data', data=data,
                    compression='lzf')
    
    # Custom chunking and compression
    f.create_dataset('optimized', data=data,
                    chunks=True,
                    compression='gzip',
                    compression_opts=6,
                    shuffle=True,
                    fletcher32=True)

Parallel I/O

# Parallel HDF5 with MPI
import h5py
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

# Parallel file access
with h5py.File('parallel.h5', 'w', driver='mpio', comm=comm) as f:
    # Each process writes to different section
    total_size = 1000
    local_size = total_size // size
    start = rank * local_size
    end = start + local_size
    
    # Create dataset collectively
    dset = f.create_dataset('parallel_data', (total_size,), dtype='f')
    
    # Write local data
    local_data = np.random.random(local_size)
    dset[start:end] = local_data

External Links

# Create external links between files
with h5py.File('main.h5', 'w') as f:
    # Create local data
    f.create_dataset('local_data', data=np.arange(100))
    
    # Link to external file
    f['external_link'] = h5py.ExternalLink('external.h5', '/dataset')

# Create the external file
with h5py.File('external.h5', 'w') as f:
    f.create_dataset('dataset', data=np.random.random(50))

# Access through link
with h5py.File('main.h5', 'r') as f:
    external_data = f['external_link'][:]

Scientific Computing Applications

Time Series Data

# Store time series with metadata
import pandas as pd
import h5py

# Generate sample time series
dates = pd.date_range('2024-01-01', periods=365, freq='D')
values = np.cumsum(np.random.randn(365))

with h5py.File('timeseries.h5', 'w') as f:
    # Store timestamps as Unix time
    timestamps = f.create_dataset('timestamps', 
                                 data=[d.timestamp() for d in dates])
    
    # Store values
    values_ds = f.create_dataset('values', data=values)
    
    # Add metadata
    f.attrs['start_date'] = dates[0].isoformat()
    f.attrs['end_date'] = dates[-1].isoformat()
    f.attrs['frequency'] = 'daily'
    f.attrs['units'] = 'arbitrary'
    
    timestamps.attrs['units'] = 'seconds since epoch'
    values_ds.attrs['description'] = 'Random walk time series'

Image Stack Storage

# Store stack of images
def store_image_stack(filename, images):
    with h5py.File(filename, 'w') as f:
        # Store image stack
        image_stack = f.create_dataset('images', 
                                      data=images,
                                      chunks=True,
                                      compression='gzip')
        
        # Add metadata
        f.attrs['num_images'] = images.shape[0]
        f.attrs['image_height'] = images.shape[1]
        f.attrs['image_width'] = images.shape[2]
        f.attrs['bit_depth'] = str(images.dtype)
        
        # Per-image metadata
        metadata_group = f.create_group('metadata')
        for i in range(images.shape[0]):
            img_meta = metadata_group.create_group(f'image_{i:03d}')
            img_meta.attrs['timestamp'] = f'2024-01-{i+1:02d}'
            img_meta.attrs['exposure_time'] = 0.1
            img_meta.attrs['gain'] = 1.0

# Usage
image_data = np.random.randint(0, 256, (100, 512, 512), dtype=np.uint8)
store_image_stack('images.h5', image_data)

Simulation Results

# Store complex simulation data
with h5py.File('simulation.h5', 'w') as f:
    # Simulation parameters
    params = f.create_group('parameters')
    params.attrs['simulation_type'] = 'molecular_dynamics'
    params.attrs['timestep'] = 0.001
    params.attrs['total_steps'] = 100000
    params.attrs['temperature'] = 300.0
    params.attrs['pressure'] = 1.0
    
    # Initial conditions
    initial = f.create_group('initial_conditions')
    initial.create_dataset('positions', data=np.random.random((1000, 3)))
    initial.create_dataset('velocities', data=np.random.random((1000, 3)))
    
    # Time evolution data
    trajectory = f.create_group('trajectory')
    
    # Store trajectory in chunks (for large simulations)
    n_atoms = 1000
    n_frames = 1000
    positions = trajectory.create_dataset('positions',
                                        shape=(n_frames, n_atoms, 3),
                                        chunks=(100, n_atoms, 3),
                                        compression='gzip',
                                        dtype=np.float32)
    
    # Fill with sample data
    for frame in range(0, n_frames, 100):
        end_frame = min(frame + 100, n_frames)
        chunk_data = np.random.random((end_frame - frame, n_atoms, 3))
        positions[frame:end_frame] = chunk_data
    
    # Analysis results
    analysis = f.create_group('analysis')
    analysis.create_dataset('energy', data=np.random.random(n_frames))
    analysis.create_dataset('temperature_profile', data=np.random.random(n_frames))

Performance Optimization

Chunking Strategies

# Optimize chunking for access patterns
with h5py.File('optimized.h5', 'w') as f:
    data = np.random.random((10000, 1000))
    
    # Row-wise access optimization
    row_chunked = f.create_dataset('row_access',
                                  data=data,
                                  chunks=(1, 1000))  # One row per chunk
    
    # Column-wise access optimization
    col_chunked = f.create_dataset('col_access',
                                  data=data,
                                  chunks=(10000, 1))  # One column per chunk
    
    # Balanced chunking
    balanced = f.create_dataset('balanced',
                               data=data,
                               chunks=(100, 100))  # Square chunks

Memory-Efficient Reading

# Read large datasets efficiently
def process_large_dataset(filename, dataset_name, chunk_size=1000):
    with h5py.File(filename, 'r') as f:
        dataset = f[dataset_name]
        total_rows = dataset.shape[0]
        
        results = []
        for start in range(0, total_rows, chunk_size):
            end = min(start + chunk_size, total_rows)
            chunk = dataset[start:end]
            
            # Process chunk
            result = np.mean(chunk, axis=1)
            results.append(result)
        
        return np.concatenate(results)

Integration with Data Science

Pandas Integration

import pandas as pd

# Store DataFrame in HDF5
df = pd.DataFrame({
    'A': np.random.randn(1000),
    'B': np.random.randn(1000),
    'C': pd.date_range('2024-01-01', periods=1000, freq='H')
})

# Save to HDF5 (PyTables format)
df.to_hdf('pandas_data.h5', key='df', mode='w')

# Read back
df_loaded = pd.read_hdf('pandas_data.h5', key='df')

NumPy Integration

# Direct NumPy array storage
arrays = {
    'array1': np.random.random((1000, 500)),
    'array2': np.random.random((2000, 300)),
    'array3': np.random.random((500, 1000))
}

with h5py.File('numpy_arrays.h5', 'w') as f:
    for name, array in arrays.items():
        f.create_dataset(name, data=array, compression='gzip')
        f[name].attrs['shape'] = array.shape
        f[name].attrs['dtype'] = str(array.dtype)

Best Practices

File Organization

• Use logical group hierarchy to organize related data
• Include comprehensive metadata and attributes
• Use meaningful names for groups and datasets
• Document data structure and conventions

Performance Considerations

• Choose appropriate chunk sizes for access patterns
• Use compression for large datasets
• Enable shuffling filter for better compression
• Consider parallel I/O for very large files

Data Integrity

• Use checksums (fletcher32) for critical data
• Implement proper error handling
• Validate data types and ranges
• Maintain backup copies of important files

The H5/HDF5 format provides a robust foundation for scientific data management, enabling efficient storage and access of complex, large-scale datasets while maintaining data integrity and cross-platform compatibility.