pain001.parquet package#

Parquet data loaders for pain001 (optional feature).

Submodules#

pain001.parquet.load_parquet_data module#

Parquet data loader for payment data (optional feature).

pain001.parquet.load_parquet_data.load_parquet_data(file_path: str) → list[dict[str, Any]][source]#

Load payment data from Parquet file.

Parquet is a columnar storage format optimized for analytics workloads. It provides excellent compression and fast read performance for large datasets.

Parameters:

file_path – Path to Parquet file containing payment data.

Returns:

List of payment data dictionaries.

Raises:

  • FileNotFoundError – If file doesn’t exist.

  • DataSourceError – If pyarrow is not installed or the file is invalid or empty.

  • Exception – A FileNotFoundError or DataSourceError raised inside the read block is re-raised unchanged via a bare raise.

Examples

>>> data = load_parquet_data('payments.parquet')
# [{'id': 'MSG001', 'amount': '1000.00', ...}, ...]

Note

Requires pyarrow: pip install pyarrow

pain001.parquet.load_parquet_data.load_parquet_data_streaming(file_path: str, chunk_size: int = 1000) → Generator[list[dict[str, Any]], None, None][source]#

Load Parquet data in chunks for memory efficiency.

Uses pyarrow’s batch reader for true streaming without loading the entire file into memory.

Parameters:

  • file_path – Path to Parquet file.

  • chunk_size – Number of records per chunk (default: 1000).

Yields:

list[dict[str, Any]] – Chunks of payment data dictionaries.

Raises:

  • FileNotFoundError – If file doesn’t exist.

  • DataSourceError – If pyarrow is not installed or the file is invalid.

  • Exception – A FileNotFoundError or DataSourceError raised inside the read block is re-raised unchanged via a bare raise.

Examples

>>> for chunk in load_parquet_data_streaming('large_payments.parquet'):
...     process_batch(chunk)

Note

Requires pyarrow: pip install pyarrow