parquet-cli's Introduction

parquet-cli

Command line (CLI) tool to inspect Apache Parquet files on the go

Apache Parquet is a columnar storage format commonly used in the Hadoop ecosystem.

parq is small, easy to install, Python utility to view and get basic information from Parquet files.

Current features set are what I need, please use Github issues for any requests/suggestions.

Install

pip install parquet-cli

An executable script called parq will be installed.

Use

Once installed, you can use parq command.

View Parquet file metadata:

$ parq input.parquet

# Metadata
<pyarrow._parquet.FileMetaData object at 0x1014879a8>
created_by: parquet-mr version 1.8.1 (build 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)
num_columns: 13
num_rows: 1000
num_row_groups: 1
format_version: 1.0
serialized_size: 1125

Get schema information:

$ parq input.parquet --schema

# Schema
<pyarrow._parquet.ParquetSchema object at 0x1048b9a88>
registration_dttm: INT96
id: INT32
name: BYTE_ARRAY UTF8
email: BYTE_ARRAY UTF8
...
ip_address: BYTE_ARRAY UTF8
country: BYTE_ARRAY UTF8

Get total rows count:

$ parq input.parquet --count

Get top N records (head)

$ parq input.parquet --head 10

Get bottom N records (tail)

$ parq input.parquet --tail 10

Help

$ parq --help

usage: usage: parq file [-s [SCHEMA] | --head [HEAD] | --tail [TAIL] | -c [COUNT]]

positional arguments:
  file                  Parquet file

optional arguments:
  -h, --help            show this help message and exit
  -s [SCHEMA], --schema [SCHEMA]
                        get schema information
  --head [HEAD]         get first N rows from file
  --tail [TAIL]         get last N rows from file
  -c [COUNT], --count [COUNT]
                        get total rows count

Recommend Projects