Download / Loading data

Loading the long schema

Plain columns, no index: close_ts and base come back as regular columns in pandas and in Polars.

pandas

Filter on base like any other column. The filters argument of read_parquet is applied while reading, so only the matching rows are loaded.

import pandas as pd

df = pd.read_parquet("ohlcv_usdt_1h.parquet")
btc = df[df["base"] == "BTC"]
closes = df.pivot(index="close_ts", columns="base", values="close")

# Read only the rows and columns you need
btc_close = pd.read_parquet(
    "ohlcv_usdt_1h.parquet",
    columns=["close_ts", "close"],
    filters=[("base", "==", "BTC")],
)

Polars

Filter on base, or pivot to get one column per asset.

import polars as pl

df = pl.read_parquet("ohlcv_usdt_1h.parquet")
btc = df.filter(pl.col("base") == "BTC")
closes = df.pivot(on="base", index="close_ts", values="close")

Polars (lazy)

scan_parquet reads nothing yet: it only builds a query. Chain your operations on it, then call collect() last to run the whole query and load the result. The filter and the column selection are pushed down to the file, so only the matching rows and columns are read.

import polars as pl

lf = pl.scan_parquet("ohlcv_usdt_1h.parquet")

# Nothing is read yet: do whatever you want on lf
# (filter, select, group_by, with_columns, ...)

df = lf.collect()