Download / Loading data

Loading the one file per asset schema

One file per asset, all with the same columns. Read a single file directly, or read the whole folder and keep track of which file each row came from.

pandas

Read one file, or loop over the folder and concatenate. The asset is taken from the file name.

import pandas as pd
from pathlib import Path

btc = pd.read_parquet("data/btc_usdt_1h.parquet")

df = pd.concat(
    {p.stem.split("_")[0].upper(): pd.read_parquet(p) for p in Path("data").glob("*.parquet")},
    names=["base"],
)

Polars

A glob pattern reads every file at once. include_file_paths adds the source file as a column.

import polars as pl

btc = pl.read_parquet("data/btc_usdt_1h.parquet")

df = pl.read_parquet("data/*.parquet", include_file_paths="file")

Polars (lazy)

Files are scanned together. Chain your operations, then call collect() last: selections apply before any data is loaded.

import polars as pl

lf = pl.scan_parquet("data/*.parquet", include_file_paths="file")

# Nothing is read yet: do whatever you want on lf
# (filter, select, group_by, with_columns, ...)

df = lf.collect()