Stop using Pandas and CSV

In data work, Pandas and CSV are the default stack. Everyone knows them, every tutorial uses them, and for a small spreadsheet they're perfectly fine. But as soon as your datasets grow, years of minute candles or tick data, they become the bottleneck: slow to load, heavy on disk, and greedy with RAM.

There's a better pair of tools, and switching costs almost nothing. I first came across this stack in a video by MemLabs, and it changed how I handle market data. Here's what to use instead, and why.

1. Use Parquet instead of CSV

Typed binary instead of text

CSV is a row-oriented text format. Every number is stored as characters and has to be parsed back on read, and the format carries no types at all. Parquet is a column-oriented binary format with native types (int32, float64, string, timestamps), so what you write is exactly what you read back, with no conversion step.

That also gets rid of a whole family of bugs we've all hit at some point: a stray , or ; hiding inside a text field and shifting every column, or decimals written with . in one file and , in another.

Built-in compression

CSV has no compression, which is costly when values repeat a lot, and market data repeats a lot: the same symbol on every row, prices that move a few ticks at a time. Parquet compresses each column natively.

On OHLCV-like data, the Parquet file is about 4× smaller than the same data as CSV, and the gap stays constant as you add rows or columns.

Read only the columns you need

To read a single column from a CSV, you still have to parse the whole file. Parquet stores each column separately, so it reads only the columns you ask for.

At 5 million rows, reading the full file takes about 2.8 s as CSV versus 0.2 s as Parquet. Reading a single column from the Parquet file takes a few hundredths of a second.

When CSV is still the right call

CSV has one real advantage: it opens in Excel and every other tool, so it's easy to share with anyone. If you're sending a small file to a colleague, CSV is fine. But for large datasets, or for data that stays in your code, Parquet is the better format.

Switching is a one-line change:

import pandas as pd

df = pd.read_csv("ohlcv_1m.csv")
df.to_parquet("ohlcv_1m.parquet")  # once

df = pd.read_parquet("ohlcv_1m.parquet")  # from now on

2. Use Polars instead of Pandas

Changing the file format is half the story. The other half is the library that reads it.

Pandas loads the entire dataframe into RAM as soon as you read it, then runs each operation one after the other on a single core. Polars is column-oriented and multi-threaded, and it supports lazy evaluation: with pl.scan_parquet, you describe the whole query first, and Polars optimizes it before loading anything. It only reads the columns and rows the query actually needs.

import polars as pl

daily_volume = (
    pl.scan_parquet("ohlcv_1m.parquet")       # nothing is loaded yet
    .filter(pl.col("base") == "BTC")
    .group_by(pl.col("close_ts").dt.date())
    .agg(pl.col("base_volume").sum())
    .collect()                                # optimized, then executed
)

On a read + filter + group-by over 10 million rows, Polars in lazy mode peaks at 369 MB of RAM, against 3.1 GB for Pandas, and finishes about 10× faster.

The short version

• Store your data as Parquet: typed, compressed, and you only read the columns you need.
• Load it with Polars, and use scan_parquet for anything large so the query is optimized before the data hits RAM.
• Keep CSV for the small files you need to share with people who live in Excel.

Try it on real market data

This is also why every Angolin dataset is delivered as Parquet. You don't even need an API key: pick the exchange, assets, interval and schema on the Download page of your account, and you get a ready-to-use .parquet file. See how downloads work, then the loading examples, which show Polars alongside pandas.

Thanks to MemLabs for the video that introduced me to this stack.