Angolin Docs
Download / Universe

Long schema

Layout

One row per time slot and listed pair: close_ts, base, quote. A pair that is not listed during a slot has no row. In this daily sample, LUNC is still listed on 2022-05-13 (the row closing 2022-05-14 00:00), since it was tradable until 01:00 that day, and gone the next.

close_tsbasequote
2022-05-13T00:00:00ZBTCUSDT
2022-05-13T00:00:00ZLUNCUSDT
2022-05-14T00:00:00ZBTCUSDT
2022-05-14T00:00:00ZLUNCUSDT
2022-05-15T00:00:00ZBTCUSDT

Use it with the long OHLCV schema: join both on close_ts and base to keep only the candles of pairs that were tradable. It also works with every quote at once.

Plain columns, no index: close_ts, base and quote. The examples keep only the candles of a long OHLCV file whose pair was tradable at that time.

pandas

An inner merge on close_ts and base keeps the candles whose pair was listed. Read only these two columns of the universe.

import pandas as pd

ohlcv = pd.read_parquet("ohlcv_usdt_1h.parquet")
listed = pd.read_parquet("universe_binance_usdt_1h.parquet", columns=["close_ts", "base"])

# Keep only the candles of pairs that were tradable at that time
tradable = ohlcv.merge(listed, on=["close_ts", "base"])

Polars

A semi join keeps the candles whose pair was listed, without adding any column.

import polars as pl

ohlcv = pl.read_parquet("ohlcv_usdt_1h.parquet")
listed = pl.read_parquet("universe_binance_usdt_1h.parquet")

# Keep only the candles of pairs that were tradable at that time
tradable = ohlcv.join(listed, on=["close_ts", "base"], how="semi", maintain_order="left")

Polars (lazy)

Scan both files and join them: nothing is read until collect() runs the whole query.

import polars as pl

ohlcv = pl.scan_parquet("ohlcv_usdt_1h.parquet")
listed = pl.scan_parquet("universe_binance_usdt_1h.parquet")

# Nothing is read yet: the join runs, and the files are read, on collect()
tradable = ohlcv.join(listed, on=["close_ts", "base"], how="semi", maintain_order="left").collect()