Weston Pace

Weston Pace

Data engineer from the open source space, working on LanceDB, Arrow, Substrait.

Engineering
Columnar File Readers in Depth: Structural Encoding

Columnar File Readers in Depth: Structural Encoding

Deep dive into LanceDB's dual structural encoding approach - mini-block for small data types and full-zip for large multimodal data. Learn how this optimizes compression and random access performance compared to Parquet.

Engineering
Columnar File Readers in Depth: APIs and Fusion

Columnar File Readers in Depth: APIs and Fusion

The API used to read files has evolved over time, from simple full table reads to batch reads and eventually to iterative record batch readers. Lance takes this a step further to return a stream of read tasks.