Weston Pace

Weston Pace

Data engineer from the open source space, working on LanceDB, Arrow, Substrait.

Engineering
Columnar File Readers in Depth: Backpressure

Columnar File Readers in Depth: Backpressure

Streaming data applications can be tricky. When you can read data faster than you can process the data then bad things tend to happen. The various solutions to this problem are largely classified as backpressure.