Overview

Bytewax is an open source Python framework for building highly scalable dataflows in a streaming or batch context.

Dataflow programming is a programming paradigm where program execution is conceptualized as data flowing through a series of operations or transformations.

Bytewax is a Python native binding to the Timely Dataflow library. Timely Dataflow is a dataflow processing library written in Rust.

Benefits

At a high level, Bytewax provides a few major benefits:

• The operators in Bytewax are largely "data-parallel", meaning they can operate on independent parts of the data concurrently.
• Bytewax offers the ability to express higher-level control constructs, like iteration.
• Bytewax allows you to develop and run your code locally, and then easily scale that code to multiple workers or processes without changes.