Stream processing
purely in Python

Open source framework and distributed stream processing engine. Build streaming data pipelines and real-time apps with everything you need: recovery, scalability, windowing, aggregations, and connectors.

Leverage the Python Ecosystem

Bytewax can be used out-of-the box with any python library to connect to hundreds of data sources and use the entire ecosystem of data processing libraries.
Scrapy, PyTorch, Huggingface, Pandas, Numpy, Tensorflow, Streamlit, Polars, spaCy, Requests, scikit learn, Matplotlib, SQLAlchemy
How it works

Build streaming data applications easily. In Python.

1

Easy install

> pip install bytewax
2

Stateful operations like windowing and aggregations

from datetime import timedelta
import numpy as np

from bytewax.window import TumblingWindowConfig, SystemClockConfig

cc = SystemClockConfig()
wc = TumblingWindowConfig(length=timedelta(seconds=1))

def build_array():
    return np.empty(0)

def insert_value(np_array, value):
    return np.insert(np_array, 0, value)

flow.fold_window("window", cc, wc, build_array, insert_value)
3

Use the Python tools you are familiar with

import numpy as np

flow.map(lambda x: np.mean(x[1]))
4

Connect to data sources

flow.input(
    "events",
    KafkaInputConfig(
        brokers=["localhost:9092"],
        topic="web_events"
    )
)
flow.capture(
    KafkaOutputConfig(
        brokers=["localhost:9092"],
        topic="ip_address_by_location"
    )
)
5

Run locally

> python my_dataflow.py
6

Deploy anywhere

> waxctl df deploy my_dataflow.py
Main features
code

Python Native

No JVM required! Leverage the entirety of the python ecosystem, from Jupyter notebooks and Hugging Face transformers to Streamlit.

join_right

Stateful

Stateful operators allow you to do advanced processing like joins, windows, and aggregations

restore

Recoverable

Built-in state recovery with multiple state backend options for critical workloads

schema

Native Connectors

Built-in integrations for Kafka, Redpanda, DynamoDB, BigQuery, and more...

cloud

Scalable

Bytewax can scale across thousands of workers to meet the most demanding workloads.

speed

Performant

The Rust engine keeps things performant.

Community buzz

Loved by data

Loading...
Loading...
Loading...
Loading...
Loading...