Stream processing
purely in Python
Open source framework and distributed stream processing engine. Build streaming data pipelines and real-time apps with everything you need: recovery, scalability, windowing, aggregations, and connectors.
Leverage the Python Ecosystem
Bytewax can be used out-of-the box with any python library to connect to hundreds of data sources and use the entire ecosystem of data processing libraries.
How it worksGetting startedarrow_forward
Build streaming data applications easily. In Python.
1
Easy install
> pip install bytewax
2
Stateful operations like windowing and aggregations
from datetime import timedelta
import numpy as np
from bytewax.window import TumblingWindow, SystemClockConfig
cc = SystemClockConfig()
wc = TumblingWindow(length=timedelta(seconds=1))
def build_array():
return np.empty(0)
def insert_value(np_array, value):
return np.insert(np_array, 0, value)
flow.fold_window("window", cc, wc, build_array, insert_value)
3
Use the Python tools you are familiar with
import numpy as np
flow.map(lambda x: np.mean(x[1]))
4
Connect to data sources
from bytewax.dataflow import Dataflow
from bytewax.connectors.kafka import KafkaInput, KafkaOutput
flow = Dataflow()
flow.input(
"events",
KafkaInput(
brokers=["localhost:9092"],
topic="web_events"
)
)
flow.output(
"parsed_ip_location",
KafkaOutput(
brokers=["localhost:9092"],
topic="ip_address_by_location"
)
)
5
Run locally
> python -m bytewax.run my_dataflow:flow
6
Deploy anywhere
> waxctl df deploy my_dataflow.py
Main features
code
Python Native
No JVM required! Leverage the entirety of the python ecosystem, from Jupyter notebooks and Hugging Face transformers to Streamlit.
join_right
Stateful
Stateful operators allow you to do advanced processing like joins, windows, and aggregations
restore
Recoverable
Built-in state recovery with multiple state backend options for critical workloads
schema
Native Connectors
Built-in integrations for Kafka, Redpanda, DynamoDB, BigQuery, and more...
cloud
Scalable
Bytewax can scale across thousands of workers to meet the most demanding workloads.
speed
Performant
The Rust engine keeps things performant.
Community buzz
Loved by data
Loading...
Loading...
Loading...
Loading...
Loading...