Stream processing
purely in Python

Open source framework and distributed stream processing engine. Build streaming data pipelines and real-time apps with everything you need: recovery, scalability, windowing, aggregations, and connectors.

Leverage the Python Ecosystem

Bytewax can be used out-of-the box with any python library to connect to hundreds of data sources and use the entire ecosystem of data processing libraries.
Scrapy, PyTorch, Huggingface, Pandas, Numpy, Tensorflow, Streamlit, Polars, spaCy, Requests, scikit learn, Matplotlib, SQLAlchemy
How it works

Build streaming data applications easily. In Python.

1

Easy install 

> pip install bytewax
2

Stateful operations like windowing and aggregations

from datetime import timedelta
import numpy as np

from bytewax.window import TumblingWindow, SystemClockConfig

cc = SystemClockConfig()
wc = TumblingWindow(length=timedelta(seconds=1))

def build_array():
    return np.empty(0)

def insert_value(np_array, value):
    return np.insert(np_array, 0, value)

flow.fold_window("window", cc, wc, build_array, insert_value)
3

Use the Python tools you are familiar with

import numpy as np

flow.map(lambda x: np.mean(x[1]))
4

Connect to data sources

from bytewax.dataflow import Dataflow
from bytewax.connectors.kafka import KafkaInput, KafkaOutput

flow = Dataflow()
flow.input(
    "events",
    KafkaInput(
        brokers=["localhost:9092"],
        topic="web_events"
    )
)
flow.output(
    "parsed_ip_location",
    KafkaOutput(
        brokers=["localhost:9092"],
        topic="ip_address_by_location"
    )
)
5

Run locally

> python -m bytewax.run my_dataflow:flow
6

Deploy anywhere

> waxctl df deploy my_dataflow.py
Main features
code

Python Native

No JVM required! Leverage the entirety of the python ecosystem, from Jupyter notebooks and Hugging Face transformers to Streamlit.

join_right

Stateful

Stateful operators allow you to do advanced processing like joins, windows, and aggregations

restore

Recoverable

Built-in state recovery with multiple state backend options for critical workloads

schema

Native Connectors

Built-in integrations for Kafka, Redpanda, DynamoDB, BigQuery, and more...

cloud

Scalable

Bytewax can scale across thousands of workers to meet the most demanding workloads.

speed

Performant

The Rust engine keeps things performant.

Guides

What can you build with bytewax

YData & Bytewax
Tutorial

Profiling Streaming Time Series Data

You'll be able to handle and structure data streams into snapshots using Bytewax, and then analyze them with ydata-profiling to create a comprehensive report of data characteristics for each device at each time interval.
Recoverable shopping cart
Tutorial

Recoverable Streaming Shopping Cart Application

This tutorial will show you how to build a recoverable streaming application that aggregates shoppers data into a shopping cart that is recoverable in the instance that it fails.
Enriching data from Kafka
Tutorial

Enriching Streaming Data from Redpanda

Enriching streaming data inline is a common pattern. This tutorial will show you how you can do this with Python and provide you with code you can modify to build your own enrichment pipelines.
Cryptocurrency order book
Sample App

Real-Time Financial Exchange Order Book

At the end of this tutorial users will understand how to use Bytewax to analyze financial exchange data by connecting to a websocket and using Bytewax operators to maintain an order book and analyzing the orders.
Sessions from search logs
Tutorial

Building Sessions from Search Logs

This guide will teach you how to use Bytewax to aggregate on a custom session window on streaming data using reduce and then calculate metrics downstream.
Missing values in data streams
Tutorial

Handling Missing Values in Data Streams

Learn how to create a custom sliding window with the stateful map operator to impute values using numpy
Community buzz

Loved by data

Loading...
Loading...
Loading...
Loading...
Loading...