Start building streaming
data applications today. In Python.

Open source framework and distributed stream processing engine. Build streaming data pipelines and real-time apps with everything you need: recovery, scalability, windowing, aggregations, and connectors.

User Guide GitHub

Quick start

First steps to explore
Bytewax features

User Guide

Explore all features of our framework, starting with simple examples

Learn more

Command-line interface

Learn more about waxctl - our command-line interface for deploying dataflows

Learn more

Guides

Learn how to build real-time data apps and streaming data pipelines with Bytewax.

Learn more

How it works

Build streaming data applications easily. In Python.

Step 1Easy install

> pip install bytewax

Step 2Connect to data sources

from bytewax import operators as op
from bytewax.connectors.kafka import operators as kop
from bytewax.dataflow import Dataflow

BROKERS = ["localhost:19092"]
IN_TOPICS = ["in_topic"]
OUT_TOPIC = "out_topic"

flow = Dataflow("kafka_in_out")
kinp = kop.input("inp", flow, brokers=BROKERS, topics=IN_TOPICS)
op.inspect("inspect-errors", kinp.errs)
op.inspect("inspect-oks", kinp.oks)
kop.output("out1", kinp.oks, brokers=BROKERS, topic=OUT_TOPIC)

Step 3Stateful operations like windowing and aggregations

from datetime import timedelta
import numpy as np
from bytewax.operators import window as window_op
from bytewax.operators.window import TumblingWindow, SystemClockConfig

cc = SystemClockConfig()
wc = TumblingWindow(length=timedelta(seconds=1))

def build_array():
    return np.empty(0)

def insert_value(np_array, value):
    return np.insert(np_array, 0, value)

windowed_stream = wop.fold_window("window", stream, cc, wc, build_array, insert_value)

Step 4Use the Python tools you are familiar with

import numpy as np

avg_stream = flow.map("average", windowed_stream, lambda x: np.mean(x[1]))

Step 5Run locally

> python -m bytewax.run my_dataflow:flow

Step 6Deploy anywhere

> waxctl df deploy my_dataflow.py

Getting started

Operators

Unite batch and stream processing with stateful streaming

Map

Apply a function to each element in the data stream

FlatMap

Each item can be transformed into zero or more elements

Filter

Keep only elements that match a condition

Inspect

Pass each element through and allows side effects

MapWindow

Transforms each item in a window into a new item

InspectWindow

Inspect, but for windowed operations

Reduce

Aggregate elements using a reducer function

Join

Combine elements from two streams based on a key

StatefulMap

Maintains state between elements, allowing for more complex transformations.

Aggregate

Allows initialization, state update, and result computation

Window

Group elements into windows based on time or count, maintaining state for each

Custom Operators

Define your own operators based on your own semantic (e.g. interval joins)

Discover all operators

Guides

Learn how to build real-time data apps with Bytewax

Tutorial

Building Sessions from Search Logs

This guide will teach you how to use Bytewax to aggregate on a custom session window on streaming data using reduce and then calculate metrics downstream.

Easy

15-30 min

Sample App

Real-Time Financial Exchange Order Book

At the end of this tutorial you will understand how to use Bytewax to analyze financial exchange data. You will learn to establish connections to a WebSocket for real-time data, use Bytewax's operators to efficiently manage an order book, and apply analytical techniques to assess trading opportunities based on the dynamics of buy and sell orders.

Intermediate

15-30 min

Tutorial

Recoverable Streaming Shopping Cart Application

This tutorial will show you how to build a recoverable streaming application that aggregates shoppers data into a shopping cart that is recoverable in the instance that it fails.

Intermediate

15 min

Tutorial

Profiling Streaming Time Series Data

You'll be able to handle and structure data streams into snapshots using Bytewax, and then analyze them with ydata-profiling to create a comprehensive report of data characteristics for each device at each time interval.

Intermediate

15-30 min

Tutorial

Handling Missing Values in Data Streams

Learn how to create a custom sliding window with the stateful map operator to impute values using numpy

Easy

15 min

Bytewax can be used out-of-the box with any python library

Easy integrations

Leverage the Python Ecosystem

Bytewax can be used out-of-the box with any python library to connect to hundreds of data sources and use the entire ecosystem of data processing libraries.

Check guides

Articles