Bytewax is instrumented to offer observability of your dataflow.
The default configuration logs anything at the log level
ERROR to standard output.
You can control the log level by passing the
log_level parameter to the
If you want to see all the messages bytewax emits, set the level to
TRACE level includes everything that would be sent to an opentelemetry compatible backend,
like Jaeger, or the Opentelemetry Collector.
It is really verbose, and your stdoutput will be flooded with logs, so use it carefully.
Let's try to see what
jaeger can show us about a dataflow.
We will make bytewax talk to the opentelemetry collector, and let the collector send everything to a jaeger instance.
We will use the wikistream.py example as a reference,
since it doesn't require any other setup.
You will need docker and docker-compose to run this example.
Create a folder where you'll keep the dataflow and two more files we'll need to run everything.
mkdir bytewax-tracing cd bytewax-tracing
Now create a configuration file for the collector, telling it to receive traces from grpc and send them to jaeger, with the following content:
# file: otel-collector-config.yml receivers: otlp: protocols: grpc: exporters: jaeger: endpoint: jaeger:14250 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors:  exporters: [jaeger]
Then create a docker compose file to run both jaeger and the collector, mounting the config we just wrote:
# file: docker-compose.yml version: "3" services: jaeger: image: jaegertracing/all-in-one:latest ports: - "16686:16686" - "14250" otel-collector: image: otel/opentelemetry-collector:latest command: ["--config=/etc/otel-collector-config.yml"] volumes: - type: "bind" source: ./otel-collector-config.yml target: /etc/otel-collector-config.yml ports: - "4317:4317" depends_on: - jaeger
docker compose up and everything should be up and running.
Now we need the dataflow. Download the example in this folder:
curl https://raw.githubusercontent.com/bytewax/bytewax/main/examples/wikistream.py -o dataflow.py
To instrument your dataflow, call
bytewax.tracing with the config object you want, and keep the returned object around (if you don't assign to the
tracer variable, tracing would not work)
# file: dataflow.py from bytewax.tracing import OtlpTracingConfig, setup_tracing tracer = setup_tracing( tracing_config=OtlpTracingConfig( service_name="Wikistream", url="grpc://127.0.0.1:4317", ), log_level="ERROR", ) # # ...rest of the file #
Create a virtual environment and install the needed dependencies:
python3 -m venv .venv source .venv/bin/activate # Or activate.fish on fish shell pip install bytewax sseclient-py urllib3
Now you can run it with:
Open your browser at http://127.0.0.1:16686 and take a look at traces coming into Jaeger's UI.