Choosing the Right Streaming Platform: Kafka vs. Pulsar vs. NATS

When it comes to streaming data, there are three open source options that stand out: Apache Kafka, Apache Pulsar, and NATS. These systems are fundamental in handling large-scale, streaming data with reliability and efficiency. In this blog post, we will make a comparative analysis of Kafka, Pulsar, and NATS across five key dimensions: performance, compatibility, ecosystem, community, and reliability.

Ultimately, we'll see why the Kafka ecosystem, with its vibrant growth and compatible alternatives like Redpanda and Warpstream, makes it a compelling choice for many organizations.

Background

Apache Kafka was originally designed and developed at LinkedIn by Jay Kreps, Neha Narkhede and Jun Rao. It was open sourced in 2011. Jay, Neha and Jun started Confluent in 2014 after leaving LinkedIn to form a company around Kafka to service enterprise needs. Confluent went public in 2021, ten years after Kafka was open sourced. Today managed Apache Kafka is offered by dozens of companies including Amazon, Google and Azure and Apache Kafka is core to many companies as an internal platform or part of an external product.

NATS was initially created by Derek Collison while at Cloud Foundry in 2013 it was open sourced. Derek later went on to create Synadia to support the needs that came up as a result of wider adoption of NATS. NATS differs from Pulsar and Kafka in that it is not an Apache Foundation project, but is rather a Cloud Native Compute Foundation project. NATS started as a message protocol, but has grown to include many functionalities that are required of a streaming platform.

Apache Pulsar came out of Yahoo! and was open sourced in 2016. Two of the creators, Matteo Merli and Sijoie Guo, created StreamNative to support enterprise adoption of Pulsar in 2019. Pulsar created to support efforts to move workloads to the cloud via new technology created since the creation of Kafka like containerization via docker and orchestration with Kubernetes. Pulsar was a different approach with the separation of serving and storage and the first cloud native streaming platform.

Performance

When you hear performance, you probably think benchmark. Raw speed and throughput. Benchmarking should always be taken with a grain of salt. I have included a few benchmarks at the bottom of this section for your own comparison. Most often benchmarks are either directly done by the vendor or paid for by the vendor. Oftentimes you can spot the subtle skew of the benchmark towards the strength of the vendor. For some fun reading, you can checkout how Kai Waehner took things one step further on his blog aimed directly at Pulsar. Kai sells Kafka as a field CTO, so it's hard to believe his post is completely unbiased.

I digress. Here are some high-level differences on the performance of the three and the trade-offs made in pursuit of performance.

Kafka is renowned for its high throughput and durability. It uses a distributed commit log model, allowing it to process millions of messages per second. Kafka's performance is highly scalable and fault-tolerant, making it suitable for extensive enterprise applications.

Pulsar boasts similar high performance but adds another layer with its unique architecture that separates the serving and storage layers. This design enables Pulsar to offer better resource utilization, which can lead to improved scalability and potentially lower operational costs.

NATS, on the other hand, is lightweight and incredibly fast in scenarios that require low latency and less persistent message delivery. Its performance shines in systems where the primary requirement is the rapid transmission of small messages across distributed systems.

For more details on the technical benchmarking of performance, you should review the posts made by the individual companies comparing these to get a sense of how each positions their solution as the most performant 😁.

• Confluent put out it's benchmarking blog, aptly named in the url slug "kafka-fastest-messaging-system".
• Streamnative decided to avoid Kafka and published their own "winning" benchmark on their blog.
• And Synadia decided to hire out their benchmark with a focus on total cost of ownership and you can read the report here

Performance is of varying importance depending on your use case, but ultimately there are other considerations to take into account in your decision making. Like reliability and compatibility.

Reliability

Kafka is designed for fault tolerance and durability. It replicates data and can handle failures of individual nodes without data loss, making it highly reliable for critical applications.

Pulsar provides similar levels of reliability and adds features like geo-replication out-of-the-box, enhancing data safety and availability across multiple regions.

NATS offers high reliability in terms of delivering messages with minimal latency and supports various levels of Quality of Service (QoS). However, it doesn’t natively handle durable storage or replication as Kafka and Pulsar do.

Compatibility

Kafka supports a wide range of programming languages and platforms, thanks to its extensive client libraries and robust API support. It integrates well with big data technologies and can connect to popular data sources and sinks via Kafka Connect. Confluent, the company created by the Kafka creators, maintains many clients for different languages.

Pulsar also supports multiple languages and has its own Pulsar Functions feature for stream-native processing. Its compatibility with the Kafka API (via Kafka-on-Pulsar) means it can act as a drop-in replacement in many Kafka-based systems, which is a significant advantage.

NATS focuses on simplicity and speed, supporting numerous languages but primarily shines in environments that don't require the robust transactional guarantees or stream processing integrations that Kafka offers. NATS is not kafka API compatible and relies on a number of community developed clients.

Ease of Use

Kafka has a steep learning curve due to its rich feature set and extensive configurations. However, once mastered, it offers powerful capabilities that are unmatched. Its broad adoption has led to a wealth of documentation, tutorials, and community forums that aid new users in overcoming initial hurdles. The dearth of Kafka-as-a-service is insight into the challenge of managing Kafka.

Pulsar features a modular architecture that some users might find easier to scale and maintain compared to Kafka. Its built-in features such as native geo-replication and the separation of storage and compute layers can simplify operations for teams without extensive infrastructure management resources. However, Pulsar’s documentation and learning resources, while growing, are not as extensive as Kafka’s, which can initially hinder new users.

NATS, with its focus on simplicity and high performance, is arguably the easiest to use among the three. It is designed to be lightweight and straightforward, offering quick setup times and minimal operational demands. NATS excels in scenarios where simple messaging is required without the overhead of persistent storage or complex configuration. Its simplicity is a significant advantage for projects that need to get up and running quickly and for teams with limited resources.

Ecosystem

Kafka's ecosystem is vast. With tools like Kafka Streams for stream processing, Kafka Connect for integration, and a large number of third-party tools, frameworks, and hosted versions. Kafka offers a comprehensive solution for nearly any streaming data scenario. The Kafka client API has become the standard in the streaming space with all new technologies boasting some level of compatibility. Many new entrants are offering Kafka API compatibile with different broker/storage engines like [Redpanda], [Warpstream] and more. We will get into more about the Kafka alternatives in another article.

Pulsar's ecosystem is growing, with an increasing array of tools for expanded functionality like stream processing with Pulsar functions and multi-tenancy support. However, it still trails behind Kafka in terms of third-party integrations and mature tools. Kafka-on-pulsar exists to provide compatibility with the ecosystem developed around Kafka, however the API has some differences making it hard to directly swap out the technologies.

NATS has a more focused ecosystem that excels in messaging but lacks the broad third-party support and integrated stream processing features found in Kafka. NATS has developed the jetstream capabilities that make message retention possible to bring parity with Pulsar and Kafka capabilities, but the offering is still young and lacks a lot of the capabilities of Kafka and Pulsar.

Community

Kafka, developed by LinkedIn and now managed by the Apache Software Foundation, has a large, active community. Confluent, heavily contributes to and supports the Kafka community, providing extensive resources, documentation, and enterprise solutions. Kafka is the center of gravity in for the streaming data developer community with series of conferences, meetups and developer chapters across the world. ⭐️ 27k+ 👫 1.1k+

Pulsar also benefits from strong community support and is an Apache project. It is supported commercially by Stream Native. The project continues to gain momentum in terms of contributing large organizations. There are far fewer available developer resources for people starting out. ⭐️ 13k+ 👫 600+

NATS has a smaller but very dedicated community. It is also open-source and has commercial backing from Synadia, which focuses on supporting the NATS ecosystem. NATS is a CNCF project. ⭐️ 14k+ :👫 150+

Conclusion: Why Kafka Reigns Supreme

While Pulsar and NATS offer compelling features, the explosion of the Kafka ecosystem makes it the standout choice for enterprises needing a robust, scalable, and versatile messaging platform. Innovations like Redpanda, which is a cheaper and faster, yet almost 100% compatible offering as well as the hosted of hosted soltuions, including Confluent's enterprise solutions ensure Kafka not only meets the current demands of large-scale data processing but is also poised to handle future challenges.

Whether it's through performance, compatibility, ecosystem richness, community support, or reliability, Kafka continues to lead the way in streaming data technology.

Bytewax offers a performant and resilient processing solution paired with Kafka API compatible services. It's easy to get started.

from bytewax import operators as op
from bytewax.connectors.kafka import operators as kop
from bytewax.dataflow import Dataflow

BROKERS = ["localhost:19092"]
IN_TOPICS = ["in_topic"]
OUT_TOPIC = "out_topic"

flow = Dataflow("kafka_in_out")
kinp = kop.input("inp", flow, brokers=BROKERS, topics=IN_TOPICS)
op.inspect("inspect-errors", kinp.errs)
op.inspect("inspect-oks", kinp.oks)
kop.output("out1", kinp.oks, brokers=BROKERS, topic=OUT_TOPIC)

Working with streaming data? Reach out to us on slack.

And don't forget to join our newsletter ⬇️ to get content like this delivered weekly. 🐝