With the demand for real-time data growing by the day, more and more organizations are adopting event-driven architectures driven by event streaming and processing technologies. In this blog post, we’ll take a look at some of the most popular stream processing platforms, analyzing their strengths and shortcomings.
Flow processing in
a nutshell
As a quick definition, stream processing is the real-time or near-real-time processing of “in motion” data. Unlike batch processing, where data is collected over time and then analyzed, stream processing allows you to query and analyze continuous data streams and react to critical events within a short period of time (usually milliseconds).
Stream processing goes hand in hand with event streaming. Now let’s briefly explain what we mean by that. For example, Apache Kafka, a popular open source pub/sub messaging solution, is primarily used to enable data flow between applications and back-end services. It has event publishing (writing) services for Kafka topics and other services that consume (read) Kafka events, all in real time. This is called event streaming.
In addition, Kafka also provides flow processing capabilities. Kafka Streams is a client library that allows you to write Java and Scala applications that can:
Consume Kafka event streams
- Analyze, join, aggregate
- them Publish the output back to Kafka
, and transform
Organizations leverage stream processing to make smarter, faster business decisions, real-time analytics and insights, act on time-sensitive and mission-critical data, and create functions delivered to end-user devices in real time. Here are some of the key use cases for stream processing:
Real-time fraud detection and
- payments
- IoT sensor data
- Real-time dashboards, e.g. medical BI dashboard
- Log, traffic and network monitoring Context-sensitive
- online advertising
- of user behavior Geofencing and vehicle tracking
- Cybersecurity
and tracking
8 Flow Processing Platforms to
Consider
When it comes to choosing a flow processing solution, there are many options available for you to choose from. Now let’s look at 8 of the most popular stream processing platforms and review their features, strengths, and shortcomings.
Apache Spark
Apache Spark
is a unified open source analytics engine designed for large-scale big data processing. The platform runs workloads 100 times faster than Hadoop and can process large volumes of complex data at high speed seamlessly.
Apache Spark is prepared with an intuitive API that makes big data processing and distributed computing so easy for developers. It supports programming languages such as Python, Java, Scala and SQL.
Spark can run independently in cluster mode. In addition, it can integrate with other cluster nodes such as Hadoop YARN, Kubernetes, Apache Mesos.
Advantages
- It is fault tolerant Supports
- multiple languages
- Supports advanced analysis
- Performance is very fast
- Easy to do batch processing
Cons
- Steep learning curve
- Consumes a lot of memory
- No built-in
caching algorithm Examples of companies using Spark are Uber, Shopify, Slack, Delivery Hero, and HubSpot.Apache Spark on G2Apache Spark on Stackshare
Apache Kafka
Streams Kafka Streams is a stream-processing Java API provided by Apache Kafka that allows developers to access filtering, joining, aggregation, and grouping without writing any code. Being a Java library, it’s easy to integrate with whatever service you’re using and turn them into sophisticated, scalable, and fault-tolerant applications. Apache Kafka Streams has a low barrier entry. Writing and deploying standard Java and Scala applications on the client side is very accessible. In addition, you don’t have to integrate any special cluster administrators to keep it running. Pros
- Integrates with existing applications Offers
- low latency value of up to 10 milliseconds
- Reduces the need for multiple integrations
- It serves as a perfect replacement for traditional message brokers.
- Cons Lacks vital messaging paradigm like
- Falls short in terms of analysis
- Tends to behave awkwardly if the number of queues in a
peer-to-peer queues
Kafka cluster increases Examples of companies using Apache Kafka Streams
are Zalando, TransferWise, Pinterest, and Groww.Apache Kafka Streams on Stackshare Apache
Flink
Apache Flink is an open source stream processing framework that is developed to compute unlimited and limited data streams. You can run stateful streaming applications at any scale and run batch and stream processing seamlessly. With Flink, you can ingest streaming data from many sources, process it, and distribute it across multiple nodes. The interface is easy to navigate and doesn’t need a steep learning curve. It also comes with built-in connectors with messaging queues and third-party databases. It can also integrate with cluster resource managers such as Hadoop YARN and Kubernetes. Flink can process millions of events per second. In addition, it can handle graph processing, machine learning, and other complex event processes. Pros
- Offers low latency, high throughput
- Simple and straightforward user interface
- Dynamically analyzes and optimizes tasks
- API and clean data flow documentation
Cons
- Integration with the Hadoop ecosystem YARN can be challenging
- Only Scala and Java
- Limited forum and community support
supported
Examples of companies using Flink are Zalando, Lime, Gympass and CRED. Apache Flink on G2Apache Flink on Stackshare
Spring Cloud Data Flow
Spring Cloud
Data Flow is a microservices-based batch processing and streaming platform. It provides developers with the unique tools needed to create data pipelines for common use cases. You can use this platform for data ingest or for ETL import/export, event streaming, and predictive analytics. Developers can adopt message-based microservices from Spring Cloud Stream and run them on-premises or in the cloud. Spring Cloud Data Flows has an intuitive graphical editor that makes creating data pipelines interactive for developers. Not only that, they can always see deployable apps using monitoring systems like Wavefront and Prometheus. Pros
- Developers can deploy using DSL, Shell, REST-APIs, and Admin-UI.
- Allows you to scale flow and batch pipelines without disrupting data flows Great
- integrations with platforms like Kafka and ElasticSearch
Cons
- Visual UI might use some improvements
- Features like monitoring tools need more development
Examples of companies using Spring Cloud Data Flow are Corelogic, Health Care Service Corporation, and Liberty Mutual.Spring Cloud Data Flow on G2
Amazon Kinesis
Amazon Kinesis Streams is a durable service that helps collect, process, and analyze streaming data in real time. It is designed to allow you to get important information that is needed to make faster decisions in time. You can ingest real-time data from event streams, social media feeds, application logs, and other applications. The streaming platform is fully managed. You can use it to build real-time applications, such as monitoring user behavior or detecting fraud. Pros
- Easy to set up and maintain
- Can handle any amount of streaming
- It integrates with Amazon’s big data toolset, such as Amazon Kinesis Data Analytics, Amazon Kinesis Data Firehose, and AWS Glue Schema Registry.
data
Cons
- Your commercial cloud service is priced per hour per fragment
- Documentation is not straightforward
- Does not offer support for direct streaming
- The Data Consuming Library (KCL) is very clunky and heavy
.
Examples of companies using Kinesis are Figma, Instacart, Deliveroo, and Lyft.
Amazon Kinesis on
G2Amazon Kinesis on Stackshare
Google Cloud Dataflow
Cloud
Dataflow is a Google-powered processing platform designed to run data processing pipelines. With the platform, you can develop simple streaming data pipelines with lower data latency. Google Cloud Dataflow has a serverless approach that shifts developers’ focus to programming instead of managing countless server clusters. It offers infinite capacity to manage your workloads. And with that, you don’t have to worry about high ownership costs. In addition, Cloud Dataflow uses the Apache Beam SDK for MapReduce operations and precision control for batch and streaming data. In the long run, it reduces complexities and makes flow analysis highly accessible to both data analysts and data engineers. The framework can be used to develop anomaly detection applications or a real-time website analytics dashboard or a pipeline that processes log entries from multiple sources. Pros
- Fully managed
- Eliminates operational complexities
- Minimize pipeline latency
- Provides access to native integrations with AI Platform, BigQuery
- Unified flow analysis and
- Real-time
data processing
AI-powered processing patterns
- Cons Restricted to cloud
- data warehouse service only BigQuery/DataFlow in streaming mode can be expensive
- Content Delivery Network does not work with Sources
- Not suitable for experimental data processing work
Examples of companies using Google Cloud Dataflow are Spotify, The New York Times, and Snowplow.
Cloud Dataflow on G2Google Cloud Dataflow on Stackshare
Apache
Pulsar Apache Pulsar
is a cloud-native distributed messaging and streaming platform. Originally deployed within Yahoo, Pulsar serves as the consolidated messaging platform that connects Yahoo Finance, Yahoo Mail, and Flick to data. Pulsar provides a high-performance solution for server-to-server messaging and geo-replication of messages between clusters. In addition, it can scale to over a million topics and expand to hundreds of nodes. It’s lightweight, easy to deploy, and doesn’t need an external flow processing engine. The processing platform has a multi-layered architecture. Each of these layers is scalable and can be distributed and decoupled from the other. Not only that, but it has granular resource management that prevents producers, consumers, and issues from overwhelming the cluster. Pros
- Easy to integrate with existing applications
- Low publishing latency with strong durability guarantees
- Supports high-level APIs for Java, Go, Python, C++, and C#
- Integrated geo-aware replication that allows data to be replicated between data centers in different geographic locations. Full end-to-end
- encryption from client to storage nodes
.
Cons
- Has a small community and forums may not be helpful
- Requires greater operational complexity
- Does not allow consumers to recognize the message of a different thread
Examples of companies using Pulsar are MercadoLibre, Verizon Media and Splunk.Apache Click on Stackshare
IBM
Streams IBM
Streams
allows you to build real-time analytical applications using Streams Processing Language (SPL) or Python. It is easy for developers to use and allows you to deploy applications that can run on IBM Cloud. Streams powers a stream analytics service that allows you to ingest and analyze millions of events per second. You can create queries to focus on specific data and create filters to refine the data in your dashboard to dig deeper. Source: IBM DevelopersPros
- End-to-end processing with sub-millisecond latency.
- A complete set of toolkits
- link to IDE to collaborate with other applications
- Allows you to use the Stream runtime using Java and Python
- Visual guided interface
Allows you to
- Cons Creating reports can be overwhelming
- Steep learning curve
IBM
Streams
at G2
Stream Processing and Ably
hope you find this article useful in your search for a stream processing engine. There are many options available, each with its advantages and disadvantages, and you should analyze them carefully before choosing the right one for your specific use case.
Ably is an enterprise-grade real-time pub/sub messaging platform. We work alongside backend stream and stream processing solutions, enabling organizations to efficiently design, quickly ship, and seamlessly scale critical real-time functionality delivered to end users.
For example, we integrated with Kafka Streams, helping Experity deliver critical data to real-time BI dashboards that enable urgent care providers to drive efficiency and improve patient care.
Not only can you use Ably to stream events from your backend stream processing engines directly to web clients, mobile and IoT in real time, but our platform can also send data in another way to your streaming service.
Ably is mathematically modeled around the four pillars of reliability, so we can ensure that events are delivered to consumers with data integrity assurances with consistently low latencies through a secure, reliable and highly available global perimeter network.
If you want to learn more about how Ably works alongside stream processing engines, allowing you to build reliable real-time applications, get in touch or sign up for a free account.
Further reading
- How to choose a stream processor for your app
- A real-world introduction to event-driven architecture
- The shift to Event-Driven: evolution, benefits and competitive advantages