STREAMING DATA AND MESSAGE QUEUING IN THE ENTERPRISE

When an information architect needs to move data around, how should it be done? Recently, there has been a dramatic change in the thinking about this subject. Driven by an increased need for real-time integration of high-velocity data, streaming solutions are increasingly being considered where ETL used to dominate.

A Gap in Features

Enterprise application integration (EAI) and enterprise service buses (ESB) were the initial responses, but they lacked the scale required for many modern integration challenges. Today’s streaming solutions have solved the scaling problems associated with these early attempts at real-time data integration.

In the current landscape of streaming and message-queuing technology, a gap has emerged between message queuing capabilities and scale. Either a platform is more streaming data oriented (such as Kafka and Amazon Kinesis) or more message-queuing oriented (such as RabbitMQ, Apache ActiveMQ, Artemis, and Google Cloud Pub/Sub).

If a company goes with a solution built for streaming data, it may have to give up capabilities that are more message-queuing oriented, such as consumer and producer queue definition, conditional message routing, batch fetch and delivery, broker push, and message rejection and resending.

For the rest of the article, please see link.