We help IT Professionals succeed at work.

What can RabbitMQ/ActiveMQ do that Kafka CAN'T?

I realize this is a strange way to post this question. We are in the process of deciding on message passing tools for a new Microservices (MS) infrastructure. Initially, the majority of microservices will just be accessing data from databases and simply passing those results to the requester. Eventually, this data will be passed back and forth amongst microservices for more processing. The data may get quite large.

I've been looking at traditional message queues like RabbitMQ and ActiveMQ - which started as transactional, producer-pushed, point-to-point queues ... and have since branched out into pub-sub, streaming, and topics with multiple consumers.

Apache Kafka seemed to come from the other direction starting with streaming, non-point-to-point topics, and then adding point-to-point, and guaranteed one-delivery-only options.

At the start of our project, a traditional, very simple ActiveMQ/RabbitMQ message queue (where one producer sends one message to one consumer) will work fine. It is yet undecided where the use cases will expand to in the future.

To me, it seems that Apache Kafka can do everything that ActiveMQ/RabbitMQ can do. The biggest difference being that the *MQs are producer pushed (producer controlled) whereas Kafka is consumer controlled.

So, based on the above ...

What can ActiveMQ/RabbitMQ do that Kafka CAN'T do?

What benefits would I get from RabbitMQ/ActiveMQ that I CAN'T get from Kafka?
Comment
Watch Question

nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

Kafka 0.9 was Producer, Consumer... Kafka 0.10 started with streams.... so not sure if you did check the kafka site..

there are more MQ's... the reason there are many s several software builders had their own version.  with all working on the same characteristic.

And with considerable installed base they all have their place.


I think you have to look at what you know possibly do performance test to 

AMQ, RabbitMQ also do MMQT didnt see that one with Kafka  so there may be differences in support fo endpoints.

You have to decide what you need... and make a shortlist of topics, then products that support them (performance test?) and then choose.


I am not sure what you mean by Consumer Controlled/ Producer controlled.   Without producers consumers get nothing. Same the otehr way around if there are no consumers sooner or later the queues will overflow.


Author

Commented:
@Noci

Thanks for the response.

Consumer Controlled/ Producer controlled.
Basically, I mean that Kafka uses a "consumer pull" method, where the consumer is responsible for determining if all messages have successfully been received and Kafka does not track that - whereas with traditional MQs (like Rabbit and Active) the producer (I.e. the rabbitMQ and activeMQ infrastructure) pushes the message to consumers, and tracks whether messages are delivered.
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

AMQ, RabbitMQ, IBM's MQ, DEC/MQ etc.  all have two message classes persistent & non-persistent messages.

Non-persistent can be lost..., persistent message once accepted by the local broker  will be eventually be delivered if the intermediate & final brokers & apps are running.  Guaranteed delivery. If needed message are stored on disk. by a broker.  Producers just Shove the messages to their local broker until the buffers get filled.


How should consumers know whether a message has been sent or not... 


The difference is this: Kafka will delete message after some time if not picked up. So it cannot do Guaranteed delivery 

Durability — Kafka does not track which messages were read by each consumer. Kafka keeps all messages for a finite amount of time, and it is consumers' responsibility to track their location per topic, i.e. offsets.

I didn't notice this on the kafka site, this is on someone elses site: https://jaceklaskowski.gitbooks.io/apache-kafka/kafka-overview.html

Author

Commented:
@Noci

Durability — Kafka does not track which messages were read by each consumer. Kafka keeps all messages for a finite amount of time, and it is consumers' responsibility to track their location per topic, i.e. offsets.

I had noticed this too. I couldn't decide if this was a negative or a positive. The Kafka documentation discusses it as a benefit, where consumers are in control of the queue and (since messages can be held for an infinite-or-less amount of time) they will never "miss" a message should they go down. This is part of Kafka's mindset of "dumb pipes and smart consumers."

Others see this as an issue. Rabbit and Active are both "smart pipes with dumb consumers" - meaning the pipes track the messages, so the consumers just consume. BUT, if a consumer fails too many times, or the MQ service fails, lots of messages can be lost.

Do you have any opinion on this idea? I really don't know which was to fall :)
Software Engineer
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:

If you have items with deadlines. which can be discarded if they are not met.... then kafka will suffice, and skip some work that may not need to be done... 

In a banking application you don't want to miss any transaction. Here Kafka may seriously harm business. Or one needs to be very careful with deadlines... (it might be an attack vector by a rogue sysmanager modifying some queues). 


*MQ service will not loose persistent messages it will not accept them from a producer either when pipelines fill-up.

I never worked with systems where either the Producer or the consumer needed to track messages.

Producers needed to fire & be able to forget... The consumers were sitting ducks handling what they could ...  both had no intelligence on routing or each others availability or needed to.