I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but you are making things harder than they needed to be.
Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA.
If you need all those things, there just are not a lot of options.
The way people choose to use feedback on HN never fails to suprise me - we've got a generally intelligent user base here, but the most common type of feedback voting isn't because something is wrong but rather a childish "I don't like it - I want to suppress this comment".
In this case it's something different - this was an honest question, and received two useful replies, so why downvote?! The mental model of people using Kafka is useful to know - in this case the published data being more log-like than stream-like since it's retained per a TTL policy, with each "subscriber" having their own controllable read index.
I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?"
The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.
Using S3 was better, but we still hit data loss problems (on our end, not S3
's, to clear) which isn't great when you need to bill on some of that data.
Then I heard about Kafka, bought a copy of I <3 Logs, and decided that maybe Kafka was worth the complexity, and boom, it was. No more data loss, and a happier business management.
I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."
But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.
And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.
Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."
Have you found a good Head for Kafka to easily query the Topics using a SQL like language? Especially something that can infer table schema from the Schema Registry.
True, they've gone all in on Flink, which can also do what you want. And I suspect SparkSQL can do it also these days, but I haven't looked at that for ages.
I do not recommend Redis (janky implementation, subscribers drop randomly, java libraries are a crime against humanity) or RabbitMQ (memory issues), ZMQ is not a messaging queue, named pipes are not reliable and what the hell is SQS.
SQS is simple queueing service in AWS. It's ok, guarantees at least one time delivery, but I am not sure how useful it is for large volumes of messages (by this I just mean that we use it for low volume messages and I don't have experience when using larger volumes).
SQS is fantastic at exceptionally high total volumes of messages - you probably can't saturate it. But it's not great for streaming a list of ordered messages. SQS has a FIFO mode but performance will never be what you can get out of Kafka.
Also, SQS isn't pub/sub. Kafka and SQS really have very different use cases.
How have you had so many issues with Redis. We used it at a previous place it was basically bullet proof. That being said we didn’t use Java but Python. Idk.
lettuce - doesn't reconnect properly during redis restarts (1/10 chance)
jedis - subscriptions drop and stop receiving for no reason
however, my latest wrapper for jedis does seem to be holding up and haven't had too many issues, but I have a very robust checking for dropped connections.
IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case.
I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.
In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.
I'm a bit hazy on the full details because it was a few years ago, but basically it gets more complicated because you subscribe by installing an async callback which needs to ack/nak messages, needing locking, and will be called from the context of the network event loop that also needs locking. If you do any real work in the message processing callback then you'll be blocking the event loop, so the callback has to defer processing by queuing C++ lambdas capturing the context, and running those in a thread pool.
Let's not forget NATS.io which does queuing (and a whole lot of other things from at least once low latency pub/sub to request/reply, streaming and all the way to KV and Object store) very well and is very light weight to run and much simpler to run than Kafka.
> Why they'd need Kafka was always our first question, never got a good answer from a single one of them
"To follow the hype train, Bro" is often the real answer.
> If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.
Or just freaking MQTT.
MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).
It's resume-driven development. It honestly can make sense for both company and employee.
Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle.
And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack overflow solutions for common issues figured out. Vs picking some niche thing that you are now 1 of like six total people in the entire world using in prod.
Obviously the dose makes the poison and don't use kafka for your small internal app thing and don't use k8s where docker will do, but also, probably use k8s if you need more than docker instead of using some weird other thing nobody will know about.
That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.
Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.
We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.
So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?
I use MQTT daily. I'm not sure why the commenter suggested it; it is orthogonal to queueing or log streams.
MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quickly as possible. ( I know this from experience - MQTT brokers get bogged down rapidly if there are too many messages "in flight")
A very common pattern is to use MQTT for communications, and then Kafka or RabbitMq for large scale queuing of those messages for downstream applications.
I presume one will want to use https://github.com/eclipse-mosquitto/mosquitto if going that route because I seem to recall the "mainstream" MQTT project doing a rugpull but since I'm not deeply in that community, I don't have substantiating links handy
MQTT and Kafka solve different problems. At my current company, we use both.
Kafka isn’t a queue. It’s overkill to use it as one.
Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.
MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.
Still not what I would call 'proper' queuing (IMHO!) because ultimately it's still just Kafka and just append only: you can't 'atomically acknowledge+delete' individual messages anywhere in the stream, you only dealing with offsets and that's a fundamental thing that has many consequences (e.g. you don't get ordering between batches) and also means that you will get re-deliveries in some scenarios that you would not have with a system that implements proper queuing.
I'm really unsure where the drive for this is coming from tbh. (cough CFLT share price since IPO, big enterprise customers cough) If this was so desirable, everyone would've jumped ship to Pulsar already.
