Comet channel best practices and HA

Hi Experts,

We're planning on doing a comet channel to a server that potentially runs beside other servers in a high-availability environment.  If I have a global queue of requests that I want posted to a given channel, what mechanism do people use to replicate this among the other servers in the HA group?  I can imagine using a database is one idea - but my point of using a comet channel is to get quick responses to simple events happening without the need to repeatedly poll a database.  That leaves message passing between servers as the only other viable solution.

An example is - I just posted a request to a server and somehow I get reassigned to another server.  I want that other server to know that I want certain types of events - without the client having to re-request them.

Many thanks,
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Shalom CarmelCTOCommented:
You have the right idea, but not the best implementation.
What you should use is a queue implementation using a real queue, not a database simulation of a queue. Rabbitmq, Kafka, even IBM WMQ if you don't mind paying the fees.

Another viable variation is to use a key-value store like redis.
You will queue only the keys, and write the full message to redis to be processed by your HA cluster. This technique should be used if there is a large processing time associated with each request, and you must verify that all messages are processed regardless of the server status. It is a write-once, read-once with recovery scenario.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
threadyAuthor Commented:
Interesting shalomc, thank you very much I'll read up on all that!
threadyAuthor Commented:
So sorry for coming back to this so late - I finally had the opportunity to read a little about about your answer and I can see that I have a lot to learn here.  Would you mind recommending me a good book if you've been blown away by anything related to system design or something like this?

Awesome answer, thanks a lot for your help!
The 7 Worst Nightmares of a Sysadmin

Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.

threadyAuthor Commented:
Wow - these systems are going to be a hard sell.  It does seem like the only right way to do this, at least if you're including HA in there, is to use a real queue like you're saying.  I just wish there were something that didn't require a whole new component for.
threadyAuthor Commented:
Apache Kafka is very impressive.
Shalom CarmelCTOCommented:
Can you tell a bit about your application and environment?
Apache Kafka is great, but it is open source in incubator stage (no support), and has issues with persistence.  Rabbitmq is much more mature.
I would recommend the following:

Need for speed or Startup --> Kafka
Reliability matters or Small business --> Rabbitmq
Enterprise --> Websphere MQ

A word about WMQ is in order.
DO NOT GO FOR WEBSPHERE MESSAGE BROKER. Unless you really really have to.

WMQ aka MQseries is the underlying tech below message broker. It has been around for more than 15 years, and is the most mature and reliable queue system around. It is trusted by a lot of financial and health institutions for assured delivery of messages, in a most impressive array of deployment options, and has features Rabbit hasn't dreamt of yet. Of course, it comes with a price, so it is a good fit for risk averse enterprises.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.