troubleshooting Question

Handling big data with microservices and message queues

Avatar of Mike R.
Mike R. asked on
Big DataDatabasesProgramming* Software Architecture
1 Comment1 Solution96 ViewsLast Modified:
We're designing a microservice infrastructure, using (currently) RabbitMQ as the message passing service - although Kafka is also an option.

In traditional REST/MQ microservice (MS) environments, the data being passed between MSs is generally small (<1MB.) So the response message can carry the actual data requested.

In our case, the query will request a data search from a database service, which could result in large amounts of data needing to be returned.

Our current idea is to have the message queue deliver the search criteria via message queue, and the database service populate a temporary database (perhaps redis) with the data - and simply return the location of the redis DB in the response message back to the requester.

Annotation-2020-04-09-094819.png
My question is; has anyone done a similar microservices structure and/or is there a better way to do this?
ASKER CERTIFIED SOLUTION
mccarl
IT Business Systems Analyst / Software Developer

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Top Expert 2015

The Distinguished Expert awards are presented to the top veteran and rookie experts to earn the most points in the top 50 topics.

Join our community to see this answer!
Unlock 1 Answer and 1 Comment.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 1 Comment.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros