Avatar of Mike R.
Mike R.

asked on 

Handling big data with microservices and message queues

We're designing a microservice infrastructure, using (currently) RabbitMQ as the message passing service - although Kafka is also an option.

In traditional REST/MQ microservice (MS) environments, the data being passed between MSs is generally small (<1MB.) So the response message can carry the actual data requested.

In our case, the query will request a data search from a database service, which could result in large amounts of data needing to be returned.

Our current idea is to have the message queue deliver the search criteria via message queue, and the database service populate a temporary database (perhaps redis) with the data - and simply return the location of the redis DB in the response message back to the requester.

My question is; has anyone done a similar microservices structure and/or is there a better way to do this?
Big DataDatabasesProgramming* Software Architecture

Avatar of undefined
Last Comment

8/22/2022 - Mon