Mike R.

<div>
<div class="content wysiwyg-content">
We're designing a microservice infrastructure, using (currently) RabbitMQ as the message passing service - although Kafka is also an option. <br />
<br />
In traditional REST/MQ microservice (MS) environments, the data being passed between MSs is generally small (&lt;1MB.) So the response message can carry the actual data requested. <br />
<br />
In our case, the query will request a data search from a database service, which could result in large amounts of data needing to be returned. <br />
<br />
Our current idea is to have the message queue deliver the search criteria via message queue, and the database service populate a temporary database (perhaps redis) with the data - and simply return the location of the redis DB in the response message back to the requester.<br />
<br />
<a href="https://filedb.experts-exchange.com/incoming/2020/04_w15/1449420/Annotation-2020-04-09-094819.png" target="_blank" title="Annotation-2020-04-09-094819.png (59 KB)"><img alt="Annotation-2020-04-09-094819.png" class="file-block lazyload" style="max-width: 800px;" data-src="https://filedb.experts-exchange.com/incoming/2020/04_w15/800_1449420/Annotation-2020-04-09-094819.png" /></a><br />
<b>My question is; has anyone done a similar microservices structure and/or is there a better way to do this?</b>
</div>
</div>


Annotation-2020-04-09-094819.png

We're designing a microservice infrastructure, using (currently) RabbitMQ as the message passing service - although Kafka is also an option. 

In traditional REST/MQ microservice (MS) environments, the data being passed between MSs is generally small (<1MB.) So the response message can carry the actual data requested. 

In our case, the query will request a data search from a database service, which could result in large amounts of data needing to be returned. 

Our current idea is to have the message queue deliver the search criteria via message queue, and the database service populate a temporary database (perhaps redis) with the data - and simply return the location of the redis DB in the response message back to the requester.



My question is; has anyone done a similar microservices structure and/or is there a better way to do this?

Handling big data with microservices and message queues

Big data describes data sets that are so large or complex that traditional data processing applications are inadequate. Challenges include analysis, capture, data curation, search, sharing, storage, transfer, visualization, querying and information privacy. The term often refers simply to the use of predictive analytics or certain other advanced methods to extract value from data, and seldom to a particular size of data set.

Big Data

Databases are organized collections of data, most commonly accessed through management systems including schemas, tables, queries and processes that allow users to enter and manipulate the information or utilize it in other fashions, such as with web applications or for reporting purposes.

Databases

Programming includes both the specifics of the language you’re using, like Visual Basic, .NET, Java and others, but also the best practices in user experience and interfaces and the management of projects, version control and development. Other programming topics are related to web and cloud development and system and hardware programming.