advice Question

How do you handle huge amount of data in memory?

Avatar of mmingfeilam
mmingfeilam asked on
Java* Software Architecture
11 Comments1 Solution18 ViewsLast Modified:

How do you handle huge amount of data in memory when you are writing code?  I have been asked the question twice in interviews, and have not been able to come up with a satisfactory answer.  Once it was in the form of calculating a median of this huge array, and another time in the form of searching for same word in a huge list of words.  


I tried a few approaches:

1) filter out some of the data, e.g. filter out words with different length as the search word, using maybe a Map that uses key as length, and value as list of words that fits the length

2) add more memory to the machine,

3) divide up the data to segments and process the search using multiple threads, processes, or even machines and then combine the results

4) using DB that can handle large amount of data (may not be what they want since this is a coding interview, and they are probably looking for the right data structure)

5) Big data platform.

6) Stream the data or progressive loading.


If you think I am missing something, please let me know.  Thanks.





ASKER CERTIFIED SOLUTION
krakatoa

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 11 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 11 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros