Design Theory Advice

Posted on 2007-08-10
Last Modified: 2013-11-05

I am looking for advice basically on users idea of how to achieve the following. Lets Say

I add 2 apis to my application.
I make use of the following two functions from each api respectively

So I have two different datasources which:
 - have over 100K results
 - have integrated paging (i.e. TotalPages, TotalRecords, CurrentPage, RecordsPerPage)

Now my question is, or rather my search for others take on how they would approach this:

I will retrieve both resultsets based on CurrentPage, RecordsPerPage on two separate threads to speed up collection.  I then need to combine these two resultsets into 1.

I know there will be tons of theories on this including design patterns, state management, unified data structures. But I am just curious as to others take on how they would approach this.


Question by:REA_ANDREW
    LVL 7

    Accepted Solution

    Here's what I would do.

    You have two sources of data for the same datatype (sales data) and you need to combine the data from both. You want to retrieve data from both sources simultaneously to speed up.

    i.e. you have two data pipes feeding into a data aggregator. The "pipes" are responsible for retrieving the data in the most optimal way (based on the details of the api, the network, the database, etc.). The data aggregator is responsible for the most efficient aggregation of this data -
    for example, should it wait for all the data to be delivered by both pipes before trying to aggregate them? That is not usually the quickest approach, but if the data is too loosely typed, or even if there is no unique key, it might be the only feasible choice.

    1. So at a high level, I would design the aggregator and pipe interfaces. I would definitely want the pipe classes to use a Factory pattern for instantiation - where there are two data sources, there could be many more. I would also want a Factory for my aggregator - it is not at all certain that one implementation of Aggregator would be optimal for all situations.

    2. As discussed earlier, I would want the the pipes to retrieve data in chunks (pages/whatever) in the most efficient  approach for retrieval, and feed the data immediately to the aggregator, and let the aggregator decide when to do the aggregation.

    So, the aggregator would implement an Observer pattern so it can get notified whenever the pipe(s) have a chunk of data available. On notification, the aggregator pulls data from the pipe (the alternative is that the entire chunk of data is sent in the notification itself, which I don't think is a great idea). So, the pipe has to be able to handle "clogging" - i.e. if the pipe retrieves data faster than the aggregator pulls it.

    3. Neither the pipes nor the aggregator should care about whether they are operating in a multi-threaded environment or not. This design should work just as well in a single threaded environment, or (with appropriate modifications) in a multi-process environment where the pipes and the aggregator communicate across a remoting interface! So, setting up the threads etc. should be outside these objects, in the initialization of the application.

    - Ravs
    LVL 20

    Author Comment

    That is great advice.  Cheers for the advice about using the Factory pattern too. Your points about that make perfect sense.



    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Looking for New Ways to Advertise?

    Engage with tech pros in our community with native advertising, as a Vendor Expert, and more.

    Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
    Whether you’re a college noob or a soon-to-be pro, these tips are sure to help you in your journey to becoming a programming ninja and stand out from the crowd.
    The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
    The viewer will learn how to pass data into a function in C++. This is one step further in using functions. Instead of only printing text onto the console, the function will be able to perform calculations with argumentents given by the user.

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    12 Experts available now in Live!

    Get 1:1 Help Now