Solved

Improve nested for each performance

Posted on 2013-11-13
1
222 Views
Last Modified: 2013-11-14
Datatable  dt with three columns
ID
BusDate
Return


listReturns(Type string,  Value int)
Daily 1
Weekly 7
Bi Weekly 14
Monthly 30


For very row in dt I need to calculate the daily weekly, bi wekly and monthly returns by finding the Return i1 day ago, 7 days ago or a month ago.
Then divide the value with that of the given date.

For every ID I need to loop four times for each return type.
The nested for each takes an hour for 71000 Rows in dt. I am asuming my linq query to get the tri value is taking some time



                  foreach (DataRow item in dt.Rows)
            {
                                 
                    foreach (var mt in listReturns)
                              {
                                  decimal tri = (from r in dtHistoricalPrices.AsEnumerable()
                                   where r["ID"].ToString() == item["ID"].ToString()
                                   && Convert.ToDateTime(r["BusDate"]) == Convert.ToDateTime(item["BusDate"]).AddDays(-mt.Value)
                                 
                                   && r["Return"] != DBNull.Value && r["Return"].ToSting() != string.Empty
                                   select Convert.ToDecimal(r["Return"]).FirstOrDefault();
                                                   
                                    decimal dreturn =       (Convert.ToDecimal(item["Return"]) / tri) - 1;         
                                  DataRow dr = dt2.NewRow();
                        dr["ID"] = item["ID"];
                        dr["BusDate"] = item["BusDate"];
                                    dr["ReturnType"] = mt.Type;
                                    dr["ReturnValue"] = dreturn
                              }
                  }
0
Comment
Question by:countrymeister
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 10

Accepted Solution

by:
joriszwaenepoel earned 500 total points
ID: 39647259
Are you sure the Convert.ToDateTime(r["BusDate"])  is really necessary?  That needs to be executed again and again, for every row in dtHistoricalPrices, for every value in listReturns, for every row in dt.

Datasets and datatables are easy to work with, but they have a lot of overhead.  In a lot of cases you can improve the performance drastically if you avoid using datasets.  Try getting the data from the database in a collection of objects, with correctly typed properties for each column.  

You could start with the dtHistoricalPrices.  Create a HistoricalPrice class, Make a collection with an object of that type for every row in dtHistoricalPrices, and then do your LINQ query on that collection.  The performance will probably be a lot better.
If necessary, you can do the same thing with the dt datatable.

To start easy, you can convert the data from the datatable to the custom objects, but I would consider using a DataReader to get the data from the databse and create that collection of objects, and avoid the use of datasets and datatables.

Hope this helps.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

We all know that functional code is the leg that any good program stands on when it comes right down to it, however, if your program lacks a good user interface your product may not have the appeal needed to keep your customers happy. This issue can…
For those of you who don't follow the news, or just happen to live under rocks, Microsoft Research released a beta SDK (http://www.microsoft.com/en-us/download/details.aspx?id=27876) for the Xbox 360 Kinect. If you don't know what a Kinect is (http:…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…
Attackers love to prey on accounts that have privileges. Reducing privileged accounts and protecting privileged accounts therefore is paramount. Users, groups, and service accounts need to be protected to help protect the entire Active Directory …

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question