Ok, here's the problem.
I have a C#.NET web application that imports several excel sheets on a daily basis. 2 are massive and have outgrown my relatively simple and fast existing import process.
First sheet contains on average about 100 thousand classes , these need to be checked against a database which now has about 1 million active and inactive classes ( which will only keep growing ) to check whether it's updating a class ( whether active or inactive ) or inserting a class.
Then, after that, there's a sheet of class days, they range between 150 and 300 rows. And for those I need to insert new days and delete days from classes that exist in the system and have a record in the new sheet, but not up to that number, example class A has 3 days in the database and this new sheet only has 2, need to delete that 3rd day. But if Class B exists in the system, but not on the sheet at all, then it's days remain in tact.
Anyway, I've had this system going for over a year ( back when the system was empty and the import sheets were less than 1/4 the size ). Then they grew, the database grew, and now the method I used to import these sheets in a timely fashion has become incredibly bad for memory use.
If I kill the aspnet_wp.exe process it clears out the memory usage, and I run the imports a few times before it starts crashing out due to memory limits.
So, anyway, what's the most memory efficient way to do this!
( My memory management skills in .net are not very high as I learned the language on the fly while doing projects freelance a few years ago and let it do its own garbage collection )
Concept suggestions are fine, don't need to suggest specific code or anything. Thanks!
Start Free Trial
