Solved

I got out of memory when processing a Linq query

Posted on 2011-02-25
5
1,149 Views
Last Modified: 2013-11-10
Hi,

  The following code is leading the system to out of memory. The database is FireBird. Let me explain what happens:

  - srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()) goes to the database, by Linq, and reads about 500,000 records
  - foreach gets each record and saves it to another table
  - OLAP_VENDAS olapv = new OLAP_VENDAS(); creates a new object in the Linq entity.
  - entity.AddObject(olapv.GetType().Name, olapv); insert the entity in the object
  - entity.SaveChanges(); it saves the data each 500 read reacords

  The procedure works up to 100,000 records processed. After this I get the out of memory message. My machine has 6Gb of RAM and it starts with 2.6Gb allocated RAM and raises up 6Gb very quickly.

  Is is not possible to dispose the OLAP_VENDAS object. How can I keep the RAM short?

  Thanks,
  Marco Castro
foreach (var item in srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()))
{
 OLAP_VENDAS olapv = new OLAP_VENDAS();
 olapv.VENDAS = new Misc().GenID("OLAP_VENDAS");
 olapv.VENDAS_CABECALHO = olap.VENDAS_CABECALHO;
 olapv.ANO = item.Ano;
 olapv.CLIENTE = item.ClienteID;
 olapv.EDITORA = item.MarcaID;
 olapv.LIVRO = item.ProdutoID;
 olapv.QT_ACERTO = item.QuantidadeAcertoConsig;
 olapv.QT_CONSIG = item.EnvioConsignacao;
 olapv.QT_DEVOL = item.DevolucaoConsig;
 olapv.QT_VENDA = item.QuantidadeVenda;
 olapv.QUADRIMESTRE = item.Quadrimestre;
 olapv.MES = item.Mes;
 olapv.VALOR_SAIDA = item.ValorSaida;
 olapv.VALOR_ENTRADA = item.ValorEntrada;
 try
 {
  entity.AddObject(olapv.GetType().Name, olapv);
  if (i++ > 500)
  {
   entity.SaveChanges();
   i = 0;
  }
 }
 catch (Exception e)
 {
  throw new FaultException<FaultContract>(new FaultContract(e.ToString()));
 }
}

Open in new window

0
Comment
Question by:MarcoCastro
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 3

Expert Comment

by:sergiobg57
ID: 34979563
Can't you subdivide/page the srv.consulta query?
Use the limit keyword in order to maintain a shorter list and go all the way forward till the end.

select c from t where c1 = c2  limit HOW_MANY_YOU_HAVE_READ, NUMBER_OF_RESULTS_TO_SHOW
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979722
Linq knows the folowing Queryoperators
Skip and Take

Example :
Takes the Items from 51 till 100
.Skip(50).Take(50);
The Sql query is weird, but gives only the requested Items


so then you have less Items in your select-ResultSet.
I would propose to divide your list using an 'Prosed'-Field of some kind


regards

poor beggar
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979797
Some background may help :

SQL 2000 : It uses some weird selection Query
From SQL 2005 it uses the ROW_NUMBER to return the selection

0
 

Accepted Solution

by:
MarcoCastro earned 0 total points
ID: 34991696
The solution is: clear all the created objects with this code:

for (int j = 0; j < list.Count; j++)
   entity.Detach(list[j]);

After this: entity.SaveChanges();
0
 

Author Closing Comment

by:MarcoCastro
ID: 35034602
I got the answer by myself.
0

Featured Post

[Webinar] Learn How Hackers Steal Your Credentials

Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction: When experiencing some peculiar problem with the functioning of your PC, how many times has it happened that you look for a solution and even google can’t help? It could be that you are one of the only few people on earth who ma…
Performance in games development is paramount: every microsecond counts to be able to do everything in less than 33ms (aiming at 16ms). C# foreach statement is one of the worst performance killers, and here I explain why.
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question