Solved

I got out of memory when processing a Linq query

Posted on 2011-02-25
5
1,127 Views
Last Modified: 2013-11-10
Hi,

  The following code is leading the system to out of memory. The database is FireBird. Let me explain what happens:

  - srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()) goes to the database, by Linq, and reads about 500,000 records
  - foreach gets each record and saves it to another table
  - OLAP_VENDAS olapv = new OLAP_VENDAS(); creates a new object in the Linq entity.
  - entity.AddObject(olapv.GetType().Name, olapv); insert the entity in the object
  - entity.SaveChanges(); it saves the data each 500 read reacords

  The procedure works up to 100,000 records processed. After this I get the out of memory message. My machine has 6Gb of RAM and it starts with 2.6Gb allocated RAM and raises up 6Gb very quickly.

  Is is not possible to dispose the OLAP_VENDAS object. How can I keep the RAM short?

  Thanks,
  Marco Castro
foreach (var item in srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()))
{
 OLAP_VENDAS olapv = new OLAP_VENDAS();
 olapv.VENDAS = new Misc().GenID("OLAP_VENDAS");
 olapv.VENDAS_CABECALHO = olap.VENDAS_CABECALHO;
 olapv.ANO = item.Ano;
 olapv.CLIENTE = item.ClienteID;
 olapv.EDITORA = item.MarcaID;
 olapv.LIVRO = item.ProdutoID;
 olapv.QT_ACERTO = item.QuantidadeAcertoConsig;
 olapv.QT_CONSIG = item.EnvioConsignacao;
 olapv.QT_DEVOL = item.DevolucaoConsig;
 olapv.QT_VENDA = item.QuantidadeVenda;
 olapv.QUADRIMESTRE = item.Quadrimestre;
 olapv.MES = item.Mes;
 olapv.VALOR_SAIDA = item.ValorSaida;
 olapv.VALOR_ENTRADA = item.ValorEntrada;
 try
 {
  entity.AddObject(olapv.GetType().Name, olapv);
  if (i++ > 500)
  {
   entity.SaveChanges();
   i = 0;
  }
 }
 catch (Exception e)
 {
  throw new FaultException<FaultContract>(new FaultContract(e.ToString()));
 }
}

Open in new window

0
Comment
Question by:MarcoCastro
  • 2
  • 2
5 Comments
 
LVL 3

Expert Comment

by:sergiobg57
ID: 34979563
Can't you subdivide/page the srv.consulta query?
Use the limit keyword in order to maintain a shorter list and go all the way forward till the end.

select c from t where c1 = c2  limit HOW_MANY_YOU_HAVE_READ, NUMBER_OF_RESULTS_TO_SHOW
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979722
Linq knows the folowing Queryoperators
Skip and Take

Example :
Takes the Items from 51 till 100
.Skip(50).Take(50);
The Sql query is weird, but gives only the requested Items


so then you have less Items in your select-ResultSet.
I would propose to divide your list using an 'Prosed'-Field of some kind


regards

poor beggar
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979797
Some background may help :

SQL 2000 : It uses some weird selection Query
From SQL 2005 it uses the ROW_NUMBER to return the selection

0
 

Accepted Solution

by:
MarcoCastro earned 0 total points
ID: 34991696
The solution is: clear all the created objects with this code:

for (int j = 0; j < list.Count; j++)
   entity.Detach(list[j]);

After this: entity.SaveChanges();
0
 

Author Closing Comment

by:MarcoCastro
ID: 35034602
I got the answer by myself.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction: When experiencing some peculiar problem with the functioning of your PC, how many times has it happened that you look for a solution and even google can’t help? It could be that you are one of the only few people on earth who ma…
As cyber crime continues to grow in both numbers and sophistication, a troubling trend of optimization has emerged over the last year.
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial

679 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question