Solved

I got out of memory when processing a Linq query

Posted on 2011-02-25
5
1,112 Views
Last Modified: 2013-11-10
Hi,

  The following code is leading the system to out of memory. The database is FireBird. Let me explain what happens:

  - srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()) goes to the database, by Linq, and reads about 500,000 records
  - foreach gets each record and saves it to another table
  - OLAP_VENDAS olapv = new OLAP_VENDAS(); creates a new object in the Linq entity.
  - entity.AddObject(olapv.GetType().Name, olapv); insert the entity in the object
  - entity.SaveChanges(); it saves the data each 500 read reacords

  The procedure works up to 100,000 records processed. After this I get the out of memory message. My machine has 6Gb of RAM and it starts with 2.6Gb allocated RAM and raises up 6Gb very quickly.

  Is is not possible to dispose the OLAP_VENDAS object. How can I keep the RAM short?

  Thanks,
  Marco Castro
foreach (var item in srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()))
{
 OLAP_VENDAS olapv = new OLAP_VENDAS();
 olapv.VENDAS = new Misc().GenID("OLAP_VENDAS");
 olapv.VENDAS_CABECALHO = olap.VENDAS_CABECALHO;
 olapv.ANO = item.Ano;
 olapv.CLIENTE = item.ClienteID;
 olapv.EDITORA = item.MarcaID;
 olapv.LIVRO = item.ProdutoID;
 olapv.QT_ACERTO = item.QuantidadeAcertoConsig;
 olapv.QT_CONSIG = item.EnvioConsignacao;
 olapv.QT_DEVOL = item.DevolucaoConsig;
 olapv.QT_VENDA = item.QuantidadeVenda;
 olapv.QUADRIMESTRE = item.Quadrimestre;
 olapv.MES = item.Mes;
 olapv.VALOR_SAIDA = item.ValorSaida;
 olapv.VALOR_ENTRADA = item.ValorEntrada;
 try
 {
  entity.AddObject(olapv.GetType().Name, olapv);
  if (i++ > 500)
  {
   entity.SaveChanges();
   i = 0;
  }
 }
 catch (Exception e)
 {
  throw new FaultException<FaultContract>(new FaultContract(e.ToString()));
 }
}

Open in new window

0
Comment
Question by:MarcoCastro
  • 2
  • 2
5 Comments
 
LVL 3

Expert Comment

by:sergiobg57
ID: 34979563
Can't you subdivide/page the srv.consulta query?
Use the limit keyword in order to maintain a shorter list and go all the way forward till the end.

select c from t where c1 = c2  limit HOW_MANY_YOU_HAVE_READ, NUMBER_OF_RESULTS_TO_SHOW
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979722
Linq knows the folowing Queryoperators
Skip and Take

Example :
Takes the Items from 51 till 100
.Skip(50).Take(50);
The Sql query is weird, but gives only the requested Items


so then you have less Items in your select-ResultSet.
I would propose to divide your list using an 'Prosed'-Field of some kind


regards

poor beggar
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979797
Some background may help :

SQL 2000 : It uses some weird selection Query
From SQL 2005 it uses the ROW_NUMBER to return the selection

0
 

Accepted Solution

by:
MarcoCastro earned 0 total points
ID: 34991696
The solution is: clear all the created objects with this code:

for (int j = 0; j < list.Count; j++)
   entity.Detach(list[j]);

After this: entity.SaveChanges();
0
 

Author Closing Comment

by:MarcoCastro
ID: 35034602
I got the answer by myself.
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article is for Object-Oriented Programming (OOP) beginners. An Interface contains declarations of events, indexers, methods and/or properties. Any class which implements the Interface should provide the concrete implementation for each Inter…
Microsoft Active Directory, the widely used IT infrastructure, is known for its high risk of credential theft. The best way to test your Active Directory’s vulnerabilities to pass-the-ticket, pass-the-hash, privilege escalation, and malware attacks …
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question