?
Solved

I got out of memory when processing a Linq query

Posted on 2011-02-25
5
Medium Priority
?
1,159 Views
Last Modified: 2013-11-10
Hi,

  The following code is leading the system to out of memory. The database is FireBird. Let me explain what happens:

  - srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()) goes to the database, by Linq, and reads about 500,000 records
  - foreach gets each record and saves it to another table
  - OLAP_VENDAS olapv = new OLAP_VENDAS(); creates a new object in the Linq entity.
  - entity.AddObject(olapv.GetType().Name, olapv); insert the entity in the object
  - entity.SaveChanges(); it saves the data each 500 read reacords

  The procedure works up to 100,000 records processed. After this I get the out of memory message. My machine has 6Gb of RAM and it starts with 2.6Gb allocated RAM and raises up 6Gb very quickly.

  Is is not possible to dispose the OLAP_VENDAS object. How can I keep the RAM short?

  Thanks,
  Marco Castro
foreach (var item in srv.DadosCalculados(srv.Consulta(_fromData, _toData).ToList()))
{
 OLAP_VENDAS olapv = new OLAP_VENDAS();
 olapv.VENDAS = new Misc().GenID("OLAP_VENDAS");
 olapv.VENDAS_CABECALHO = olap.VENDAS_CABECALHO;
 olapv.ANO = item.Ano;
 olapv.CLIENTE = item.ClienteID;
 olapv.EDITORA = item.MarcaID;
 olapv.LIVRO = item.ProdutoID;
 olapv.QT_ACERTO = item.QuantidadeAcertoConsig;
 olapv.QT_CONSIG = item.EnvioConsignacao;
 olapv.QT_DEVOL = item.DevolucaoConsig;
 olapv.QT_VENDA = item.QuantidadeVenda;
 olapv.QUADRIMESTRE = item.Quadrimestre;
 olapv.MES = item.Mes;
 olapv.VALOR_SAIDA = item.ValorSaida;
 olapv.VALOR_ENTRADA = item.ValorEntrada;
 try
 {
  entity.AddObject(olapv.GetType().Name, olapv);
  if (i++ > 500)
  {
   entity.SaveChanges();
   i = 0;
  }
 }
 catch (Exception e)
 {
  throw new FaultException<FaultContract>(new FaultContract(e.ToString()));
 }
}

Open in new window

0
Comment
Question by:MarcoCastro
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 3

Expert Comment

by:sergiobg57
ID: 34979563
Can't you subdivide/page the srv.consulta query?
Use the limit keyword in order to maintain a shorter list and go all the way forward till the end.

select c from t where c1 = c2  limit HOW_MANY_YOU_HAVE_READ, NUMBER_OF_RESULTS_TO_SHOW
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979722
Linq knows the folowing Queryoperators
Skip and Take

Example :
Takes the Items from 51 till 100
.Skip(50).Take(50);
The Sql query is weird, but gives only the requested Items


so then you have less Items in your select-ResultSet.
I would propose to divide your list using an 'Prosed'-Field of some kind


regards

poor beggar
0
 
LVL 10

Expert Comment

by:John Claes
ID: 34979797
Some background may help :

SQL 2000 : It uses some weird selection Query
From SQL 2005 it uses the ROW_NUMBER to return the selection

0
 

Accepted Solution

by:
MarcoCastro earned 0 total points
ID: 34991696
The solution is: clear all the created objects with this code:

for (int j = 0; j < list.Count; j++)
   entity.Detach(list[j]);

After this: entity.SaveChanges();
0
 

Author Closing Comment

by:MarcoCastro
ID: 35034602
I got the answer by myself.
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The article shows the basic steps of integrating an HTML theme template into an ASP.NET MVC project
On September 18, Experts Exchange launched the first installment of the Help Bell, a new feature for Premium Members, Team Accounts, and Qualified Experts. The Help Bell will serve as an additional tool to help teams increase question visibility.
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…

719 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question