Solved

Out of memory error using memory datasets with a Delphi application

Posted on 2011-03-23
3
1,418 Views
Last Modified: 2012-05-11
I have an application which reads data from a CSV files, import it temporary into a memory dataset, and then send the data into a physical database (it doesn't matter which RDBMS).

This application have worked for ten years, users loading data every day, in small chunks. Again ten years.
Data loads have increased over the time, but in all this time, it has been only like 10% to 20%.

But this week, due an internal change in the source of the CSV files (more detail was included) the size of these files doubled (100% to 150% increase) and my users started to get an "Out of memory" error in the middle of the loading process.

Note that I've tried with several dataset memory components, like the one with Rx, DevExpress, Jedi, and so on. Any of them, leads to the same error, aproximately in the same point, at the middle. So changing the component, didn't help.

I know I could rewrote the whole thing and avoid a memory dataset.
I know I could just tell the users to split the files into smaller chunks.
I know that somewhere, somehow, there are memory limitations and then having several gigabytes of RAM doesn't help at those situations.


But what I don't know is if there's a way to overcome these limitations.
A compiler directive may be?
Something to make Delphi aware of the huge amount of available memory and don't throw these stupids errors?

(This application is compiled with Delphi 7)

0
Comment
Question by:fischermx
3 Comments
 
LVL 24

Accepted Solution

by:
jimyX earned 300 total points
ID: 35201157
Delphi is giving that message based on your system limitation/configuration. I am not sure, but I doubt if there is such compiler directive.
What you can do is change the system parameters and let's see how it works:
http://support.microsoft.com/kb/126962
0
 
LVL 32

Assisted Solution

by:ewangoya
ewangoya earned 200 total points
ID: 35201186

There really is no easy way to overcome this except by rewriting your code more efficiently, no compiler directive for sure.
Its not only the memory dataset that holds the data in memory but the csv file itself is also loaded in memory

Here is what I would do
1. The first thing to do is use a memory mapped file instead of loading the whole file in memory
2. Get rid of the memory dataset, use query objects instead with transactions
0
 
LVL 1

Author Comment

by:fischermx
ID: 35202187
Thanks for your help.

I found the problem and actually was not the memory dataset fault :(
It was a supposedly tiny array of a record, that didn't use to grow a lot, but with the business logic changes to the sources CSV files caused it to grow.
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, just open a new email message. In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…

815 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now