Link to home
Start Free TrialLog in
Avatar of phadar
phadar

asked on

TClientDataSet

I need to save a table to a file. I'm using ClientDataSet1.SaveToFile procedure, the problem is when the table has many records I get a "Insufficient memory for this operation" error message.
The table has over 50000 records with a Blob field on each record. My computer has 4GB of RAM.
Am I missing something?
Avatar of BdLm
BdLm
Flag of Germany image

a)  some tutorial about TClientdataset is here :   http://delphi.about.com/od/usedbvcl/a/tclientdataset.htm

b) your code is OK with a low number of records ?

c) what about a different way to export a table to HTML or XML ?
eg.  save table as XML
{******************************************************************************
 *   Save a Table as XML, Using MSXML
 *
 *   Input:  TTable
 *           Filename :   String;
 *   OutPut:
 *
 *
 ******************************************************************************}
 
function SaveTableAsXml(table: TTable; Filename :  String): Integer;
var
 
 
 
  doc                 : IXMLDOMDocument;
  root, child, child1 : IXMLDomElement;
  text1, text2        : IXMLDOMText;
  nlist               : IXMLDOMNodelist;
 
 
 
 
  i   : Integer;
  xml,temp : String;
begin
 try
  table.close;
  table.open;
  xml  := 'WorkingTable';
  doc  := CreateOleObject('Microsoft.XMLDOM') as IXMLDomDocument;
 
 
  //Set the root name of the xml file as that of the table name.
  //In this case "country"
  root := doc.createElement(xml);
  doc.appendchild(root);
 
  //This while loop will go through the entire table to generate the xml file
  while not table.eof do begin
 
   //adds the first level children , Records
   child:= doc.createElement('Records');
   root.appendchild(child);
   for i:=0 to table.FieldCount-1 do begin
     //adds second level children
    child1 := doc.createElement
              (table.Fields[i].FieldName);
    child.appendchild(child1);
 
 
    //Check field types
    case TFieldType
    (Ord(table.Fields[i].DataType)) of
    ftString:
    begin
     if Table.Fields[i].AsString ='' then
      temp :='null'  //Put a default string
     else
      temp := table.Fields[i].AsString;
    end;
 
    ftInteger, ftWord, ftSmallint:
    begin
     if Table.Fields[i].AsInteger > 0 then
      temp := IntToStr(table.Fields[i].AsInteger)
     else
      temp := '0';
    end;
 
    ftFloat, ftCurrency, ftBCD:
    begin
     if table.Fields[i].AsFloat > 0 then
      temp := FloatToStr(table.Fields[i].AsFloat)
     else
      temp := '0';
    end;
 
    ftBoolean:
    begin
     if table.Fields[i].Value then
      temp:= 'True'
     else
      temp:= 'False';
    end;
 
    ftDate:
    begin
     if (not table.Fields[i].IsNull) or
        (Length(Trim(table.Fields[i].AsString))
        > 0) then
      temp := FormatDateTime
               ('MM/DD/YYYY',
               table.Fields[i].AsDateTime)
     else
      //put a valid default date
      temp:= '01/01/2000';
    end;
 
    ftDateTime:
    begin
     if (not table.Fields[i].IsNull) or
        (Length(Trim(table.Fields[i].AsString))
        > 0) then
      temp := FormatDateTime
              ('MM/DD/YYYY hh:nn:ss',
              table.Fields[i].AsDateTime)
     else
      //Put a valid default date and time
      temp := '01/01/2000 00:00:00';
    end;
 
    ftTime:
    begin
     if (not table.Fields[i].IsNull) or
        (Length(Trim(table.Fields[i].AsString))
        > 0) then
      temp := FormatDateTime
              ('hh:nn:ss',
              table.Fields[i].AsDateTime)
     else
       //Put a valid default time
      temp := '00:00:00';
    end;
   end;
    child1.appendChild(doc.createTextNode(temp));
  end;
  table.Next;
 end;
 
  doc.save(Filename);
 
  Result:=1;
 except
   on e:Exception do
     Result:=-1;
 end;
end;

Open in new window

Avatar of phadar
phadar

ASKER

>> your code is OK with a low number of records ?
Yes, no problem with small tables.

>> what about a different way to export a table to HTML or XML ?
Need to save Blob fields, that's the reason why.
what about storing the images in separate files:  see https://www.experts-exchange.com/questions/23499913/How-do-I-store-JPG-images-in-a-Paradox-database-for-display-in-a-Delphi-Imager.html 

there are functions like save blob image ,
Avatar of phadar

ASKER

I'm afraid storing tje blob fields into separate files is not an option, i have a table with over 60000 records where each records has 2 binary field (Blob).
a) did you check you mem consumpition with the task manger, really all the memory is gone ?

b) use a tool form mem. leaks  http://v.mahon.free.fr/pro/freeware/memcheck/
Avatar of Geert G
why do you need 60000 records in memory ?
the user can't possible want to edit 60000 records ???
why not just keep the displayed records in memory --> 20 or so * 2 = 40 blobs

you may have to work on some routine to only load the desired records that must be displayed
Avatar of phadar

ASKER

I never said I need the records in memory...
Read te question: "I need to save a table to a file."
the german delphi help says these data are keept in the main memory, of couse I don't know the implemention of this component

Beschreibung

Das Objekt TClientDataSet repräsentiert eine Datenmenge im Arbeitsspeicher. Sie kann folgendermaßen eingesetzt werden:
where does the data come from ?
and how ?
Avatar of phadar

ASKER

TADOConnection
         ||
TADOQuery
         ||
TDataSetProvider
         ||
TClientDataSet
Uhm maybe check out these ?

ADOQuery.SaveToFile();

ADOQuery.LoadFromFile();

Avatar of phadar

ASKER

Geert_Gruwez,
Are you bored?
there are two very good exception debugger tools:

a)  http://www.eurekalog.com/docs/

b) http://www.madshi.net/olddlp0.htm

even you know it is an out of memory issue you may get a hint how to do a workaround

c) check phys / virt. mem of your PC, try to increase   .-). may be some other stuff runs out of bounds.

d)  Maunal saving record by record using functions from above
Why ?
Solution too simple ?
may be query save to file just saves the sql string?
Avatar of phadar

ASKER

BdLm,

I think the problem has to do with the TClientDataSet SaveTofile procedure.
Seems that procedure cannot handle large tables with binary fields. As mentioned before, I try this procedure on a machine equiped with 4GB RAM which is more than any standard memory.
Did you try to achieve that with a 60000 records table with a blob field on each record?

Thanks
Avatar of phadar

ASKER

Dear Geert_Gruwez,
No... it's not because your solution is too simple, it's not a solution (:
If you read my question, you'll see that I try to save a table using the TClientDataSet SaveTofile procedure but get a memory error on large tables.
what is stored in the blobs ?
text ? images ?
my practical experience:
------------------------------

I did not try to put all the images to a blob, and store inside the DB file,
 I perfer using the directory for keeping the images and just store the path inside the DB.
I'm just dealing with ~ 5k ...10k images.
Avatar of phadar

ASKER

>> what is stored in the blobs ?
Stream
a stream of what ?
be more specific what the data is ...
with certain types of data there is a workaround
loop through all the records / fields and use a saveto file function as below ?
https://www.experts-exchange.com/questions/23333986/save-and-load-msxml-to-Blob.html 
procedure SaveXMLDocToTable(aTable: TTable; aXMLField : TBlobField; aXMLdoc : IXMLDomDocument);
var ss: TStream;
begin
  with aTable do
  begin
    Edit;
    ss := aTable.CreateBlobStream(aXMLField, bmWrite);
    try
      aXMLdoc.SaveToStream(ss);
    finally
      FreeAndNil(ss);
    end;
    Post;
  end;
end;
 
{***********************************************
 *
 *  reverse function of above
 ***********************************************}
 
procedure SaveTableToXMLDoc(aTable: TTable; aXMLField : TBlobField; var aXMLdoc : IXMLDomDocument);
var ss: TStream;
begin
  with aTable do
  begin
    ss := aTable.CreateBlobStream(aXMLField, bmRead);
    try
      ss.Seek(0, soFromBeginning);
      aXMLdoc.loadXML(ss.ReadString(ss.Size));
    finally
      ss.free;
    end;
  end;
end;

Open in new window

if you have windows XP you are only using 2.xxx GB
i have been trying this from several approaches.
Basically i allways come to this conclusion:
--> doesn't work with TClientDataset for large datasets



for Briefcase model ->
if the database is SQLServer of Oracle (or other) you could install a local database server
and import the data from the server database instance to the local running database instance
and let the database engine handle the problem
Have you tried the VirtualTable from DevArt instead of the TClientDataset ?
it's free

ClientDataset obviously can't handle very much data
what ever data in the TClientDataSet is residing in the local memory and its just a disconnected architecture. The table has over 50000 records with a Blob field on each record will eat your memory.
ClientDataSet1.SaveToFile will also take its own memory to do its task - write data to an external file.
A combination of these will make your memory full  and throw 'Insufficient memory for this operation'

I am sure that you will not get this error when you avoid blob field.

One approach is get chunk or records and do your work, then get the next chunk of records and so on.
Also increase your virtual memory in your system.
Mycomputer > properties > Advanced > Performance settings > Virtual memory Change > Custom size > 
 
Note that there should a proportion in Initial and Max size

Hope this helps





Posted a new comment, may be it will help to solve the issue
Are you saving the file in XML or CDS (binary) format?

I think you will need a hybrid solution or will have to roll your own solution.  First, you need to determine the output format.  If you require CDS (binary) format, you are screwed because the details of the format are not published.  Assuming you are outputting in XML format, you should do the following:
1. Reduce the number of records in your table to 1-3 range.
2. SaveToFile in XML format
3. Look at the XML file.

The XML format is MyBase.  There are metadata tags at the start of the file followed by actual rowdata tags.

The trick is to find the storage/representation format of the BLOB fields.  I suspect each byte will be represented by three characters.  This increase might account for the huge increase in memory.

Likely solutions:
A. Output your 60k row table in 5k-10k row chunks and then programmatically append the RowData tags from the second-through-sixth/twelveth chunks.  Don't forget you will need to move the ending tags from the initial output chunk to the end of the last appended chunk.

B. Consider outputting your own XML file in the MyBase format.  Once you know how your BLOB fields are encoded, you should be able to do this yourself.

C. Output the 60k rows of a table with empty BLOB fields and then open the XML file with a TXMLDocument interface and add the BLOB information to each record.

D. Copy the TClientDataset records to another DataSet-interface component (that doesn't suffer the memory ceiling) and have it save/persist the data.  A good start is kbmMemTable, although it is not the only alternative.

==================
I don't know if it would work, but you might want to try outputing a few records with the BLOB fields surrounded by the <! [CDATA[    ]]> tag.  If a TClientData will read it correctly, .LoadFromFile, then you won't have to replicate the BLOB encoding scheme.
Avatar of phadar

ASKER

I found out that the all stuff hangs as soon as I make my ClientDataSet active, way before the save step. So I guess the problem is not about saving after all but holding over 50000 records.
How are you populating the TClientDataset?  There is a known problem if you are trying to populate with Insert or Append methods.
https://www.experts-exchange.com/questions/21393852/ClientDataset-performance-problem-workaround-alternative-s-needed.html

What controls are bound to it?
@phadar

You still haven't isolated the point(s) where you might be having a performance problem with your TClientDataset.  Even though you are still getting "insufficient memory", this discussion is still a valid question for determining the cause of the error.  We now know that the error occurs prior to the .SaveToFile method invocation.
Avatar of phadar

ASKER

Keep this question running for now.
@phadar

Start by answering some of the questions I've posed in my earlier comment.
Avatar of phadar

ASKER

aikimark,
I'm not populating with insert or append.

  ADOQuery1.Sql.Clear;
  ADOQuery1.Sql.Add('Select * From Examtbl');
  ADOQuery1.Open;
  ClientDataSet1.Active := True;   ===> he it hangs
  ClientDataSet1.SaveToFile(ExtractFilePath(ParamStr(0)) + 'Exam.Stm');
  ADOQuery1.Close;
What happens if you only retrieve 10, 100, 500, 1000 records instead of all records?

What version of Delphi are you using?

What do you have bound to the TClientDataset?
Avatar of phadar

ASKER

1. Runs ok

2. Using D7

3.
TADOConnection
         ||
TADOQuery
         ||
TDataSetProvider
         ||
TClientDataSet
@phadar

Please continue to increase the number of records, starting at 5000 and incrementing by 5000 until you reach the insufficient memory condition.  Also, please note any increases in execution times on a per-record basis (elapsedtime/NumberOfRecords).  If you start your Windows Task Manager before these tests, you should be able to track the memory growth.

Are there any bound controls on the TClientDataset, such as a grid or edit control?

Another thought is that somewhere near the end of the source dataset is/are record(s) with VERY LARGE BLOB fields (much larger than average of earlier records).  Are all the BLOB fields roughly the same size?

Have you tried a TADODataset?  Did it also have the same problem?
I object.  phadar has failed to answer questions posed during this thread and requests for (run-time) experimental timing data.
it doesn't work with a TClientDataset.  The TClientDataset has a bug for lots of records

you should do it without the TClientDataset
TADOConnection
         ||
TADOQuery
         ||
Save the records to file here and you probably will have to use multiple steps to do this
a chuck at a time like from 1..10000 and 10001 to 20000 and so on

what you do know example
TADOQuery -> memory = 2GB
using TClientDataset just copies the whole set so +2GB

but why do you want to use 2 datasets (TADOQuery and TClientDataset) ?

if you are writing the records to files then why copy them first to a TClientDataset ?

It's like going from Paris to London and taking a (short) detour to New York


Avatar of phadar

ASKER

@Geert_Gruwez
You are right and I already figure that out!. That's the reason I decided to delete this question because I had to change the all approach.
At the time I posted this question, I was not familiar at all with TClientDataSet. After studying this component I understood that it's like you wrote (short detour).
But I posted on July 14th that schema, so where were you guys? Why didn't I get that explanation a month and a half ago (:
From the start i was wondering WHY you were taking the detour.
I created a test application and allways came to the conclusion that the dataset loads all the records in memory until it hangs (or is out of memory) ...  I even did the test with only 1 TADOQuery (same problem)

This test only took me 30 min
conclusion: ID:22005006This I pointed out and some people commented on this too

Now, how did you solve it ?
Chunks ?



Avatar of phadar

ASKER

I don't know if chunks is the right definition. I load the records to TADOQuery and add each record to 2 files, one containing all the non-blob records and the other one with the blob records.
Be glad if you have a better/faster method.
nope :( sorry
Avatar of phadar

ASKER

That's ok. Thanks fro trying (:
in the end you did follow the experts advise:
BdLm: save the dataset to separate files (2, 1 for data and 1 for blob)
we all pointed out that memory is eaten by the blobs and the TClientdataset stores all in memory
and that you would better save a set of records at a time

if you have a completely different solution to this, then you can close it.
Otherwise you should accept the solutions
Avatar of phadar

ASKER

Far away from that.

BdLm advised to export to Xml file - ID:21991925.
After rejecting that advice, Bdlm advised to store each blob in different files - ID:21991955. That was rejected too.

I ended up by storing each row on a single file (not xml) where that file point to the blob objects ( second file) position and size. Total of 2 files only and not one xml with 60000 blob files.

I don't se any sugestions posted that are even close to that solution.
I beleive we ehausted this question to the max.
you aren't using the TClientDataset any more ?
Avatar of phadar

ASKER

No I'm not!!!!! read ID:22375355
well then you are following my suggestion as to not use a TClientDataset
but you had to ask Are You Bored ?

your solution is a mix of things suggested here and maybe with some modifications.
and saving to xml or any other extension is still saving to a file.

it looks like you are deleting half of the questions you pose. why ?
you could simply accept a post by yourself.
the more you delete questions the more objections you will get ...


what i'm trying to show you:
there is no point in deleting this question.

Post your solution to your problem.  Accept it.
And others will see the struggle, the problem and the solution.

Deleting just solves nothing.
@phadar

"...one containing all the non-blob records and the other one with the blob records..."

Did you mean to write:
...one containing all the non-blob fields and the other one with the blob fields...?

========================
I would object to this discussion being deleted.  It contains valuable information in the comments.  

As an acceptable closing, I would advise that you give some recognition to the experts for their suggestions, even though you arrived at the solution indirectly (grades 'B' or 'C').  If you post your code, you may choose to accept that as the answer and request your points be refunded.
Avatar of phadar

ASKER

@aikimark
Yes

Well... I didn't think that my solution would be so valuable, that is the reason I decided to delete this question.
I'll post the entire solution in the next days so everyone can see it.
@phadar

In our comments, we suggested several problems with TClientDataset, work-around solutions (including substituting TADODataset), and your custom solution.  Future readers will benefit from this.  

Your TClientDataset pain may result in future developers' gain.  I'm looking forward to seeing your code snippet.  It may be a class waiting to be written.

=======================
I do have one additional question...What happens when you populate a TClientDataset with the rowID field and the two BLOB fields and then .SaveToFile()?  It was another approach that I didn't have time to mention, but thought you might have tried along the way.  You have been trying a lot of different things to arrive at your solution.  Like Thomas Edison, you now know 100 ways NOT to populate a TClientDataset with BLOB fields.
:-)
Avatar of phadar

ASKER

@aikimark
I though of that - including just the 2 blob fields but I understood that I will have a different issue, so I didn't even try.

I will comment that issue in my codes. Basically, I could save the blobs one after another in the mem file (file 2) without having a record for as position and length of each blob within the main file (file 1).
Now why would I need that? Simply because I want to be able to load back only a part of the records. If I don't keep a track of each blob position, I'll end up with a mixed record/blob meaning that the blob of record 10 for example will be loaded into record 5 etc..
@phadar

Populating two separate TClientDatasets (one BLOB, one BLOB-less) might give you quick access to all the data, whether linked or retrieved as necessary at run time.

Your custom solution does provide more flexibility to retrieve a limited subset of the BLOB data.  The trade-off is ease of SaveToFile/LoadFromFile versus your own I/O routines and record position tracking.
ASKER CERTIFIED SOLUTION
Avatar of phadar
phadar

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
@phadar

OMG!  That's an awesome post (IMHO).  Thanks.
Avatar of phadar

ASKER

Your ara welcome. I forgot to mention, I added 2 other tiny files to the all deal, one for the table structure and the other one for indexes structure.