asked on

EStreamError Out of Memory while expanding memory stream

I get an EStreamError when i use TMemory stream
and when I use TFileStream it is to slow
Is there an better way to write some info to memory or file but vary fast
I want to be able to write ,read and insert to stream

I add a program to test TMemoryStream and TFileStream how long it will take in milliseconds.

does any one how to work with the api functions CreateFile, CreateFileMapping & MapViewOfFile
want to know how to write, read and importent to insert into

{ On a clean form put a TLabel and two TButons }

procedure TForm1.Button1Click(Sender: TObject);
var TMStart,TMEnd : TDateTime;
TmeDiv : TDateTime;
a : Shortint;
b : Smallint;
c : Longint;
Sn : Longint;
ClDFL1 : TMemoryStream;
ClDFL2 : TMemoryStream;
Cntfr : Longint;
begin
TMStart:= now;
ClDFL1 := TMemoryStream.Create;
ClDFL2 := TMemoryStream.Create;
ClDFL1.Write(a,sizeof(a));
ClDFL1.Write(b,sizeof(b));
ClDFL1.Write(C,sizeof(C));
ClDFL1.Write(a,sizeof(a));
ClDFL1.Write(b,sizeof(b));
ClDFL1.Write(C,sizeof(C));
for c:= 1 to 10000 do {when i set 10000 to 10000000 the
program give an EStreamError
Out of memory while expanding Memory After runing a wile }
begin
a := 1;
b := a * 2;
Sn := (ClDFL1.Size div 7) - 2;
// Sn := Sn + C;
{insert into C1dFL1}
Cntfr := ClDFL1.Size - (Sn*7);
ClDFL1.Seek((sn*7),0);
ClDFL2.SetSize(0);
ClDFL2.CopyFrom(ClDFL1,Cntfr);
ClDFL2.Seek(0,0);
ClDFL1.SetSize(Sn*7);
ClDFL1.Write(a,sizeof(a));
ClDFL1.Write(b,sizeof(b));
ClDFL1.Write(C,sizeof(C));

ClDFL1.CopyFrom(ClDFL2,ClDFL2.Size);
{End of insert into C1dFL1}

end;
TMEnd := now;
TmeDiv := TMEnd - TMStart;
TmeDiv := TmeDiv / 0.0000000115740740740740740740740;
Label1.Caption := FormatFloat('0 ms',TmeDiv);

ClDFL1.Free;
ClDFL2.Free;

end;

procedure TForm1.Button2Click(Sender: TObject);
var TMStart,TMEnd : TDateTime;
TmeDiv : TDateTime;
a : Shortint;
b : Smallint;
c : Longint;
Sn : Longint;
ClDFL1 : TFileStream;
ClDFL2 : TFileStream;
Cntfr : Longint;
begin
TMStart:= now;
ClDFL1 := TFileStream.Create('file1.ttt',fmCreate);
ClDFL2 := TFileStream.Create('file2.ttt',fmCreate);
ClDFL1.Write(a,sizeof(a));
ClDFL1.Write(b,sizeof(b));
ClDFL1.Write(C,sizeof(C));
ClDFL1.Write(a,sizeof(a));
ClDFL1.Write(b,sizeof(b));
ClDFL1.Write(C,sizeof(C));

for c:= 1 to 10000 do //00000
begin
a := 1;
b := a * 2;
Sn := (ClDFL1.Size div 7) - 2;
// Sn := Sn + C;
Cntfr := ClDFL1.Size - (Sn*7);
ClDFL1.Seek((sn*7),0);
ClDFL2.Size := 0;
ClDFL2.CopyFrom(ClDFL1,Cntfr);
ClDFL2.Seek(0,0);
ClDFL1.Size := Sn*7;
ClDFL1.Write(a,sizeof(a));
ClDFL1.Write(b,sizeof(b));
ClDFL1.Write(C,sizeof(C));

ClDFL1.CopyFrom(ClDFL2,ClDFL2.Size);

end;
TMEnd := now;
TmeDiv := TMEnd - TMStart;
TmeDiv := TmeDiv / 0.0000000115740740740740740740740;
Label1.Caption := FormatFloat('0 ms',TmeDiv);

ClDFL1.Free;
ClDFL2.Free;

end;

Madshi

The file mapping functions (CreateFileMapping etc) are very fast, but you have to specify how long the resulting file will be. Do you know that at the time where you open the file? BTW, inserting is a problem (regardless which method you're using), because it involves moving of all the data that comes behind the inserting point. So if you want to be fast, avoid inserting as much as possible...

Regards, Madshi.

jnfdelpro

ASKER

I dont know what the end size will be but it will not be more than 30 mb
I have test it with Tlist and the insert work Fast
Is there any whay to get mor dynamic memory?

Is there a beter way to insert?

Madshi

TList is totally different. Each TList entry is only 4 bytes long. So inserting doesn't involve that much movement.
But let's say your file will be 20MB big and you insert one byte at the beginning of the file. The result is that you have to move 20MB only because you inserted one byte. You see what I mean?

>> Is there a beter way to insert?

The best way would be to avoid inserting at all. If that does not work, then it would be a good idea to split the big file into several smaller pieces, if that is possible. If that is not possible, you have to live with the fact, that inserting is slow (especially when inserting near the beginning of the file).

Regards, Madshi.

FrodoBeggins

Inserting is fast in dynamic structures. But there searching is slower. In the other hand, in an array inserting is slower, but searching can be varry fast (if the array is sorted).
Do you need a sorted structure at all? I don't know what's the data you're trying to save, but APPENDING is always fast.
If you have idea about the structure and the size of all the thing, you can go managing directly the memory. Delphi?s memory manager is optimized for applications that allocate large numbers of small- to medium-sized blocks, as is typical for object-oriented applications and applications that process string data. If you need a different thing you'd better use VirtualAlloc and it's brothers.
Example: You have to do a spreadsheet, MxN cells. Allocate all the memory it _might_ need and commit it when you write something in a cell. Windows will manage the memory for you.

jnfdelpro

ASKER

I am going to insert most at the time at the end of the file and somtime in the midel but not in the begining
Do u have a exsample thow to add ,read and insert into
(CreateFileMapping etc)

bkg97

I just ran your program and got no error (with 384Mb of memory).
The problem is neither the required memory, nor the TMemoryStream. It's just the amount of allocations/deallocations you make.
Playing with file mapping won't be the solution (and might be even worse).
You could change your method:
1. Change your algorithm (avoid multiple .setsize and copyfrom)
2. Allocate large memory area only one time (for both your two TMemoryStream), then insert and append data with the Move procedure.

AvonWyss

jnfdelpro, as Madhsi said, inserting is the problem you're facing. You may try this approach:
* use a TList (or similar) to keep a list of CHANGES you're doing to the stream. E.g., at position 10, insert 20 bytes... etc.
* before closing the stream, process the original stream with your modification list. By this, you will only once physically move all the content (and thus you can do it on disk pretty fast).

HTH

ASKER CERTIFIED SOLUTION

Russell Libby

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

jnfdelpro

ASKER

I get an error Published property 'Memory' cannot be of type POINTER

property Memory: PChar read FMemory;

Is there anything i must do to let it work?

jnfdelpro

ASKER

I got it right
All I need is a LoadFromFile and SaveToFile Methods

Russell Libby

infdelpro,

// Public Declarations
public
procedure SaveToFile(AFilename: String);
procedure SaveToStream(AStream: TStream);
procedure LoadFromFile(AFileName: String);
procedure LoadFromStream(AStream: TStream);

implementation

procedure TMemStream.LoadFromStream(AStream: TStream);
var dwread: Integer;
s: String;
begin

// Read data from stream starting at the streams position
dwread:=AStream.Size-AStream.Position;
SetString(s, nil, dwread);
AStream.Read(Pointer(s)^, dwread);

// Reset our stream to 0 and write the data
SetNewSize(0);
Write(Pointer(s)^, dwread);

end;

procedure TMemStream.LoadFromFile(AFileName: String);
var Stream: TStream;
begin

Stream:=TFileStream.Create(AFileName, fmOpenRead);
try
LoadFromStream(Stream);
finally
Stream.Free;
end;

end;

procedure TMemStream.SaveToStream(AStream: TStream);
begin

AStream.WriteBuffer(FMemory^, FSize);

end;

procedure TMemStream.SaveToFile(AFilename: String);
var Stream: TStream;
begin

Stream:=TFileStream.Create(AFileName, fmCreate);
try
SaveToStream(Stream);
finally
Stream.Free;
end;

end;