Solved

EStreamError Out of Memory while expanding memory stream

Posted on 2001-06-04
11
5,455 Views
Last Modified: 2013-01-16
I get an EStreamError when i use TMemory stream
and when I use TFileStream it is to slow
Is there an better way to write some info to memory or file but vary fast
I want to be able to write ,read and insert to stream

I add a program to test TMemoryStream and TFileStream how long it will take in milliseconds.

does any one how to work with the api functions CreateFile, CreateFileMapping & MapViewOfFile
want to know how to write, read and importent to insert into

{ On a clean form put  a TLabel and two TButons }

procedure TForm1.Button1Click(Sender: TObject);
var TMStart,TMEnd : TDateTime;
TmeDiv : TDateTime;
a : Shortint;
b : Smallint;
c : Longint;
Sn : Longint;
 ClDFL1  : TMemoryStream;
 ClDFL2  : TMemoryStream;
Cntfr : Longint;
begin
TMStart:= now;
ClDFL1 := TMemoryStream.Create;
ClDFL2 := TMemoryStream.Create;
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));
for c:= 1 to 10000 do    {when i set 10000 to 10000000 the
                          program give an EStreamError
                          Out of memory while expanding Memory After runing a wile }
begin
  a := 1;
  b := a * 2;
  Sn := (ClDFL1.Size div 7) - 2;
//  Sn := Sn + C;
{insert into C1dFL1}
  Cntfr := ClDFL1.Size - (Sn*7);
  ClDFL1.Seek((sn*7),0);
  ClDFL2.SetSize(0);
  ClDFL2.CopyFrom(ClDFL1,Cntfr);
  ClDFL2.Seek(0,0);
  ClDFL1.SetSize(Sn*7);
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));

  ClDFL1.CopyFrom(ClDFL2,ClDFL2.Size);
{End of insert into C1dFL1}

end;
TMEnd  := now;
 TmeDiv := TMEnd - TMStart;
  TmeDiv := TmeDiv / 0.0000000115740740740740740740740;
Label1.Caption := FormatFloat('0 ms',TmeDiv);

ClDFL1.Free;
ClDFL2.Free;

end;

procedure TForm1.Button2Click(Sender: TObject);
var TMStart,TMEnd : TDateTime;
TmeDiv : TDateTime;
a : Shortint;
b : Smallint;
c : Longint;
Sn : Longint;
 ClDFL1  : TFileStream;
 ClDFL2  : TFileStream;
Cntfr : Longint;
begin
TMStart:= now;
ClDFL1 := TFileStream.Create('file1.ttt',fmCreate);
ClDFL2 := TFileStream.Create('file2.ttt',fmCreate);
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));

for c:= 1 to 10000 do  //00000
begin
  a := 1;
  b := a * 2;
  Sn := (ClDFL1.Size div 7) - 2;
//  Sn := Sn + C;
  Cntfr := ClDFL1.Size - (Sn*7);
  ClDFL1.Seek((sn*7),0);
  ClDFL2.Size := 0;
  ClDFL2.CopyFrom(ClDFL1,Cntfr);
  ClDFL2.Seek(0,0);
  ClDFL1.Size := Sn*7;
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));

  ClDFL1.CopyFrom(ClDFL2,ClDFL2.Size);


end;
TMEnd  := now;
 TmeDiv := TMEnd - TMStart;
  TmeDiv := TmeDiv / 0.0000000115740740740740740740740;
Label1.Caption := FormatFloat('0 ms',TmeDiv);

ClDFL1.Free;
ClDFL2.Free;

end;
0
Comment
Question by:jnfdelpro
  • 4
  • 2
  • 2
  • +3
11 Comments
 
LVL 20

Expert Comment

by:Madshi
Comment Utility
The file mapping functions (CreateFileMapping etc) are very fast, but you have to specify how long the resulting file will be. Do you know that at the time where you open the file? BTW, inserting is a problem (regardless which method you're using), because it involves moving of all the data that comes behind the inserting point. So if you want to be fast, avoid inserting as much as possible...

Regards, Madshi.
0
 

Author Comment

by:jnfdelpro
Comment Utility
I dont know what the end size will be but it will not be more than 30 mb
I have test it with Tlist and the insert work Fast
Is there any whay to get mor dynamic memory?

Is there a beter way to insert?
0
 
LVL 20

Expert Comment

by:Madshi
Comment Utility
TList is totally different. Each TList entry is only 4 bytes long. So inserting doesn't involve that much movement.
But let's say your file will be 20MB big and you insert one byte at the beginning of the file. The result is that you have to move 20MB only because you inserted one byte. You see what I mean?

>> Is there a beter way to insert?

The best way would be to avoid inserting at all. If that does not work, then it would be a good idea to split the big file into several smaller pieces, if that is possible. If that is not possible, you have to live with the fact, that inserting is slow (especially when inserting near the beginning of the file).

Regards, Madshi.
0
 
LVL 2

Expert Comment

by:FrodoBeggins
Comment Utility
Inserting is fast in dynamic structures. But there searching is slower. In the other hand, in an array inserting is slower, but searching can be varry fast (if the array is sorted).
Do you need a sorted structure at all? I don't know what's the data you're trying to save, but APPENDING is always fast.
If you have idea about the structure and the size of all the thing, you can go managing directly the memory. Delphi?s memory manager is optimized for applications that allocate large numbers of small- to medium-sized blocks, as is typical for object-oriented applications and applications that process string data. If you need a different thing you'd better use VirtualAlloc and it's brothers.
Example: You have to do a spreadsheet, MxN cells. Allocate all the memory it _might_ need and commit it when you write something in a cell. Windows will manage the memory for you.
0
 

Author Comment

by:jnfdelpro
Comment Utility
I am going to insert most at the time at the end of the file and somtime in the midel but not in the begining
Do u have a exsample thow to add ,read and insert into
(CreateFileMapping etc)
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Expert Comment

by:bkg97
Comment Utility
I just ran your program and got no error (with 384Mb of memory).
The problem is neither the required memory, nor the TMemoryStream. It's just the amount of allocations/deallocations you make.
Playing with file mapping won't be the solution (and might be even worse).
You could change your method:
1. Change your algorithm (avoid multiple .setsize and copyfrom)
2. Allocate large memory area only one time (for both your two TMemoryStream), then insert and append data with the Move procedure.
0
 
LVL 14

Expert Comment

by:AvonWyss
Comment Utility
jnfdelpro, as Madhsi said, inserting is the problem you're facing. You may try this approach:
* use a TList (or similar) to keep a list of CHANGES you're doing to the stream. E.g., at position 10, insert 20 bytes... etc.
* before closing the stream, process the original stream with your modification list. By this, you will only once physically move all the content (and thus you can do it on disk pretty fast).

HTH
0
 
LVL 26

Accepted Solution

by:
Russell Libby earned 300 total points
Comment Utility
infdelpro,

The comment from bkg97 is very accurate, it is the allocation/reallocation that is VERY time consuming. For each change in size, the stream will realloc to the new size. I have written you an object that you can drop in to replace the TMemoryStream (i called this one TMemStream). The theory behind it is that it only realloc's mem when the size is increased. And, it realloc's in 8KB blocks (the user is given the size, but the object holds the actual amount). So setting size to zero will not release any memory, but the size property and position are changed. Then when size is increased, the memory is already there, and no reallocs are done. If you need to explicitly free the memory, there is a ShrinkToSize procedure that can be used (clear will also realloc as well)

The timing results were very promising:

TMemoryStream = 19000 ms for 1,000 iterations
TFileStream   = 7741 ms for 10,000 iterations
TMemStream    = 20 ms for 10,000 iterations

Russell

unit memstream;

interface

uses
  Windows, SysUtils, Classes;

const
  ALLOC_SIZE     =  8192;

type
  TMemStream     =  class(TObject)
  private
     // Private declarations
     FMemory:    PChar;
     FPosition:  Integer;
     FSize:      Integer;
     FActual:    Integer;
  protected
     // Protected declarations
     procedure   SetPosition(APosition: Integer);
     procedure   SetNewSize(ASize: Integer);
  public
     // Public declarations
     constructor Create;
     destructor  Destroy; override;
     procedure   Clear;
            procedure   SetSize(ASize: Integer);
            function    Write(const ABuffer; ACount: Integer): Integer;
     function    Read(var ABuffer; ACount: Integer): Integer;
            function    Seek(AOffset: Integer; AOrigin: Word): Integer;
     function    CopyFrom(ASource: TMemStream; ACount: Integer): Integer;
     procedure   ShrinkToSize;
  published
     // Published declarations
     property    Memory: PChar read FMemory;
     property    Position: Integer read FPosition write SetPosition;
     property    Size: Integer read FSize write SetNewSize;
  end;

implementation

function TMemStream.CopyFrom(ASource: TMemStream; ACount: Integer): Integer;
var  dwCopy:     Integer;
begin

  // Get the number of bytes that we will actually copy
  if (ACount = 0) then
  begin
     // Copy all of source stream
     ASource.Position:=0;
     dwCopy:=ASource.Size;
  end
  else
  begin
     // Check to see if count+position is greater than size
     if (ACount+ASource.Position > ASource.Size) then
        dwCopy:=ASource.Size-ASource.Position
     else
        dwCopy:=ACount;
  end;

  // Make sure we allocate enough memory for the write
  if (FPosition+dwCopy >= FActual) then SetNewSize(FPosition+dwCopy);

  // Perform the copy
  Move(ASource.Memory[ASource.Position], FMemory[FPosition], dwCopy);

  // Update the source position
  ASource.Position:=ASource.Position+dwCopy;

  // Update our stream position (and possibly size)
  Inc(FPosition, dwCopy);
  if (FPosition > FSize) then FSize:=FPosition;

  // Return number of bytes actually copied
  result:=dwCopy;

end;

procedure TMemStream.SetPosition(APosition: Integer);
begin

  // Check bounds of APosition
  if (APosition < 0) or (APosition > FSize) then exit;

  // Set new position
  FPosition:=APosition;

end;

procedure TMemStream.SetNewSize(ASize: Integer);
var  dwBytes:    Integer;
begin

  // Exit if new size is less than zero
  if (ASize < 0) then exit;

  // Is new size smaller than what we actually have?
  if (ASize < FActual) then
  begin
     // Update the size and position
     if (ASize < FSize) then FPosition:=ASize;
     FSize:=ASize;
  end
  else
  begin
     // Allocate more memory for the stream
     dwBytes:=(ASize div ALLOC_SIZE + 1) * ALLOC_SIZE;
     ReallocMem(FMemory, dwBytes);
     FActual:=dwBytes;
     FSize:=ASize;
  end;

end;

procedure TMemStream.Clear;
begin

  // Free existing memory and create new ALLOC_SIZE block
  FreeMem(FMemory);
  FMemory:=AllocMem(ALLOC_SIZE);
  FActual:=ALLOC_SIZE;
  FSize:=0;
  FPosition:=0;

end;

procedure TMemStream.SetSize(ASize: Integer);
begin

  // Just call SetNewSize
  SetNewSize(ASize);

end;

function TMemStream.Seek(AOffset: Integer; AOrigin: Word): Integer;
var  dwPos:      Integer;
begin

  // Determine the new position
  dwPos:=AOffset;
  case AOrigin of
     soFromBeginning   :  dwPos:=AOffset;
     soFromCurrent         :  dwPos:=FPosition+AOffset;
     soFromEnd         :  dwPos:=FSize+AOffset;
  end;

  // Range checking
  if (dwPos < 0) then dwPos:=0
  else if (dwPos > FSize) then dwPos:=FSize;

  // Set new position
  FPosition:=dwPos;
  result:=FPosition;

end;

function TMemStream.Write(const ABuffer; ACount: Integer): Integer;
begin

  // Set default result
  result:=0;

  // Make sure count is greater than zero
  if (ACount < 0) then exit;

  // Make sure we have enough memory
  if (FPosition+ACount >= FActual) then SetNewSize(FPosition+ACount);

  // Move the data
  Move(ABuffer, FMemory[FPosition], ACount);

  // Update the position and size
  Inc(FPosition, ACount);
  if (FPosition > FSize) then FSize:=FPosition;

  // Set result
  result:=ACount;

end;

function TMemStream.Read(var ABuffer; ACount: Integer): Integer;
begin

  // Set default result
  result:=0;

  // Make sure count is greater than zero
  if (ACount < 0) then exit;

  // Make sure we have enough memory, otherwise we will update the read count
  if (FPosition+ACount > FSize) then
     result:=FSize-FPosition
  else
     result:=ACount;

  // If result is zero than there is nothing to do
  if (result = 0) then exit;

  // Move the data to passed in buffer
  Move(FMemory[FPosition], ABuffer, result);

  // Update the position and size
  Inc(FPosition, result);

end;

procedure TMemStream.ShrinkToSize;
begin

  // This will set the actual bytes to the current size
  ReallocMem(FMemory, FSize);
  FActual:=FSize;

end;

constructor TMemStream.Create;
begin

  // Perform the inherited
  inherited Create;

  // Allocate ALLOC_SIZE to start with
  FMemory:=AllocMem(ALLOC_SIZE);
  FActual:=ALLOC_SIZE;

  // The position and size are set to zero
  FPosition:=0;
  FSize:=0;

end;

destructor TMemStream.Destroy;
begin

  // Free the allocated memory
  FreeMem(FMemory, FActual);

  // Perform the inherited
  inherited Destroy;

end;

end.



0
 

Author Comment

by:jnfdelpro
Comment Utility
I get an error Published property 'Memory' cannot be of type POINTER

    property    Memory: PChar read FMemory;

Is there anything i must do to let it work?
0
 

Author Comment

by:jnfdelpro
Comment Utility
I got it right
All I need is a LoadFromFile and SaveToFile Methods
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility
infdelpro,

// Public Declarations
public
    procedure   SaveToFile(AFilename: String);
    procedure   SaveToStream(AStream: TStream);
    procedure   LoadFromFile(AFileName: String);
    procedure   LoadFromStream(AStream: TStream);

implementation

procedure TMemStream.LoadFromStream(AStream: TStream);
var  dwread:     Integer;
     s:          String;
begin

  // Read data from stream starting at the streams position
  dwread:=AStream.Size-AStream.Position;
  SetString(s, nil, dwread);
  AStream.Read(Pointer(s)^, dwread);

  // Reset our stream to 0 and write the data
  SetNewSize(0);
  Write(Pointer(s)^, dwread);

end;

procedure TMemStream.LoadFromFile(AFileName: String);
var  Stream:     TStream;
begin

  Stream:=TFileStream.Create(AFileName, fmOpenRead);
  try
     LoadFromStream(Stream);
  finally
     Stream.Free;
  end;

end;

procedure TMemStream.SaveToStream(AStream: TStream);
begin

  AStream.WriteBuffer(FMemory^, FSize);

end;

procedure TMemStream.SaveToFile(AFilename: String);
var  Stream:     TStream;
begin

  Stream:=TFileStream.Create(AFileName, fmCreate);
  try
     SaveToStream(Stream);
  finally
     Stream.Free;
  end;

end;

0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now