Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 5858
  • Last Modified:

EStreamError Out of Memory while expanding memory stream

I get an EStreamError when i use TMemory stream
and when I use TFileStream it is to slow
Is there an better way to write some info to memory or file but vary fast
I want to be able to write ,read and insert to stream

I add a program to test TMemoryStream and TFileStream how long it will take in milliseconds.

does any one how to work with the api functions CreateFile, CreateFileMapping & MapViewOfFile
want to know how to write, read and importent to insert into

{ On a clean form put  a TLabel and two TButons }

procedure TForm1.Button1Click(Sender: TObject);
var TMStart,TMEnd : TDateTime;
TmeDiv : TDateTime;
a : Shortint;
b : Smallint;
c : Longint;
Sn : Longint;
 ClDFL1  : TMemoryStream;
 ClDFL2  : TMemoryStream;
Cntfr : Longint;
begin
TMStart:= now;
ClDFL1 := TMemoryStream.Create;
ClDFL2 := TMemoryStream.Create;
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));
for c:= 1 to 10000 do    {when i set 10000 to 10000000 the
                          program give an EStreamError
                          Out of memory while expanding Memory After runing a wile }
begin
  a := 1;
  b := a * 2;
  Sn := (ClDFL1.Size div 7) - 2;
//  Sn := Sn + C;
{insert into C1dFL1}
  Cntfr := ClDFL1.Size - (Sn*7);
  ClDFL1.Seek((sn*7),0);
  ClDFL2.SetSize(0);
  ClDFL2.CopyFrom(ClDFL1,Cntfr);
  ClDFL2.Seek(0,0);
  ClDFL1.SetSize(Sn*7);
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));

  ClDFL1.CopyFrom(ClDFL2,ClDFL2.Size);
{End of insert into C1dFL1}

end;
TMEnd  := now;
 TmeDiv := TMEnd - TMStart;
  TmeDiv := TmeDiv / 0.0000000115740740740740740740740;
Label1.Caption := FormatFloat('0 ms',TmeDiv);

ClDFL1.Free;
ClDFL2.Free;

end;

procedure TForm1.Button2Click(Sender: TObject);
var TMStart,TMEnd : TDateTime;
TmeDiv : TDateTime;
a : Shortint;
b : Smallint;
c : Longint;
Sn : Longint;
 ClDFL1  : TFileStream;
 ClDFL2  : TFileStream;
Cntfr : Longint;
begin
TMStart:= now;
ClDFL1 := TFileStream.Create('file1.ttt',fmCreate);
ClDFL2 := TFileStream.Create('file2.ttt',fmCreate);
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));

for c:= 1 to 10000 do  //00000
begin
  a := 1;
  b := a * 2;
  Sn := (ClDFL1.Size div 7) - 2;
//  Sn := Sn + C;
  Cntfr := ClDFL1.Size - (Sn*7);
  ClDFL1.Seek((sn*7),0);
  ClDFL2.Size := 0;
  ClDFL2.CopyFrom(ClDFL1,Cntfr);
  ClDFL2.Seek(0,0);
  ClDFL1.Size := Sn*7;
  ClDFL1.Write(a,sizeof(a));
  ClDFL1.Write(b,sizeof(b));
  ClDFL1.Write(C,sizeof(C));

  ClDFL1.CopyFrom(ClDFL2,ClDFL2.Size);


end;
TMEnd  := now;
 TmeDiv := TMEnd - TMStart;
  TmeDiv := TmeDiv / 0.0000000115740740740740740740740;
Label1.Caption := FormatFloat('0 ms',TmeDiv);

ClDFL1.Free;
ClDFL2.Free;

end;
0
jnfdelpro
Asked:
jnfdelpro
  • 4
  • 2
  • 2
  • +3
1 Solution
 
MadshiCommented:
The file mapping functions (CreateFileMapping etc) are very fast, but you have to specify how long the resulting file will be. Do you know that at the time where you open the file? BTW, inserting is a problem (regardless which method you're using), because it involves moving of all the data that comes behind the inserting point. So if you want to be fast, avoid inserting as much as possible...

Regards, Madshi.
0
 
jnfdelproAuthor Commented:
I dont know what the end size will be but it will not be more than 30 mb
I have test it with Tlist and the insert work Fast
Is there any whay to get mor dynamic memory?

Is there a beter way to insert?
0
 
MadshiCommented:
TList is totally different. Each TList entry is only 4 bytes long. So inserting doesn't involve that much movement.
But let's say your file will be 20MB big and you insert one byte at the beginning of the file. The result is that you have to move 20MB only because you inserted one byte. You see what I mean?

>> Is there a beter way to insert?

The best way would be to avoid inserting at all. If that does not work, then it would be a good idea to split the big file into several smaller pieces, if that is possible. If that is not possible, you have to live with the fact, that inserting is slow (especially when inserting near the beginning of the file).

Regards, Madshi.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
FrodoBegginsCommented:
Inserting is fast in dynamic structures. But there searching is slower. In the other hand, in an array inserting is slower, but searching can be varry fast (if the array is sorted).
Do you need a sorted structure at all? I don't know what's the data you're trying to save, but APPENDING is always fast.
If you have idea about the structure and the size of all the thing, you can go managing directly the memory. Delphi?s memory manager is optimized for applications that allocate large numbers of small- to medium-sized blocks, as is typical for object-oriented applications and applications that process string data. If you need a different thing you'd better use VirtualAlloc and it's brothers.
Example: You have to do a spreadsheet, MxN cells. Allocate all the memory it _might_ need and commit it when you write something in a cell. Windows will manage the memory for you.
0
 
jnfdelproAuthor Commented:
I am going to insert most at the time at the end of the file and somtime in the midel but not in the begining
Do u have a exsample thow to add ,read and insert into
(CreateFileMapping etc)
0
 
bkg97Commented:
I just ran your program and got no error (with 384Mb of memory).
The problem is neither the required memory, nor the TMemoryStream. It's just the amount of allocations/deallocations you make.
Playing with file mapping won't be the solution (and might be even worse).
You could change your method:
1. Change your algorithm (avoid multiple .setsize and copyfrom)
2. Allocate large memory area only one time (for both your two TMemoryStream), then insert and append data with the Move procedure.
0
 
AvonWyssCommented:
jnfdelpro, as Madhsi said, inserting is the problem you're facing. You may try this approach:
* use a TList (or similar) to keep a list of CHANGES you're doing to the stream. E.g., at position 10, insert 20 bytes... etc.
* before closing the stream, process the original stream with your modification list. By this, you will only once physically move all the content (and thus you can do it on disk pretty fast).

HTH
0
 
Russell LibbySoftware Engineer, Advisory Commented:
infdelpro,

The comment from bkg97 is very accurate, it is the allocation/reallocation that is VERY time consuming. For each change in size, the stream will realloc to the new size. I have written you an object that you can drop in to replace the TMemoryStream (i called this one TMemStream). The theory behind it is that it only realloc's mem when the size is increased. And, it realloc's in 8KB blocks (the user is given the size, but the object holds the actual amount). So setting size to zero will not release any memory, but the size property and position are changed. Then when size is increased, the memory is already there, and no reallocs are done. If you need to explicitly free the memory, there is a ShrinkToSize procedure that can be used (clear will also realloc as well)

The timing results were very promising:

TMemoryStream = 19000 ms for 1,000 iterations
TFileStream   = 7741 ms for 10,000 iterations
TMemStream    = 20 ms for 10,000 iterations

Russell

unit memstream;

interface

uses
  Windows, SysUtils, Classes;

const
  ALLOC_SIZE     =  8192;

type
  TMemStream     =  class(TObject)
  private
     // Private declarations
     FMemory:    PChar;
     FPosition:  Integer;
     FSize:      Integer;
     FActual:    Integer;
  protected
     // Protected declarations
     procedure   SetPosition(APosition: Integer);
     procedure   SetNewSize(ASize: Integer);
  public
     // Public declarations
     constructor Create;
     destructor  Destroy; override;
     procedure   Clear;
            procedure   SetSize(ASize: Integer);
            function    Write(const ABuffer; ACount: Integer): Integer;
     function    Read(var ABuffer; ACount: Integer): Integer;
            function    Seek(AOffset: Integer; AOrigin: Word): Integer;
     function    CopyFrom(ASource: TMemStream; ACount: Integer): Integer;
     procedure   ShrinkToSize;
  published
     // Published declarations
     property    Memory: PChar read FMemory;
     property    Position: Integer read FPosition write SetPosition;
     property    Size: Integer read FSize write SetNewSize;
  end;

implementation

function TMemStream.CopyFrom(ASource: TMemStream; ACount: Integer): Integer;
var  dwCopy:     Integer;
begin

  // Get the number of bytes that we will actually copy
  if (ACount = 0) then
  begin
     // Copy all of source stream
     ASource.Position:=0;
     dwCopy:=ASource.Size;
  end
  else
  begin
     // Check to see if count+position is greater than size
     if (ACount+ASource.Position > ASource.Size) then
        dwCopy:=ASource.Size-ASource.Position
     else
        dwCopy:=ACount;
  end;

  // Make sure we allocate enough memory for the write
  if (FPosition+dwCopy >= FActual) then SetNewSize(FPosition+dwCopy);

  // Perform the copy
  Move(ASource.Memory[ASource.Position], FMemory[FPosition], dwCopy);

  // Update the source position
  ASource.Position:=ASource.Position+dwCopy;

  // Update our stream position (and possibly size)
  Inc(FPosition, dwCopy);
  if (FPosition > FSize) then FSize:=FPosition;

  // Return number of bytes actually copied
  result:=dwCopy;

end;

procedure TMemStream.SetPosition(APosition: Integer);
begin

  // Check bounds of APosition
  if (APosition < 0) or (APosition > FSize) then exit;

  // Set new position
  FPosition:=APosition;

end;

procedure TMemStream.SetNewSize(ASize: Integer);
var  dwBytes:    Integer;
begin

  // Exit if new size is less than zero
  if (ASize < 0) then exit;

  // Is new size smaller than what we actually have?
  if (ASize < FActual) then
  begin
     // Update the size and position
     if (ASize < FSize) then FPosition:=ASize;
     FSize:=ASize;
  end
  else
  begin
     // Allocate more memory for the stream
     dwBytes:=(ASize div ALLOC_SIZE + 1) * ALLOC_SIZE;
     ReallocMem(FMemory, dwBytes);
     FActual:=dwBytes;
     FSize:=ASize;
  end;

end;

procedure TMemStream.Clear;
begin

  // Free existing memory and create new ALLOC_SIZE block
  FreeMem(FMemory);
  FMemory:=AllocMem(ALLOC_SIZE);
  FActual:=ALLOC_SIZE;
  FSize:=0;
  FPosition:=0;

end;

procedure TMemStream.SetSize(ASize: Integer);
begin

  // Just call SetNewSize
  SetNewSize(ASize);

end;

function TMemStream.Seek(AOffset: Integer; AOrigin: Word): Integer;
var  dwPos:      Integer;
begin

  // Determine the new position
  dwPos:=AOffset;
  case AOrigin of
     soFromBeginning   :  dwPos:=AOffset;
     soFromCurrent         :  dwPos:=FPosition+AOffset;
     soFromEnd         :  dwPos:=FSize+AOffset;
  end;

  // Range checking
  if (dwPos < 0) then dwPos:=0
  else if (dwPos > FSize) then dwPos:=FSize;

  // Set new position
  FPosition:=dwPos;
  result:=FPosition;

end;

function TMemStream.Write(const ABuffer; ACount: Integer): Integer;
begin

  // Set default result
  result:=0;

  // Make sure count is greater than zero
  if (ACount < 0) then exit;

  // Make sure we have enough memory
  if (FPosition+ACount >= FActual) then SetNewSize(FPosition+ACount);

  // Move the data
  Move(ABuffer, FMemory[FPosition], ACount);

  // Update the position and size
  Inc(FPosition, ACount);
  if (FPosition > FSize) then FSize:=FPosition;

  // Set result
  result:=ACount;

end;

function TMemStream.Read(var ABuffer; ACount: Integer): Integer;
begin

  // Set default result
  result:=0;

  // Make sure count is greater than zero
  if (ACount < 0) then exit;

  // Make sure we have enough memory, otherwise we will update the read count
  if (FPosition+ACount > FSize) then
     result:=FSize-FPosition
  else
     result:=ACount;

  // If result is zero than there is nothing to do
  if (result = 0) then exit;

  // Move the data to passed in buffer
  Move(FMemory[FPosition], ABuffer, result);

  // Update the position and size
  Inc(FPosition, result);

end;

procedure TMemStream.ShrinkToSize;
begin

  // This will set the actual bytes to the current size
  ReallocMem(FMemory, FSize);
  FActual:=FSize;

end;

constructor TMemStream.Create;
begin

  // Perform the inherited
  inherited Create;

  // Allocate ALLOC_SIZE to start with
  FMemory:=AllocMem(ALLOC_SIZE);
  FActual:=ALLOC_SIZE;

  // The position and size are set to zero
  FPosition:=0;
  FSize:=0;

end;

destructor TMemStream.Destroy;
begin

  // Free the allocated memory
  FreeMem(FMemory, FActual);

  // Perform the inherited
  inherited Destroy;

end;

end.



0
 
jnfdelproAuthor Commented:
I get an error Published property 'Memory' cannot be of type POINTER

    property    Memory: PChar read FMemory;

Is there anything i must do to let it work?
0
 
jnfdelproAuthor Commented:
I got it right
All I need is a LoadFromFile and SaveToFile Methods
0
 
Russell LibbySoftware Engineer, Advisory Commented:
infdelpro,

// Public Declarations
public
    procedure   SaveToFile(AFilename: String);
    procedure   SaveToStream(AStream: TStream);
    procedure   LoadFromFile(AFileName: String);
    procedure   LoadFromStream(AStream: TStream);

implementation

procedure TMemStream.LoadFromStream(AStream: TStream);
var  dwread:     Integer;
     s:          String;
begin

  // Read data from stream starting at the streams position
  dwread:=AStream.Size-AStream.Position;
  SetString(s, nil, dwread);
  AStream.Read(Pointer(s)^, dwread);

  // Reset our stream to 0 and write the data
  SetNewSize(0);
  Write(Pointer(s)^, dwread);

end;

procedure TMemStream.LoadFromFile(AFileName: String);
var  Stream:     TStream;
begin

  Stream:=TFileStream.Create(AFileName, fmOpenRead);
  try
     LoadFromStream(Stream);
  finally
     Stream.Free;
  end;

end;

procedure TMemStream.SaveToStream(AStream: TStream);
begin

  AStream.WriteBuffer(FMemory^, FSize);

end;

procedure TMemStream.SaveToFile(AFilename: String);
var  Stream:     TStream;
begin

  Stream:=TFileStream.Create(AFileName, fmCreate);
  try
     SaveToStream(Stream);
  finally
     Stream.Free;
  end;

end;

0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

  • 4
  • 2
  • 2
  • +3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now