Solved

Alternative to TMemoryStream

Posted on 2004-03-25
24
517 Views
Last Modified: 2010-04-05
I've written a component that uses a TMemoryStream to buffer data. I'm running into a problem on some older machines (with 64 megs of RAM) that the stream gets to large for the system (50 megs of so).

Is there an alternative to TMemoryStream that I can use the physical disk space as my memory instead. I need to be able to size on the fly like TMemoryStream and also access the memory directly like TMemoryStream.memory.


Any ideas?


P.S. I wish I could simply make them upgrade the RAM, but I can't. The buffer is for video playback and I keep the video in memory after its played to allow Rewind and Fast Forward.
0
Comment
Question by:rbohac
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 9
  • 5
24 Comments
 
LVL 11

Expert Comment

by:shaneholmes
ID: 10682372
TFileStream

Shane
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10682739

TFileStream would let you truncate/expand the file (using SetFilePointer, SetEndOfFile), thus fulfilling one requirement.

Accessing data directly could also be done by creating a child class of TFileStream and exposing an indexed property called

Bytes[Index: Integer]: Byte read GetByte write SetByte;

This would allow you to read in a small block of the file to memory, allow you to manipulate it (and have some cached to speed up sequential access), and also allow you to save any changes should the bytes get modified. Not exactly like direct memory manipulation, but close.

Anyways, if something like this would work for you, I can put it together...

Regards,
Russell


0
 
LVL 11

Expert Comment

by:shaneholmes
ID: 10682805
Thanks RLibby, Your like the third person who has stepped on me today

Shane
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 11

Expert Comment

by:shaneholmes
ID: 10682808
But hey go ahead, its all yours....

Shane
0
 
LVL 6

Author Comment

by:rbohac
ID: 10682886
That would be an interesting way to go.  How would you create the child class?
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10682948

Shane,
I don't see how offering other suggestions/additions/comments/etc can be considered "stepping" on someone.... but sorry you feel that way. I was only trying to elaborate on your ONE word comment.

rbohac, I will put together a sample object that demos some of the ideas I had in mind.

-----

Russell


0
 
LVL 6

Author Comment

by:rbohac
ID: 10682973
technically two :)

Thanks Russell
0
 
LVL 11

Expert Comment

by:shaneholmes
ID: 10683054
My one word comment was waiting for him to respond if he wanted to go that route, so yes its stepping on it, but het like i said, its all yours RUSSELL

Shane
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10683277
Shane,

Im not going to get into a shouting match with you, but your "commenting" to a question does not make it "yours" exclusively. This is a collaborative forum, and we have ALL had to share. If you are willing to put the time into a TFileStream descendant... then by all means, do so.

But TFileStream as it stands would not cut it (it does not meet the asker's requirements).

Regards,
Russell
0
 
LVL 11

Expert Comment

by:shaneholmes
ID: 10683367
Im not going to give you the satisfaction of a shouting match russell, i said its all yours good luck... end of subject!

Shane

0
 
LVL 6

Author Comment

by:rbohac
ID: 10683713
Russell, I hope you'll still help me out...

BTW. I looked at TFileStream anyway before I posted. I actual posted this Q because it doesn't have memory access like the TMemoryStream
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10683928
Still here, regardless of othe comments....

Here is my first cut at this...
First, I should say, that if your checking a block of mem, the quickest way to do this is to perform a Stream.Read(...). This will perform the read as a single block, and evaluation can be done on the data quickly (as it is in memory).

This class does allow for direct byte manipulation, and is optimized for sequential access (both r/w). Please note though, due to file IO, it will still not be as fast as direct memory access. I am still looking at further optimizations for this..

Regards,
Russell

---------
unit MemFileStream;

interface

uses
  Windows, SysUtils, Classes;

type
  TMemFileStream =  class(TFileStream)
  private
     // Protected declarations
     FSize:      Integer;
     FPosition:  Integer;
     FFileName:  String;
  protected
     // Protected declaration
     function    GetByte(Index: Integer): Byte;
     procedure   SetByte(Index: Integer; Value: Byte);
     procedure   SetSize(NewSize: Longint); override;
  public
     // Public declarations
     constructor Create;
     destructor  Destroy; override;
     function    Read(var Buffer; Count: Longint): Longint; override;
     function    Write(const Buffer; Count: Longint): Longint; override;
     function    Seek(Offset: Longint; Origin: Word): Longint; override;
     property    Bytes[Index: Integer]: Byte read GetByte write SetByte;
  end;

implementation

function TMemFileStream.Read(var Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Read(Buffer, Count);

  // Update position
  FPosition:=Position;

end;

function TMemFileStream.Write(const Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Write(Buffer, Count);

  // Update position and size
  FSize:=Size;
  FPosition:=Position;

end;

function TMemFileStream.Seek(Offset: Longint; Origin: Word): Longint;
begin

  // Update position
  FPosition:=inherited Seek(Offset, Origin);

  // Return position
  result:=FPosition;

end;

procedure TMemFileStream.SetSize(NewSize: Longint);
begin

  // Set new size and position
  FSize:=Seek(NewSize, soFromBeginning);
  FPosition:=FSize;
  Win32Check(SetEndOfFile(Handle));

end;

function TMemFileStream.GetByte(Index: Integer): Byte;
var  dwSize:     Integer;
begin

  // Is the read outside of size?
  if (Index >= FSize) then
  begin
     SetSize(Succ(Index));
     FPosition:=Succ(Index);
     result:=0;
  end
  else
  begin
     // Position change?
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     FileRead(Handle, result, SizeOf(Char));
     Inc(FPosition);
  end;

end;

procedure TMemFileStream.SetByte(Index: Integer; Value: Byte);
var  dwSize:     Integer;
begin

  // Check index against size
  if (Index > FSize) then
  begin
     // Expand the file
     SetSize(Index);
     // Set new file position
     Position:=Index;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FSize);
     Inc(FPosition);
  end
  else
  begin
     // Check write position
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FPosition);
  end;

end;

constructor TMemFileStream.Create;
var  lpszPath:   Array [0..MAX_PATH] of Char;
     lpszFile:   Array [0..MAX_PATH] of Char;
begin

  // Create temp file for stream backing
  if (GetTempPath(MAX_PATH, lpszPath) > 0) then
  begin
     if (GetTempFileName(lpszPath, 'mfs', 0, lpszFile) <> 0) then
     begin
        // Save FileName
        FFileName:=lpszFile;
        // Perform inherited
        inherited Create(FFileName, fmOpenReadWrite or fmShareDenyWrite);
        // Set defaults
        FSize:=0;
        FPosition:=0;
     end
     else
        // Failed to create temp file
        RaiseLastWin32Error;
  end
  else
     // Failed to get temp path
     RaiseLastWin32Error;

end;

destructor TMemFileStream.Destroy;
begin

  // Close the file by calling inherited
  inherited Destroy;

  // Now delete the temp file backing
  DeleteFile(FFileName);

end;

end.

0
 
LVL 6

Author Comment

by:rbohac
ID: 10689429
Great! I'll start adding this in today although I probably won't be able to do much testing until Tuesday. I'm leaving town shortly.
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10689454

Have a good weekened then, and let me know if you run into problems next week

Russell
0
 
LVL 6

Author Comment

by:rbohac
ID: 10690517
I just thought of something. The reason I was using the MemoryStream.Memory was to that I could read from the memory stream without changing the position. This is because while I am reading the data (and laying back the video) I am also writing the data to the stream as it downloads.

Any ideas?
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10690559

The read and writes are overriden, so you could save an internal reader position (seperate from the Position property), and perform something along the lines of:

hold = seek(...)
read(...)
seek(... hold ...)

Allowing you to maintain your write position. It would help (in optimizing) if i could see some real world (code) usage for what you currently have as well.

This help any?
Russell
0
 
LVL 6

Author Comment

by:rbohac
ID: 10690882
There is too much code to throw on here.. plus I have to clean it up from my many attempts to do this.

Basically I have two threads. What happens is..

 - A thread that downloads the part of the data to a buffer in that thread
- the thread then calls a meth via synchronize that writes the data to the TMemoryStream
- It continues to do that until the file is completely downloaded

While this is happening I am running a loop in the main thread that parses that data
- It looks at the last position if read from and waits until the TMemoryStream has enough data to continue reading from
- It then reads a portion of that memory (via move) without updating the position and parses it accordingly

I wonder how much of a performance loss I will have by continually moving the position back and forth like that in the TMemFileStream
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10691040
Okay, I'm getting to see the picture here....

Actually, the hit should be minimal, and here is why.
-------
The file pointer will always be kept at the end, so the next write appends the data to the end of the stream. But, internally (in the object class), you can keep a read position indicator. When there is enough data to perform the read (by checking the read location against size or whatever you need to check), then the file position is marked, the seek is made, data is read into a block of memory, and the file pointer is reset to its last position, which just happens to be the end of file.

So the only time the file position is changed is when a block read occurs.

Another possibilty as well is this (I will need to do some testing):, open up 2 handles to the same file, one for reading, the other writing.

Read Mode  = fmOpenRead or fmShareDenyNone;
Write Mode  = fmOpenWrite or fmShareDenyWrite;

This way, each operation has its own file pointer, and no seeking needs to be done. IMHO, this would be the optimal approach for a reader/writer sharing the same medium.

----------
Russell
0
 
LVL 6

Author Comment

by:rbohac
ID: 10691321
I like that second approach, however I thought I read somewhere that that causes problems if you resize the file after it has been opened
0
 
LVL 26

Accepted Solution

by:
Russell Libby earned 250 total points
ID: 10691425

Not that my testing shows. I guess in the following scenario it could cause a read problem: read pos is at 1000 (for example), and the writer trucates the file to 100 bytes. When the reader goes to read, it reads in zero bytes.

This should not be your case through.

This is the test that I ran to check file sizing:

var  test:       TReaderWriter;
begin

  test:=TReaderWriter.Create;
  Randomize;
  while not(Application.Terminated) do
  begin
     Application.ProcessMessages;
     test.Writer.Size:=Random(1000000);
     Assert(test.Reader.Size = test.Writer.Size, 'File sizes do not match');
  end;
  test.Free;

end;

And here is the source for the TReaderWriter class and the stream types it exposes. This should provide a straight forward solution to your problem.

Russell

-----------

unit ReaderWriter;

interface

uses
  Windows, SysUtils, Classes, Consts;

type
  EFileStreamEx     =  class(Exception);

resourcestring
  resInvalidMode    =  'Invalid stream mode for this operation.';

type
  TFileStreamMode   =  (fsmReader, fsmWriter);
  TFileStreamEx     =  class(TStream)
  private
     // Private declarations
     FHandle:       Integer;
     FMode:         TFileStreamMode;
  protected
     // Protected declaration
     function       GetSize: Integer;
     procedure      SetSize(NewSize: Longint); override;
  public
     // Public declarations
     constructor    Create(FileName: String; Mode: TFileStreamMode);
     destructor     Destroy; override;
     function       Read(var Buffer; Count: Longint): Longint; override;
     function       Write(const Buffer; Count: Longint): Longint; override;
     function       Seek(Offset: Longint; Origin: Word): Longint; override;
     property       Size: Integer read GetSize write SetSize;
  end;

type
  TReaderWriter     =  class(TObject)
  private
     // Private declarations
     FFileName:     String;
     FReader:       TFileStreamEx;
     FWriter:       TFileStreamEx;
  protected
     // Protected declarations
  public
     // Public declarations
     constructor    Create;
     destructor     Destroy; override;
     property       Reader: TFileStreamEx read FReader;
     property       Writer: TFileStreamEx read FWriter;
  end;

implementation

// TReaderWriter
constructor TReaderWriter.Create;
var  lpszPath:   Array [0..MAX_PATH] of Char;
     lpszFile:   Array [0..MAX_PATH] of Char;
begin

  // Create temp file for stream backing
  if (GetTempPath(MAX_PATH, lpszPath) > 0) then
  begin
     if (GetTempFileName(lpszPath, 'mfs', 0, lpszFile) <> 0) then
     begin
        // Save FileName
        FFileName:=lpszFile;
        // Create reader and writer streams
        FReader:=nil;
        FWriter:=nil;
        FReader:=TFileStreamEx.Create(FFileName, fsmReader);
        FWriter:=TFileStreamEx.Create(FFileName, fsmWriter);
     end
     else
        // Failed to create temp file
        RaiseLastWin32Error;
  end
  else
     // Failed to get temp path
     RaiseLastWin32Error;

end;

destructor TReaderWriter.Destroy;
begin

  // Free the reader and writer
  FreeAndNil(FReader);
  FreeAndNil(FWriter);

  // Now delete the temp file backing
  DeleteFile(FFileName);

   // Perform inherited
  inherited Destroy;

end;

// TFileStreamEx
function TFileStreamEx.GetSize: Integer;
begin

  // Get file size directly
  result:=GetFileSize(FHandle, nil);

end;

procedure TFileStreamEx.SetSize(NewSize: Longint);
begin

  // Only writer can do this
  if (FMode = fsmReader) then raise EFileStreamEx.CreateRes(@resInvalidMode);

  // Seek to new position and set end of file
  Seek(NewSize, soFromBeginning);
  Win32Check(SetEndOfFile(FHandle));

end;

function TFileStreamEx.Read(var Buffer; Count: Longint): Longint;
begin

  // Only reader can do this
  if (FMode = fsmWriter) then raise EFileStreamEx.CreateRes(@resInvalidMode);

  // Perform the file read
  result:=FileRead(FHandle, Buffer, Count);
  if (result = -1) then result:=0;

end;

function TFileStreamEx.Write(const Buffer; Count: Longint): Longint;
begin

  // Only writer can do this
  if (FMode = fsmReader) then raise EFileStreamEx.CreateRes(@resInvalidMode);

  // Perform the file write
  result:=FileWrite(FHandle, Buffer, Count);
  if (result = -1) then result:=0;

end;

function TFileStreamEx.Seek(Offset: Longint; Origin: Word): Longint;
begin

  // Seek to new location
  result:=FileSeek(FHandle, Offset, Origin);

end;

constructor TFileStreamEx.Create(FileName: String; Mode: TFileStreamMode);
begin

  // Perform inherited
  inherited Create;

  // Open the file with the correct mode
  FMode:=Mode;
  if (FMode = fsmReader) then
     // Open reader handle
     FHandle:=FileOpen(FileName, fmOpenRead or fmShareDenyNone)
  else
     // Open writer handle
     FHandle:=FileOpen(FileName, fmOpenWrite or fmShareDenyWrite);

  // Check the handle
  if (FHandle < 0) then raise EFileStreamEx.CreateResFmt(@SFOpenError, [FileName]);

end;

destructor TFileStreamEx.Destroy;
begin

  // Close the file handle
  if (FHandle >= 0) then FileClose(FHandle);

  // Perform inherited
  inherited Destroy;

end;

end.

0
 
LVL 6

Author Comment

by:rbohac
ID: 10691529
I have to head out now. I'll check this out Tuesday. Thanks again for the help. Have a good weekend.
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10691559
No problem, have a good one ;-)

Russell
0
 
LVL 6

Author Comment

by:rbohac
ID: 10732360
Works perfectly! Thanks again Russell. You always go the extra mile!
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 10732373
You are more than welcome....

Glad I could help,
Russell
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Delphi application Soap connection 5 135
Delphi Form ownership 4 127
Need Help Delphi 2010 CheckBox1 Stored value in memo 13 82
Delphi Firemonkey : user drawing in window 3 29
Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question