Solved

Alternative to TMemoryStream

Posted on 2004-03-25
24
505 Views
Last Modified: 2010-04-05
I've written a component that uses a TMemoryStream to buffer data. I'm running into a problem on some older machines (with 64 megs of RAM) that the stream gets to large for the system (50 megs of so).

Is there an alternative to TMemoryStream that I can use the physical disk space as my memory instead. I need to be able to size on the fly like TMemoryStream and also access the memory directly like TMemoryStream.memory.


Any ideas?


P.S. I wish I could simply make them upgrade the RAM, but I can't. The buffer is for video playback and I keep the video in memory after its played to allow Rewind and Fast Forward.
0
Comment
Question by:rbohac
  • 10
  • 9
  • 5
24 Comments
 
LVL 11

Expert Comment

by:shaneholmes
Comment Utility
TFileStream

Shane
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

TFileStream would let you truncate/expand the file (using SetFilePointer, SetEndOfFile), thus fulfilling one requirement.

Accessing data directly could also be done by creating a child class of TFileStream and exposing an indexed property called

Bytes[Index: Integer]: Byte read GetByte write SetByte;

This would allow you to read in a small block of the file to memory, allow you to manipulate it (and have some cached to speed up sequential access), and also allow you to save any changes should the bytes get modified. Not exactly like direct memory manipulation, but close.

Anyways, if something like this would work for you, I can put it together...

Regards,
Russell


0
 
LVL 11

Expert Comment

by:shaneholmes
Comment Utility
Thanks RLibby, Your like the third person who has stepped on me today

Shane
0
 
LVL 11

Expert Comment

by:shaneholmes
Comment Utility
But hey go ahead, its all yours....

Shane
0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
That would be an interesting way to go.  How would you create the child class?
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

Shane,
I don't see how offering other suggestions/additions/comments/etc can be considered "stepping" on someone.... but sorry you feel that way. I was only trying to elaborate on your ONE word comment.

rbohac, I will put together a sample object that demos some of the ideas I had in mind.

-----

Russell


0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
technically two :)

Thanks Russell
0
 
LVL 11

Expert Comment

by:shaneholmes
Comment Utility
My one word comment was waiting for him to respond if he wanted to go that route, so yes its stepping on it, but het like i said, its all yours RUSSELL

Shane
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility
Shane,

Im not going to get into a shouting match with you, but your "commenting" to a question does not make it "yours" exclusively. This is a collaborative forum, and we have ALL had to share. If you are willing to put the time into a TFileStream descendant... then by all means, do so.

But TFileStream as it stands would not cut it (it does not meet the asker's requirements).

Regards,
Russell
0
 
LVL 11

Expert Comment

by:shaneholmes
Comment Utility
Im not going to give you the satisfaction of a shouting match russell, i said its all yours good luck... end of subject!

Shane

0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
Russell, I hope you'll still help me out...

BTW. I looked at TFileStream anyway before I posted. I actual posted this Q because it doesn't have memory access like the TMemoryStream
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility
Still here, regardless of othe comments....

Here is my first cut at this...
First, I should say, that if your checking a block of mem, the quickest way to do this is to perform a Stream.Read(...). This will perform the read as a single block, and evaluation can be done on the data quickly (as it is in memory).

This class does allow for direct byte manipulation, and is optimized for sequential access (both r/w). Please note though, due to file IO, it will still not be as fast as direct memory access. I am still looking at further optimizations for this..

Regards,
Russell

---------
unit MemFileStream;

interface

uses
  Windows, SysUtils, Classes;

type
  TMemFileStream =  class(TFileStream)
  private
     // Protected declarations
     FSize:      Integer;
     FPosition:  Integer;
     FFileName:  String;
  protected
     // Protected declaration
     function    GetByte(Index: Integer): Byte;
     procedure   SetByte(Index: Integer; Value: Byte);
     procedure   SetSize(NewSize: Longint); override;
  public
     // Public declarations
     constructor Create;
     destructor  Destroy; override;
     function    Read(var Buffer; Count: Longint): Longint; override;
     function    Write(const Buffer; Count: Longint): Longint; override;
     function    Seek(Offset: Longint; Origin: Word): Longint; override;
     property    Bytes[Index: Integer]: Byte read GetByte write SetByte;
  end;

implementation

function TMemFileStream.Read(var Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Read(Buffer, Count);

  // Update position
  FPosition:=Position;

end;

function TMemFileStream.Write(const Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Write(Buffer, Count);

  // Update position and size
  FSize:=Size;
  FPosition:=Position;

end;

function TMemFileStream.Seek(Offset: Longint; Origin: Word): Longint;
begin

  // Update position
  FPosition:=inherited Seek(Offset, Origin);

  // Return position
  result:=FPosition;

end;

procedure TMemFileStream.SetSize(NewSize: Longint);
begin

  // Set new size and position
  FSize:=Seek(NewSize, soFromBeginning);
  FPosition:=FSize;
  Win32Check(SetEndOfFile(Handle));

end;

function TMemFileStream.GetByte(Index: Integer): Byte;
var  dwSize:     Integer;
begin

  // Is the read outside of size?
  if (Index >= FSize) then
  begin
     SetSize(Succ(Index));
     FPosition:=Succ(Index);
     result:=0;
  end
  else
  begin
     // Position change?
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     FileRead(Handle, result, SizeOf(Char));
     Inc(FPosition);
  end;

end;

procedure TMemFileStream.SetByte(Index: Integer; Value: Byte);
var  dwSize:     Integer;
begin

  // Check index against size
  if (Index > FSize) then
  begin
     // Expand the file
     SetSize(Index);
     // Set new file position
     Position:=Index;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FSize);
     Inc(FPosition);
  end
  else
  begin
     // Check write position
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FPosition);
  end;

end;

constructor TMemFileStream.Create;
var  lpszPath:   Array [0..MAX_PATH] of Char;
     lpszFile:   Array [0..MAX_PATH] of Char;
begin

  // Create temp file for stream backing
  if (GetTempPath(MAX_PATH, lpszPath) > 0) then
  begin
     if (GetTempFileName(lpszPath, 'mfs', 0, lpszFile) <> 0) then
     begin
        // Save FileName
        FFileName:=lpszFile;
        // Perform inherited
        inherited Create(FFileName, fmOpenReadWrite or fmShareDenyWrite);
        // Set defaults
        FSize:=0;
        FPosition:=0;
     end
     else
        // Failed to create temp file
        RaiseLastWin32Error;
  end
  else
     // Failed to get temp path
     RaiseLastWin32Error;

end;

destructor TMemFileStream.Destroy;
begin

  // Close the file by calling inherited
  inherited Destroy;

  // Now delete the temp file backing
  DeleteFile(FFileName);

end;

end.

0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 6

Author Comment

by:rbohac
Comment Utility
Great! I'll start adding this in today although I probably won't be able to do much testing until Tuesday. I'm leaving town shortly.
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

Have a good weekened then, and let me know if you run into problems next week

Russell
0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
I just thought of something. The reason I was using the MemoryStream.Memory was to that I could read from the memory stream without changing the position. This is because while I am reading the data (and laying back the video) I am also writing the data to the stream as it downloads.

Any ideas?
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility

The read and writes are overriden, so you could save an internal reader position (seperate from the Position property), and perform something along the lines of:

hold = seek(...)
read(...)
seek(... hold ...)

Allowing you to maintain your write position. It would help (in optimizing) if i could see some real world (code) usage for what you currently have as well.

This help any?
Russell
0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
There is too much code to throw on here.. plus I have to clean it up from my many attempts to do this.

Basically I have two threads. What happens is..

 - A thread that downloads the part of the data to a buffer in that thread
- the thread then calls a meth via synchronize that writes the data to the TMemoryStream
- It continues to do that until the file is completely downloaded

While this is happening I am running a loop in the main thread that parses that data
- It looks at the last position if read from and waits until the TMemoryStream has enough data to continue reading from
- It then reads a portion of that memory (via move) without updating the position and parses it accordingly

I wonder how much of a performance loss I will have by continually moving the position back and forth like that in the TMemFileStream
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility
Okay, I'm getting to see the picture here....

Actually, the hit should be minimal, and here is why.
-------
The file pointer will always be kept at the end, so the next write appends the data to the end of the stream. But, internally (in the object class), you can keep a read position indicator. When there is enough data to perform the read (by checking the read location against size or whatever you need to check), then the file position is marked, the seek is made, data is read into a block of memory, and the file pointer is reset to its last position, which just happens to be the end of file.

So the only time the file position is changed is when a block read occurs.

Another possibilty as well is this (I will need to do some testing):, open up 2 handles to the same file, one for reading, the other writing.

Read Mode  = fmOpenRead or fmShareDenyNone;
Write Mode  = fmOpenWrite or fmShareDenyWrite;

This way, each operation has its own file pointer, and no seeking needs to be done. IMHO, this would be the optimal approach for a reader/writer sharing the same medium.

----------
Russell
0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
I like that second approach, however I thought I read somewhere that that causes problems if you resize the file after it has been opened
0
 
LVL 26

Accepted Solution

by:
Russell Libby earned 250 total points
Comment Utility

Not that my testing shows. I guess in the following scenario it could cause a read problem: read pos is at 1000 (for example), and the writer trucates the file to 100 bytes. When the reader goes to read, it reads in zero bytes.

This should not be your case through.

This is the test that I ran to check file sizing:

var  test:       TReaderWriter;
begin

  test:=TReaderWriter.Create;
  Randomize;
  while not(Application.Terminated) do
  begin
     Application.ProcessMessages;
     test.Writer.Size:=Random(1000000);
     Assert(test.Reader.Size = test.Writer.Size, 'File sizes do not match');
  end;
  test.Free;

end;

And here is the source for the TReaderWriter class and the stream types it exposes. This should provide a straight forward solution to your problem.

Russell

-----------

unit ReaderWriter;

interface

uses
  Windows, SysUtils, Classes, Consts;

type
  EFileStreamEx     =  class(Exception);

resourcestring
  resInvalidMode    =  'Invalid stream mode for this operation.';

type
  TFileStreamMode   =  (fsmReader, fsmWriter);
  TFileStreamEx     =  class(TStream)
  private
     // Private declarations
     FHandle:       Integer;
     FMode:         TFileStreamMode;
  protected
     // Protected declaration
     function       GetSize: Integer;
     procedure      SetSize(NewSize: Longint); override;
  public
     // Public declarations
     constructor    Create(FileName: String; Mode: TFileStreamMode);
     destructor     Destroy; override;
     function       Read(var Buffer; Count: Longint): Longint; override;
     function       Write(const Buffer; Count: Longint): Longint; override;
     function       Seek(Offset: Longint; Origin: Word): Longint; override;
     property       Size: Integer read GetSize write SetSize;
  end;

type
  TReaderWriter     =  class(TObject)
  private
     // Private declarations
     FFileName:     String;
     FReader:       TFileStreamEx;
     FWriter:       TFileStreamEx;
  protected
     // Protected declarations
  public
     // Public declarations
     constructor    Create;
     destructor     Destroy; override;
     property       Reader: TFileStreamEx read FReader;
     property       Writer: TFileStreamEx read FWriter;
  end;

implementation

// TReaderWriter
constructor TReaderWriter.Create;
var  lpszPath:   Array [0..MAX_PATH] of Char;
     lpszFile:   Array [0..MAX_PATH] of Char;
begin

  // Create temp file for stream backing
  if (GetTempPath(MAX_PATH, lpszPath) > 0) then
  begin
     if (GetTempFileName(lpszPath, 'mfs', 0, lpszFile) <> 0) then
     begin
        // Save FileName
        FFileName:=lpszFile;
        // Create reader and writer streams
        FReader:=nil;
        FWriter:=nil;
        FReader:=TFileStreamEx.Create(FFileName, fsmReader);
        FWriter:=TFileStreamEx.Create(FFileName, fsmWriter);
     end
     else
        // Failed to create temp file
        RaiseLastWin32Error;
  end
  else
     // Failed to get temp path
     RaiseLastWin32Error;

end;

destructor TReaderWriter.Destroy;
begin

  // Free the reader and writer
  FreeAndNil(FReader);
  FreeAndNil(FWriter);

  // Now delete the temp file backing
  DeleteFile(FFileName);

   // Perform inherited
  inherited Destroy;

end;

// TFileStreamEx
function TFileStreamEx.GetSize: Integer;
begin

  // Get file size directly
  result:=GetFileSize(FHandle, nil);

end;

procedure TFileStreamEx.SetSize(NewSize: Longint);
begin

  // Only writer can do this
  if (FMode = fsmReader) then raise EFileStreamEx.CreateRes(@resInvalidMode);

  // Seek to new position and set end of file
  Seek(NewSize, soFromBeginning);
  Win32Check(SetEndOfFile(FHandle));

end;

function TFileStreamEx.Read(var Buffer; Count: Longint): Longint;
begin

  // Only reader can do this
  if (FMode = fsmWriter) then raise EFileStreamEx.CreateRes(@resInvalidMode);

  // Perform the file read
  result:=FileRead(FHandle, Buffer, Count);
  if (result = -1) then result:=0;

end;

function TFileStreamEx.Write(const Buffer; Count: Longint): Longint;
begin

  // Only writer can do this
  if (FMode = fsmReader) then raise EFileStreamEx.CreateRes(@resInvalidMode);

  // Perform the file write
  result:=FileWrite(FHandle, Buffer, Count);
  if (result = -1) then result:=0;

end;

function TFileStreamEx.Seek(Offset: Longint; Origin: Word): Longint;
begin

  // Seek to new location
  result:=FileSeek(FHandle, Offset, Origin);

end;

constructor TFileStreamEx.Create(FileName: String; Mode: TFileStreamMode);
begin

  // Perform inherited
  inherited Create;

  // Open the file with the correct mode
  FMode:=Mode;
  if (FMode = fsmReader) then
     // Open reader handle
     FHandle:=FileOpen(FileName, fmOpenRead or fmShareDenyNone)
  else
     // Open writer handle
     FHandle:=FileOpen(FileName, fmOpenWrite or fmShareDenyWrite);

  // Check the handle
  if (FHandle < 0) then raise EFileStreamEx.CreateResFmt(@SFOpenError, [FileName]);

end;

destructor TFileStreamEx.Destroy;
begin

  // Close the file handle
  if (FHandle >= 0) then FileClose(FHandle);

  // Perform inherited
  inherited Destroy;

end;

end.

0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
I have to head out now. I'll check this out Tuesday. Thanks again for the help. Have a good weekend.
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility
No problem, have a good one ;-)

Russell
0
 
LVL 6

Author Comment

by:rbohac
Comment Utility
Works perfectly! Thanks again Russell. You always go the extra mile!
0
 
LVL 26

Expert Comment

by:Russell Libby
Comment Utility
You are more than welcome....

Glad I could help,
Russell
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now