We help IT Professionals succeed at work.

Check out our new AWS podcast with Certified Expert, Phil Phillips! Listen to "How to Execute a Seamless AWS Migration" on EE or on your favorite podcast platform. Listen Now

x

Alternative to TMemoryStream

rbohac
rbohac asked
on
Medium Priority
643 Views
Last Modified: 2010-04-05
I've written a component that uses a TMemoryStream to buffer data. I'm running into a problem on some older machines (with 64 megs of RAM) that the stream gets to large for the system (50 megs of so).

Is there an alternative to TMemoryStream that I can use the physical disk space as my memory instead. I need to be able to size on the fly like TMemoryStream and also access the memory directly like TMemoryStream.memory.


Any ideas?


P.S. I wish I could simply make them upgrade the RAM, but I can't. The buffer is for video playback and I keep the video in memory after its played to allow Rewind and Fast Forward.
Comment
Watch Question

TFileStream

Shane
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:

TFileStream would let you truncate/expand the file (using SetFilePointer, SetEndOfFile), thus fulfilling one requirement.

Accessing data directly could also be done by creating a child class of TFileStream and exposing an indexed property called

Bytes[Index: Integer]: Byte read GetByte write SetByte;

This would allow you to read in a small block of the file to memory, allow you to manipulate it (and have some cached to speed up sequential access), and also allow you to save any changes should the bytes get modified. Not exactly like direct memory manipulation, but close.

Anyways, if something like this would work for you, I can put it together...

Regards,
Russell


Thanks RLibby, Your like the third person who has stepped on me today

Shane
But hey go ahead, its all yours....

Shane

Author

Commented:
That would be an interesting way to go.  How would you create the child class?
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:

Shane,
I don't see how offering other suggestions/additions/comments/etc can be considered "stepping" on someone.... but sorry you feel that way. I was only trying to elaborate on your ONE word comment.

rbohac, I will put together a sample object that demos some of the ideas I had in mind.

-----

Russell


Author

Commented:
technically two :)

Thanks Russell
My one word comment was waiting for him to respond if he wanted to go that route, so yes its stepping on it, but het like i said, its all yours RUSSELL

Shane
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:
Shane,

Im not going to get into a shouting match with you, but your "commenting" to a question does not make it "yours" exclusively. This is a collaborative forum, and we have ALL had to share. If you are willing to put the time into a TFileStream descendant... then by all means, do so.

But TFileStream as it stands would not cut it (it does not meet the asker's requirements).

Regards,
Russell
Im not going to give you the satisfaction of a shouting match russell, i said its all yours good luck... end of subject!

Shane

Author

Commented:
Russell, I hope you'll still help me out...

BTW. I looked at TFileStream anyway before I posted. I actual posted this Q because it doesn't have memory access like the TMemoryStream
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:
Still here, regardless of othe comments....

Here is my first cut at this...
First, I should say, that if your checking a block of mem, the quickest way to do this is to perform a Stream.Read(...). This will perform the read as a single block, and evaluation can be done on the data quickly (as it is in memory).

This class does allow for direct byte manipulation, and is optimized for sequential access (both r/w). Please note though, due to file IO, it will still not be as fast as direct memory access. I am still looking at further optimizations for this..

Regards,
Russell

---------
unit MemFileStream;

interface

uses
  Windows, SysUtils, Classes;

type
  TMemFileStream =  class(TFileStream)
  private
     // Protected declarations
     FSize:      Integer;
     FPosition:  Integer;
     FFileName:  String;
  protected
     // Protected declaration
     function    GetByte(Index: Integer): Byte;
     procedure   SetByte(Index: Integer; Value: Byte);
     procedure   SetSize(NewSize: Longint); override;
  public
     // Public declarations
     constructor Create;
     destructor  Destroy; override;
     function    Read(var Buffer; Count: Longint): Longint; override;
     function    Write(const Buffer; Count: Longint): Longint; override;
     function    Seek(Offset: Longint; Origin: Word): Longint; override;
     property    Bytes[Index: Integer]: Byte read GetByte write SetByte;
  end;

implementation

function TMemFileStream.Read(var Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Read(Buffer, Count);

  // Update position
  FPosition:=Position;

end;

function TMemFileStream.Write(const Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Write(Buffer, Count);

  // Update position and size
  FSize:=Size;
  FPosition:=Position;

end;

function TMemFileStream.Seek(Offset: Longint; Origin: Word): Longint;
begin

  // Update position
  FPosition:=inherited Seek(Offset, Origin);

  // Return position
  result:=FPosition;

end;

procedure TMemFileStream.SetSize(NewSize: Longint);
begin

  // Set new size and position
  FSize:=Seek(NewSize, soFromBeginning);
  FPosition:=FSize;
  Win32Check(SetEndOfFile(Handle));

end;

function TMemFileStream.GetByte(Index: Integer): Byte;
var  dwSize:     Integer;
begin

  // Is the read outside of size?
  if (Index >= FSize) then
  begin
     SetSize(Succ(Index));
     FPosition:=Succ(Index);
     result:=0;
  end
  else
  begin
     // Position change?
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     FileRead(Handle, result, SizeOf(Char));
     Inc(FPosition);
  end;

end;

procedure TMemFileStream.SetByte(Index: Integer; Value: Byte);
var  dwSize:     Integer;
begin

  // Check index against size
  if (Index > FSize) then
  begin
     // Expand the file
     SetSize(Index);
     // Set new file position
     Position:=Index;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FSize);
     Inc(FPosition);
  end
  else
  begin
     // Check write position
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FPosition);
  end;

end;

constructor TMemFileStream.Create;
var  lpszPath:   Array [0..MAX_PATH] of Char;
     lpszFile:   Array [0..MAX_PATH] of Char;
begin

  // Create temp file for stream backing
  if (GetTempPath(MAX_PATH, lpszPath) > 0) then
  begin
     if (GetTempFileName(lpszPath, 'mfs', 0, lpszFile) <> 0) then
     begin
        // Save FileName
        FFileName:=lpszFile;
        // Perform inherited
        inherited Create(FFileName, fmOpenReadWrite or fmShareDenyWrite);
        // Set defaults
        FSize:=0;
        FPosition:=0;
     end
     else
        // Failed to create temp file
        RaiseLastWin32Error;
  end
  else
     // Failed to get temp path
     RaiseLastWin32Error;

end;

destructor TMemFileStream.Destroy;
begin

  // Close the file by calling inherited
  inherited Destroy;

  // Now delete the temp file backing
  DeleteFile(FFileName);

end;

end.

Author

Commented:
Great! I'll start adding this in today although I probably won't be able to do much testing until Tuesday. I'm leaving town shortly.
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:

Have a good weekened then, and let me know if you run into problems next week

Russell

Author

Commented:
I just thought of something. The reason I was using the MemoryStream.Memory was to that I could read from the memory stream without changing the position. This is because while I am reading the data (and laying back the video) I am also writing the data to the stream as it downloads.

Any ideas?
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:

The read and writes are overriden, so you could save an internal reader position (seperate from the Position property), and perform something along the lines of:

hold = seek(...)
read(...)
seek(... hold ...)

Allowing you to maintain your write position. It would help (in optimizing) if i could see some real world (code) usage for what you currently have as well.

This help any?
Russell

Author

Commented:
There is too much code to throw on here.. plus I have to clean it up from my many attempts to do this.

Basically I have two threads. What happens is..

 - A thread that downloads the part of the data to a buffer in that thread
- the thread then calls a meth via synchronize that writes the data to the TMemoryStream
- It continues to do that until the file is completely downloaded

While this is happening I am running a loop in the main thread that parses that data
- It looks at the last position if read from and waits until the TMemoryStream has enough data to continue reading from
- It then reads a portion of that memory (via move) without updating the position and parses it accordingly

I wonder how much of a performance loss I will have by continually moving the position back and forth like that in the TMemFileStream
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:
Okay, I'm getting to see the picture here....

Actually, the hit should be minimal, and here is why.
-------
The file pointer will always be kept at the end, so the next write appends the data to the end of the stream. But, internally (in the object class), you can keep a read position indicator. When there is enough data to perform the read (by checking the read location against size or whatever you need to check), then the file position is marked, the seek is made, data is read into a block of memory, and the file pointer is reset to its last position, which just happens to be the end of file.

So the only time the file position is changed is when a block read occurs.

Another possibilty as well is this (I will need to do some testing):, open up 2 handles to the same file, one for reading, the other writing.

Read Mode  = fmOpenRead or fmShareDenyNone;
Write Mode  = fmOpenWrite or fmShareDenyWrite;

This way, each operation has its own file pointer, and no seeking needs to be done. IMHO, this would be the optimal approach for a reader/writer sharing the same medium.

----------
Russell

Author

Commented:
I like that second approach, however I thought I read somewhere that that causes problems if you resize the file after it has been opened
Software Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005
Commented:
Unlock this solution with a free trial preview.
(No credit card required)
Get Preview

Author

Commented:
I have to head out now. I'll check this out Tuesday. Thanks again for the help. Have a good weekend.
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:
No problem, have a good one ;-)

Russell

Author

Commented:
Works perfectly! Thanks again Russell. You always go the extra mile!
Russell LibbySoftware Engineer, Advisory
CERTIFIED EXPERT
Top Expert 2005

Commented:
You are more than welcome....

Glad I could help,
Russell
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a free trial preview!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.