Alternative to TMemoryStream

TFileStream

Shane

TFileStream would let you truncate/expand the file (using SetFilePointer, SetEndOfFile), thus fulfilling one requirement.

Accessing data directly could also be done by creating a child class of TFileStream and exposing an indexed property called

Bytes[Index: Integer]: Byte read GetByte write SetByte;

This would allow you to read in a small block of the file to memory, allow you to manipulate it (and have some cached to speed up sequential access), and also allow you to save any changes should the bytes get modified. Not exactly like direct memory manipulation, but close.

Anyways, if something like this would work for you, I can put it together...

Regards,
Russell

Thanks RLibby, Your like the third person who has stepped on me today

Shane

But hey go ahead, its all yours....

Shane

That would be an interesting way to go.  How would you create the child class?

Shane,
I don't see how offering other suggestions/additions/comments/etc can be considered "stepping" on someone.... but sorry you feel that way. I was only trying to elaborate on your ONE word comment.

rbohac, I will put together a sample object that demos some of the ideas I had in mind.

-----

Russell

technically two :)

Thanks Russell

My one word comment was waiting for him to respond if he wanted to go that route, so yes its stepping on it, but het like i said, its all yours RUSSELL

Shane

Shane,

Im not going to get into a shouting match with you, but your "commenting" to a question does not make it "yours" exclusively. This is a collaborative forum, and we have ALL had to share. If you are willing to put the time into a TFileStream descendant... then by all means, do so.

But TFileStream as it stands would not cut it (it does not meet the asker's requirements).

Regards,
Russell

Im not going to give you the satisfaction of a shouting match russell, i said its all yours good luck... end of subject!

Shane

Russell, I hope you'll still help me out...

BTW. I looked at TFileStream anyway before I posted. I actual posted this Q because it doesn't have memory access like the TMemoryStream

Still here, regardless of othe comments....

Here is my first cut at this...
First, I should say, that if your checking a block of mem, the quickest way to do this is to perform a Stream.Read(...). This will perform the read as a single block, and evaluation can be done on the data quickly (as it is in memory).

This class does allow for direct byte manipulation, and is optimized for sequential access (both r/w). Please note though, due to file IO, it will still not be as fast as direct memory access. I am still looking at further optimizations for this..

Regards,
Russell

---------
unit MemFileStream;

interface

uses
  Windows, SysUtils, Classes;

type
  TMemFileStream =  class(TFileStream)
  private
     // Protected declarations
     FSize:      Integer;
     FPosition:  Integer;
     FFileName:  String;
  protected
     // Protected declaration
     function    GetByte(Index: Integer): Byte;
     procedure   SetByte(Index: Integer; Value: Byte);
     procedure   SetSize(NewSize: Longint); override;
  public
     // Public declarations
     constructor Create;
     destructor  Destroy; override;
     function    Read(var Buffer; Count: Longint): Longint; override;
     function    Write(const Buffer; Count: Longint): Longint; override;
     function    Seek(Offset: Longint; Origin: Word): Longint; override;
     property    Bytes[Index: Integer]: Byte read GetByte write SetByte;
  end;

implementation

function TMemFileStream.Read(var Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Read(Buffer, Count);

  // Update position
  FPosition:=Position;

end;

function TMemFileStream.Write(const Buffer; Count: Longint): Longint;
begin

  // Perform inherited
  result:=inherited Write(Buffer, Count);

  // Update position and size
  FSize:=Size;
  FPosition:=Position;

end;

function TMemFileStream.Seek(Offset: Longint; Origin: Word): Longint;
begin

  // Update position
  FPosition:=inherited Seek(Offset, Origin);

  // Return position
  result:=FPosition;

end;

procedure TMemFileStream.SetSize(NewSize: Longint);
begin

  // Set new size and position
  FSize:=Seek(NewSize, soFromBeginning);
  FPosition:=FSize;
  Win32Check(SetEndOfFile(Handle));

end;

function TMemFileStream.GetByte(Index: Integer): Byte;
var  dwSize:     Integer;
begin

  // Is the read outside of size?
  if (Index >= FSize) then
  begin
     SetSize(Succ(Index));
     FPosition:=Succ(Index);
     result:=0;
  end
  else
  begin
     // Position change?
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     FileRead(Handle, result, SizeOf(Char));
     Inc(FPosition);
  end;

end;

procedure TMemFileStream.SetByte(Index: Integer; Value: Byte);
var  dwSize:     Integer;
begin

  // Check index against size
  if (Index > FSize) then
  begin
     // Expand the file
     SetSize(Index);
     // Set new file position
     Position:=Index;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FSize);
     Inc(FPosition);
  end
  else
  begin
     // Check write position
     if (FPosition <> Index) then
     begin
        Position:=Index;
        FPosition:=Index;
     end;
     // Write the byte
     FileWrite(Handle, Value, SizeOf(Char));
     Inc(FPosition);
  end;

end;

constructor TMemFileStream.Create;
var  lpszPath:   Array [0..MAX_PATH] of Char;
     lpszFile:   Array [0..MAX_PATH] of Char;
begin

  // Create temp file for stream backing
  if (GetTempPath(MAX_PATH, lpszPath) > 0) then
  begin
     if (GetTempFileName(lpszPath, 'mfs', 0, lpszFile) <> 0) then
     begin
        // Save FileName
        FFileName:=lpszFile;
        // Perform inherited
        inherited Create(FFileName, fmOpenReadWrite or fmShareDenyWrite);
        // Set defaults
        FSize:=0;
        FPosition:=0;
     end
     else
        // Failed to create temp file
        RaiseLastWin32Error;
  end
  else
     // Failed to get temp path
     RaiseLastWin32Error;

end;

destructor TMemFileStream.Destroy;
begin

  // Close the file by calling inherited
  inherited Destroy;

  // Now delete the temp file backing
  DeleteFile(FFileName);

end;

end.

Great! I'll start adding this in today although I probably won't be able to do much testing until Tuesday. I'm leaving town shortly.

Have a good weekened then, and let me know if you run into problems next week

Russell

I just thought of something. The reason I was using the MemoryStream.Memory was to that I could read from the memory stream without changing the position. This is because while I am reading the data (and laying back the video) I am also writing the data to the stream as it downloads.

Any ideas?

The read and writes are overriden, so you could save an internal reader position (seperate from the Position property), and perform something along the lines of:

hold = seek(...)
read(...)
seek(... hold ...)

Allowing you to maintain your write position. It would help (in optimizing) if i could see some real world (code) usage for what you currently have as well.

This help any?
Russell

There is too much code to throw on here.. plus I have to clean it up from my many attempts to do this.

Basically I have two threads. What happens is..

 - A thread that downloads the part of the data to a buffer in that thread
- the thread then calls a meth via synchronize that writes the data to the TMemoryStream
- It continues to do that until the file is completely downloaded

While this is happening I am running a loop in the main thread that parses that data
- It looks at the last position if read from and waits until the TMemoryStream has enough data to continue reading from
- It then reads a portion of that memory (via move) without updating the position and parses it accordingly

I wonder how much of a performance loss I will have by continually moving the position back and forth like that in the TMemFileStream

Okay, I'm getting to see the picture here....

Actually, the hit should be minimal, and here is why.
-------
The file pointer will always be kept at the end, so the next write appends the data to the end of the stream. But, internally (in the object class), you can keep a read position indicator. When there is enough data to perform the read (by checking the read location against size or whatever you need to check), then the file position is marked, the seek is made, data is read into a block of memory, and the file pointer is reset to its last position, which just happens to be the end of file.

So the only time the file position is changed is when a block read occurs.

Another possibilty as well is this (I will need to do some testing):, open up 2 handles to the same file, one for reading, the other writing.

Read Mode  = fmOpenRead or fmShareDenyNone;
Write Mode  = fmOpenWrite or fmShareDenyWrite;

This way, each operation has its own file pointer, and no seeking needs to be done. IMHO, this would be the optimal approach for a reader/writer sharing the same medium.

----------
Russell

I like that second approach, however I thought I read somewhere that that causes problems if you resize the file after it has been opened

I have to head out now. I'll check this out Tuesday. Thanks again for the help. Have a good weekend.

No problem, have a good one ;-)

Russell

Works perfectly! Thanks again Russell. You always go the extra mile!

You are more than welcome....

Glad I could help,
Russell