Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 322
  • Last Modified:

How to use TStream to read a binary file containing both integers and Cstrings?

I am trying to determine how to use TStream or descendents to read a binary file containing a mixture of integers, words, and null-terminated strings (Cstrings).

I can read any fixed size objects with the Tstream ReadBuffer method. This requires the object size known in advance, to provide as a parameter. TStringStreams support string types only, and has no means to read integers.

What is the easiest/best way to read both variable-size, null-terminated strings and integer values in an arbitrary mix, from the same stream?

Is there any approach better than brute force? (IE, creating a function that reads one byte at a time from the stream, copying them into a string, and ending on a zero byte)

0
tgodfrey
Asked:
tgodfrey
  • 3
  • 2
1 Solution
 
Russell LibbySoftware Engineer, Advisory Commented:

If the storage is not in any predefined format (eg a record structure), then a common technique is to use a byte type identifier that indicates the data that is to follow.

eg
#1 = Word data
[01 00 02 ...]

#2 = Integer data
[02 00 00 00 03 ...]

#3 = String data, with additional byte (or word, etc) indicating length of string plus null(eg "file")
[03 05  66 69 6C 65 00 ...]

This would allow you to read data in an efficient manner, and allow you to get string data in 3 read (data type, data size, data), vs reading byte by byte.

Regards,
Russell

0
 
tgodfreyAuthor Commented:
To clarify, I do not control the format of the binary file. It is an existing file. I know the format, but my program does not write it, and I don't have the ability to change the program that generates the binary file.
0
 
Russell LibbySoftware Engineer, Advisory Commented:

My mistake; partially....

I thought you were talking about writing the data out / reading the data in (full control). Being that you have zero control over what is there, then you are left with few options.

One option you MAY have; again, I don't have all the details and can't say if this is feasible for you, as this is dependant on file size...  is to load the file using a TMemoryStream. This would allow you to cast the Memory property (generic pointer) to a PChar data type, which would then allow fast access (plus memory inc/dec/add/subtract) to the data.

For example, reading a word, integer, variable length nts string (null terminated), then word from a binary stream.

var  strmMemory:    TMemoryStream;
     lpMemory:      PChar;
     lpszData:      PChar;
     wData:         Word;
     dwData:        Integer;
begin

  // Create memory stream
  strmMemory:=TMemoryStream.Create;
  try
     // Load from file
     strmMemory.LoadFromFile(FileName); // Replace FileName with your file's name
     // Get start
     lpMemory:=strmMemory.Memory;
     // Check assignment
     if Assigned(lpMemory) then
     begin

        // NOTE: for simplicity, there is no stream size checking in this example. For safety,
        // final code should have data size / stream size checking in place to make sure
        // memory violations do not occur.

        // Get word
        MoveMemory(@wData, lpMemory, SizeOf(Word));
        Inc(lpMemory, SizeOf(Word));

        // Get integer
        MoveMemory(@dwData, lpMemory, SizeOf(Integer));
        Inc(lpMemory, SizeOf(Integer));

        // Get pchar (trivial, due to null termination)
        lpszData:=lpMemory;
        // Move to end of string data
        lpMemory:=StrEnd(lpMemory);
        // Skip null
        Inc(lpMemory);

        // Get word
        MoveMemory(@wData, lpMemory, SizeOf(Word));
        Inc(lpMemory, SizeOf(Word));

        //
        // Etc...
        //

     end;
  finally
     // Free stream
     strmMemory.Free;
  end;

end;

----

Regards,
Russell

0
 
Russell LibbySoftware Engineer, Advisory Commented:

I should have clarified the term "feasible" in the above comment.... By that I meant that if this file will always be less than a few MB of data (or whatever you consider reasonable to load directly into memory), then there should be no issues loading it directly into memory and accessing the data using a memory stream.

Now, if there is a chance that the file is larger than the user's (or yours) total "physical" memory, then loading it into memory is not really viable for you. At that point you will be stuck block reading the data and handling it that way. Eg, using a file stream, etc...

Russell
0
 
tgodfreyAuthor Commented:
Thanks Russell. I hadn't thought of accessing the file directly using a pointer to the in-memory version of the stream, and it works for the particular case I have where files are only a few Mbytes. As you note, you have add a lot of checks for the "unsafe" memory access, but performance of  MoveMemory for long strings would be better than byte-at-a-time Tstream.ReadBuffer calls.

0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now