How to use TStream to read a binary file containing both integers and Cstrings?

I am trying to determine how to use TStream or descendents to read a binary file containing a mixture of integers, words, and null-terminated strings (Cstrings).

I can read any fixed size objects with the Tstream ReadBuffer method. This requires the object size known in advance, to provide as a parameter. TStringStreams support string types only, and has no means to read integers.

What is the easiest/best way to read both variable-size, null-terminated strings and integer values in an arbitrary mix, from the same stream?

Is there any approach better than brute force? (IE, creating a function that reads one byte at a time from the stream, copying them into a string, and ending on a zero byte)

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Russell LibbySoftware Engineer, Advisory Commented:

If the storage is not in any predefined format (eg a record structure), then a common technique is to use a byte type identifier that indicates the data that is to follow.

#1 = Word data
[01 00 02 ...]

#2 = Integer data
[02 00 00 00 03 ...]

#3 = String data, with additional byte (or word, etc) indicating length of string plus null(eg "file")
[03 05  66 69 6C 65 00 ...]

This would allow you to read data in an efficient manner, and allow you to get string data in 3 read (data type, data size, data), vs reading byte by byte.


tgodfreyAuthor Commented:
To clarify, I do not control the format of the binary file. It is an existing file. I know the format, but my program does not write it, and I don't have the ability to change the program that generates the binary file.
Russell LibbySoftware Engineer, Advisory Commented:

My mistake; partially....

I thought you were talking about writing the data out / reading the data in (full control). Being that you have zero control over what is there, then you are left with few options.

One option you MAY have; again, I don't have all the details and can't say if this is feasible for you, as this is dependant on file size...  is to load the file using a TMemoryStream. This would allow you to cast the Memory property (generic pointer) to a PChar data type, which would then allow fast access (plus memory inc/dec/add/subtract) to the data.

For example, reading a word, integer, variable length nts string (null terminated), then word from a binary stream.

var  strmMemory:    TMemoryStream;
     lpMemory:      PChar;
     lpszData:      PChar;
     wData:         Word;
     dwData:        Integer;

  // Create memory stream
     // Load from file
     strmMemory.LoadFromFile(FileName); // Replace FileName with your file's name
     // Get start
     // Check assignment
     if Assigned(lpMemory) then

        // NOTE: for simplicity, there is no stream size checking in this example. For safety,
        // final code should have data size / stream size checking in place to make sure
        // memory violations do not occur.

        // Get word
        MoveMemory(@wData, lpMemory, SizeOf(Word));
        Inc(lpMemory, SizeOf(Word));

        // Get integer
        MoveMemory(@dwData, lpMemory, SizeOf(Integer));
        Inc(lpMemory, SizeOf(Integer));

        // Get pchar (trivial, due to null termination)
        // Move to end of string data
        // Skip null

        // Get word
        MoveMemory(@wData, lpMemory, SizeOf(Word));
        Inc(lpMemory, SizeOf(Word));

        // Etc...

     // Free stream





Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Russell LibbySoftware Engineer, Advisory Commented:

I should have clarified the term "feasible" in the above comment.... By that I meant that if this file will always be less than a few MB of data (or whatever you consider reasonable to load directly into memory), then there should be no issues loading it directly into memory and accessing the data using a memory stream.

Now, if there is a chance that the file is larger than the user's (or yours) total "physical" memory, then loading it into memory is not really viable for you. At that point you will be stuck block reading the data and handling it that way. Eg, using a file stream, etc...

tgodfreyAuthor Commented:
Thanks Russell. I hadn't thought of accessing the file directly using a pointer to the in-memory version of the stream, and it works for the particular case I have where files are only a few Mbytes. As you note, you have add a lot of checks for the "unsafe" memory access, but performance of  MoveMemory for long strings would be better than byte-at-a-time Tstream.ReadBuffer calls.

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.