How to use TStream to read a binary file containing both integers and Cstrings?

Posted on 2004-11-03
Last Modified: 2010-04-05
I am trying to determine how to use TStream or descendents to read a binary file containing a mixture of integers, words, and null-terminated strings (Cstrings).

I can read any fixed size objects with the Tstream ReadBuffer method. This requires the object size known in advance, to provide as a parameter. TStringStreams support string types only, and has no means to read integers.

What is the easiest/best way to read both variable-size, null-terminated strings and integer values in an arbitrary mix, from the same stream?

Is there any approach better than brute force? (IE, creating a function that reads one byte at a time from the stream, copying them into a string, and ending on a zero byte)

Question by:tgodfrey
    LVL 26

    Expert Comment

    by:Russell Libby

    If the storage is not in any predefined format (eg a record structure), then a common technique is to use a byte type identifier that indicates the data that is to follow.

    #1 = Word data
    [01 00 02 ...]

    #2 = Integer data
    [02 00 00 00 03 ...]

    #3 = String data, with additional byte (or word, etc) indicating length of string plus null(eg "file")
    [03 05  66 69 6C 65 00 ...]

    This would allow you to read data in an efficient manner, and allow you to get string data in 3 read (data type, data size, data), vs reading byte by byte.



    Author Comment

    To clarify, I do not control the format of the binary file. It is an existing file. I know the format, but my program does not write it, and I don't have the ability to change the program that generates the binary file.
    LVL 26

    Accepted Solution


    My mistake; partially....

    I thought you were talking about writing the data out / reading the data in (full control). Being that you have zero control over what is there, then you are left with few options.

    One option you MAY have; again, I don't have all the details and can't say if this is feasible for you, as this is dependant on file size...  is to load the file using a TMemoryStream. This would allow you to cast the Memory property (generic pointer) to a PChar data type, which would then allow fast access (plus memory inc/dec/add/subtract) to the data.

    For example, reading a word, integer, variable length nts string (null terminated), then word from a binary stream.

    var  strmMemory:    TMemoryStream;
         lpMemory:      PChar;
         lpszData:      PChar;
         wData:         Word;
         dwData:        Integer;

      // Create memory stream
         // Load from file
         strmMemory.LoadFromFile(FileName); // Replace FileName with your file's name
         // Get start
         // Check assignment
         if Assigned(lpMemory) then

            // NOTE: for simplicity, there is no stream size checking in this example. For safety,
            // final code should have data size / stream size checking in place to make sure
            // memory violations do not occur.

            // Get word
            MoveMemory(@wData, lpMemory, SizeOf(Word));
            Inc(lpMemory, SizeOf(Word));

            // Get integer
            MoveMemory(@dwData, lpMemory, SizeOf(Integer));
            Inc(lpMemory, SizeOf(Integer));

            // Get pchar (trivial, due to null termination)
            // Move to end of string data
            // Skip null

            // Get word
            MoveMemory(@wData, lpMemory, SizeOf(Word));
            Inc(lpMemory, SizeOf(Word));

            // Etc...

         // Free stream




    LVL 26

    Expert Comment

    by:Russell Libby

    I should have clarified the term "feasible" in the above comment.... By that I meant that if this file will always be less than a few MB of data (or whatever you consider reasonable to load directly into memory), then there should be no issues loading it directly into memory and accessing the data using a memory stream.

    Now, if there is a chance that the file is larger than the user's (or yours) total "physical" memory, then loading it into memory is not really viable for you. At that point you will be stuck block reading the data and handling it that way. Eg, using a file stream, etc...


    Author Comment

    Thanks Russell. I hadn't thought of accessing the file directly using a pointer to the in-memory version of the stream, and it works for the particular case I have where files are only a few Mbytes. As you note, you have add a lot of checks for the "unsafe" memory access, but performance of  MoveMemory for long strings would be better than byte-at-a-time Tstream.ReadBuffer calls.


    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
    In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
    It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
    Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

    761 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    9 Experts available now in Live!

    Get 1:1 Help Now