Read binary file - efficiently

billcch,

Use memory mapped files and process 4 bytes at a time instead of 1.

b.

Umm.. note that the code I listed isn't exactly the same as your code.  The difference is your code skips over whitespaces (spaces, tabs, newline, etc..).  My code processes every byte.

For example let's say your input file contains a space (ascii code 0x20) and the letter A (code 0x41).  When you make the statement:

>> ifile >> c;

It skips over the space and gets 'A'.  Adds 128 to 'A' (0x41) to give 0xC1 and sends it to the output file.  So you have

Input file: {0x20, 0x41, 0x20}

Your codes output: {0xC1}

My codes output: {0xA0, 0xC1, 0xA0}

Not sure which you want but to change my code to do the same as your's just throw some if statements into the while loop.

b.

Oops..

Replace while loop with:

     lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
     while (lpInputPtr != lpInputEnd) {
          *lpOutput++ = *lpInputPtr++ | 0x80808080;
     }
     for (i = dwSizeLow & 0x03;i;i--) {
          *(char *)lpOutput = *(char *)lpInputPtr | 0x80;
          (char *)lpOutput += 1;
          (char *)lpInputPtr += 1;
     }

Regardless of how you decide to read from the file, processing 4 bytes at a time will be faster than processing one at a time.

Mercantilum,

>> It has been the object of a looong discussion in another thread

It may be helpful to the OP (and others who are participating) if you provide a link.

b.

Hmm regarding the link https://www.experts-exchange.com/questions/20949716/Efficient-way-to-get-a-string-contains-at-most-3000-chars.html

Mercantilum,

... I ran your code against my code in a profiler...

OS: WinXP
Compiler: VC++7
Profiler: Vtune
Test file size: 7 MB

                                                                 Time
Syscall open/read/write:                              664,740
Memory Mapped files:                                 116,951

Memory mapped files seems about 6 times faster... so I tried again but this time I dumbed down the code to process one byte at a time.

                                                                 Time
Syscall open/read/write:                              717,742
Memory Mapped files w/byte processing:      259,052

Still about 3 times faster.  I used your code as is.. just cut and paste.  My code is as is also except for a couple (oops) syntax corrections.  I placed your code in one function and my code in another function.  My function is called first so if there is any advantage gained from having buffered the previous file opening, your code would stand to benefit.

Thoughts?

b.

When there are r/w ian optimization can help.
Can you please provide your (working) code, I'd like to test on g++, thanks.

Mercantilum,

Hold on.. let me clean it up a bit..

b.

Mercantilum,

I forgot to mention all optimizations were off.

///////////////////////////////////////////////////////////////////////////////

// Uncomment for single byte processing as opposed to processing 4 bytes at a time
//#define SINGLE_BYTE_PROCESSING

int MMTest() {
      HANDLE hFile, hmFile; // handles for files and file mappings
      LPVOID lpInput, lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
      char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
      LPDWORD lpInputPtr, lpInputEnd, lpOutputPtr;
#endif
      DWORD dwSizeLow, dwSizeHigh;

      if ((hFile = CreateFile("E:\\test1.txt",GENERIC_READ,0,NULL,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,NULL)) == INVALID_HANDLE_VALUE)
            return 0;
      hmFile = CreateFileMapping(hFile,NULL,PAGE_READONLY,0,0,NULL);
      dwSizeLow = GetFileSize(hFile, &dwSizeHigh);
      CloseHandle(hFile);
      if (!hmFile) return 0;
      lpInput = (LPDWORD)MapViewOfFile (hmFile,FILE_MAP_READ,0,0,0);
      CloseHandle(hmFile);
      if (dwSizeHigh) return 0; // too big..
      if ((hFile = CreateFile("E:\\test2.txt",GENERIC_READ | GENERIC_WRITE,0,NULL,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,NULL)) == INVALID_HANDLE_VALUE)
            return 0;
      hmFile = CreateFileMapping(hFile,NULL,PAGE_READWRITE,dwSizeHigh,dwSizeLow,NULL);
      CloseHandle(hFile);
      if (!hmFile) return 0;
      lpOutput = MapViewOfFile (hmFile,FILE_MAP_ALL_ACCESS,0,0,0);
      CloseHandle(hmFile);

#ifdef SINGLE_BYTE_PROCESSING
      lpInputPtr = (char *)lpInput;
      lpInputEnd = (char *)lpInput + dwSizeLow;
      lpOutputPtr = (char *)lpOutput;
      while (lpInputPtr != lpInputEnd) {
            *lpOutputPtr++ = *lpInputPtr++ | 0x80;
      }
#else
      lpInputPtr = (LPDWORD)lpInput;
      lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
      lpOutputPtr = (LPDWORD)lpOutput;
      while (lpInputPtr != lpInputEnd) {
          *lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
     }
     for (int i = dwSizeLow & 0x03;i;i--) {
            *((char *)lpOutputPtr) = *(char *)lpInputPtr | 0x80;
            lpOutputPtr = (LPDWORD)((char *)lpOutputPtr + 1);
            lpInputPtr = (LPDWORD)((char *)lpInputPtr + 1);
     }
#endif

       UnmapViewOfFile (lpInput);
     UnmapViewOfFile (lpOutput);
       return 1;
}
///////////////////////////////////////////////////////////////////////////////

Oo, and I just ran the test with "Maximize Speed" optimization....

Maximize for Speed                                         Time
Syscall open/read/write:                              282,247
Memory Mapped files w/byte processing:            415

b.

Ok, just a question: in the timing, do you include the filemapping init etc... for Memory Mapped files w/byte processing ?
(since I guess it is what takes the most of the time, not the loop itself, done in memory :)

Mercantilum,

I posted the function that gets called from main.  The times posted are the total time spent in the function MMTest().  So that means opening, mapping and processing.

I cut and paste your code (opening, reading, writing, processing, and closing.. everything) and put it in a function called void SYSTest().  The times posted are for the total time spent in SYSTest().

b.

Do you mind posting the whole code, from line 1 to end, with includes, method for calc. time etc...
So that I can copy and paste as well and play :)

Mercantilum,

>> ..method for calc. time..

As I said earlier the timing is all done by the profiler.  It runs the executable and shows how much time was spent in each function, therefore, I have no code for calculating the time.  You'll have to write your own.

Anyway here is the 'whole code'.

////////////////////////////////////////////////////////////////
#include <fstream>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <io.h>
//#include <unistd.h>

#ifndef O_BINARY
#define O_BINARY  0
#endif
#define SIZE  1024

// Uncomment for single byte processing as opposed to processing 4 bytes at a time
#define SINGLE_BYTE_PROCESSING

int MMTest();
void SYSTest();

int main() {
      MMTest();
      SYSTest();
      return 0;
}

int MMTest() {
      HANDLE hFile, hmFile; // handles for files and file mappings
      LPVOID lpInput, lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
      char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
      LPDWORD lpInputPtr, lpInputEnd, lpOutputPtr;
#endif
      DWORD dwSizeLow, dwSizeHigh;

      if ((hFile = CreateFile("E:\\test1.txt",GENERIC_READ,0,NULL,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,NULL)) == INVALID_HANDLE_VALUE)
            return 0;
      hmFile = CreateFileMapping(hFile,NULL,PAGE_READONLY,0,0,NULL);
      dwSizeLow = GetFileSize(hFile, &dwSizeHigh);
      CloseHandle(hFile);
      if (!hmFile) return 0;
      lpInput = (LPDWORD)MapViewOfFile (hmFile,FILE_MAP_READ,0,0,0);
      CloseHandle(hmFile);
      if (dwSizeHigh) return 0; // too big..
      if ((hFile = CreateFile("E:\\test2.txt",GENERIC_READ | GENERIC_WRITE,0,NULL,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,NULL)) == INVALID_HANDLE_VALUE)
            return 0;
      hmFile = CreateFileMapping(hFile,NULL,PAGE_READWRITE,dwSizeHigh,dwSizeLow,NULL);
      CloseHandle(hFile);
      if (!hmFile) return 0;
      lpOutput = MapViewOfFile (hmFile,FILE_MAP_ALL_ACCESS,0,0,0);
      CloseHandle(hmFile);

#ifdef SINGLE_BYTE_PROCESSING
      lpInputPtr = (char *)lpInput;
      lpInputEnd = (char *)lpInput + dwSizeLow;
      lpOutputPtr = (char *)lpOutput;
      while (lpInputPtr != lpInputEnd) {
            *lpOutputPtr++ = *lpInputPtr++ | 0x80;
      }
#else
      lpInputPtr = (LPDWORD)lpInput;
      lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
      lpOutputPtr = (LPDWORD)lpOutput;
      while (lpInputPtr != lpInputEnd) {
       *lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
       }
       for (int i = dwSizeLow & 0x03;i;i--) {
            *((char *)lpOutputPtr) = *(char *)lpInputPtr | 0x80;
            lpOutputPtr = (LPDWORD)((char *)lpOutputPtr + 1);
            lpInputPtr = (LPDWORD)((char *)lpInputPtr + 1);
       }
#endif

       UnmapViewOfFile (lpInput);
       UnmapViewOfFile (lpOutput);
       return 0;
}

void SYSTest() {
  char buffer[SIZE];
  int i,len,hr,hw;
  hr = open("E:\\test1.txt", O_RDONLY|O_BINARY);
  hw = open("E:\\test3.txt", O_WRONLY|O_CREAT|O_TRUNC|O_BINARY);
  while ((len=read (hr, buffer, SIZE)) > 0) {
     for (i=0 ; i<len ; i++) if ( ! (buffer[i] & 0x80)) buffer[i] |= 0x80;
     write (hw, buffer, len);
  }
  close (hr);
  close (hw);
}

////////////////////////////////////////////////////////////////

have fun.

b.

Aaah didn't realize that before ; is it compiled with MS ?
So it's all but compatible.
Do you have a standard version ?? I.e. compilable with g++ or gcc ...

You know I believe you of course, open/read, system calls, may be slower than other stuff.
But the mapping has to read the file anyway at some point!

So, open/read may be slower ... ok ... but  680  times slower... I'd like to check that by myself :)

PS: here is a method to calculate the time

-----------add this code to your prog----------
#include <sys/time.h>

struct timeval tt;

void start() { gettimeofday(&tt, NULL); }

unsigned long end()
{
   struct timeval t;
   gettimeofday (&t, NULL);
   return (t.tv_sec*1000 + t.tv_usec/1000) - (tt.tv_sec*1000 + tt.tv_usec/1000);
}----------------------------------------------------

Call it this way:

int main()
{
     start();
     MMTest();
     printf ("MMTest took %ld ms\n", end());

     start();
     SYSTest();
     printf ("SysTest took %ld ms\n", end());

     return (0);
}

Mercantilum,

>> Aaah didn't realize that before ; is it compiled with MS ?

Uh.. the OP is on a windows system.   .. ifile.open("C:\\test1.txt");

>> Do you have a standard version ??

No.  Didn't bother coding for gnu compiler as the OP is not not using linux.  Feel free to port yourself.

>> So, open/read may be slower ... ok ... but  680  times slower... I'd like to check that by myself :)

Please do so.

>> PS: here is a method to calculate the time

Yes, I am well aware of how to calculate time in both linux and windows.  Thanks anyway.  Also, the profiler works fine for me.

b.

You can use mmap in Linux to accomplish the same (man mmap will tell you more).

booki,
Thank you very much,
you are not only help me the question,
but also give me the idea where my bug is.

>>You can use mmap in Linux to accomplish the same (man mmap will tell you more).

It's been my experience that file mapping works much better on Windows and UNIX then it does on Linux.

I'm not sure why, but the file mapping on Linux seems to have poor optimization, and the mapping can actually be slower then using open/read function sets.
This is not the case in the UNIX platforms I've tested, and most certainly not the case in Windows.
File mapping in Windows is much faster the C/C++ file functions.

Hi, booki:

*lpOutputPtr++ = *lpInputPtr++ | 0x80;
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;

How about
if(c > 128) c = c - 128;
else if(c < 128) c = c + 128;

Thanks

billcch,

Are you looking to toggle (if it's set then unset, if it's unset then set) the high bit?

Then you can use bitwise XOR.

*lpOutputPtr++ = *lpInputPtr++ ^ 0x80;
*lpOutputPtr++ = *lpInputPtr++ ^ 0x80808080;

But, what you posted will also work if processing one byte at a time.  It won't work if you want to process 4 at a time.

b.

This is with a 2.4.20 kernel:

# ./mmap
MMTest took 234 ms
SysTest took 594 ms
# ./mmap
MMTest took 239 ms
SysTest took 608 ms
# ./mmap
MMTest took 259 ms
SysTest took 607 ms
# ./mmap
MMTest took 245 ms
SysTest took 657 ms

This was with 4-byte read/write operations. Not the 6x factor you get with Windows, but still better than just open/read/write.

>>This was with 4-byte read/write operations. Not the 6x factor you get with Windows, but still better than just open/read/write.

What version of Linux are you running?  Is it RedHat?

I had a debate with another expert about mmap functions being faster then C/C++ functions, and on the Linux platform, he proved me wrong.
No matter what I did with his code, I couldn't get the mmap functions to perform faster then either fread or read on the Linux platform.
Furthermore, he claim that on his Linux machine the C-functions ran faster then the mmap functions.
He also claim that when he looked at the implementation for mmap on Linux, that it was just a wrapper for the C file functions, and that it didn't have any OS specific memory mapping code.

Could you post the code you're using for your test?

Correction:
I just found the previous question, and it's BSD that had the performance problem.  And the problem was in shm_open, which is used with mmap for memory mapping.
Here's the implementation he posted for BSD's shm_open function.

int shm_open (name, flags, mode)
   char    *name;
   int         flags;
   mode_t  mode;
{
   struct stat         st;
   register int    fd;

   if (! name) {
    errno = EINVAL;

    return (-1);
   }

   if ((fd = open (name, flags, mode)) < 0)
    return (-1);

   if (fstat (fd, &st) < 0)
    return (-1);

   /*
    *     The whole difference to the plain open(2) is here.
    *
    *     We check that the file descriptor we just opened
    *     refers to a regular file. We also report invalid
    *     parameter, if the file name did not give us just
    *     a plain regular file.
    *     This is to guarantee that the opened descriptor
    *     really can be mmap()ed later on.
    */

   if (! S_ISREG (st.st_mode)) {
    close (fd);

    errno = EINVAL;

    return (-1);
   }

   return (fd);
}

Axter,

>>  I had a debate with another expert about mmap functions being faster then C/C++ functions...

I'd interested in looking at the code, could you provide a link.

b.

Axter,

nevermind.  that was quick..

b.

I don't see the BSD implementation for shm_open as being a true POSIX compliant function.
It looks like they just did a quick hack to get a simulated functionality of a true shm_open function.

Anyway the primary point I was trying to make is that memory mapping is usually the most efficient way to read/write to a file, but if the OS has poor implementation, your code can actually be less efficient with memory mapping.

I see BSD as the exception, and not the rule.

Even though we now know it wasn't Linux, here is the code that I used. I took the original MMTest() function from above and Linuxified it. BTW I run the test on a SuSE 8.2.

int
MMTest ()
{
  int inputFile, outputFile;    // handles for files and file mappings
  void *lpInput, *lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
  char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
  unsigned int *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#endif
  unsigned int fileSize, dwSizeHigh;

  if ((inputFile = open ("/tmp/test1.txt", O_RDONLY)) == -1)
    return 0;

  // get the file size
  struct stat sbuf;
  fstat (inputFile, &sbuf);
  fileSize = sbuf.st_size;

  lpInput = mmap (0, fileSize, PROT_READ, MAP_SHARED, inputFile, 0);

  if ((outputFile =
       open ("/tmp/test2.txt", O_RDWR | O_CREAT | O_TRUNC, 0644)) == -1)
    return 0;

  ftruncate (outputFile, fileSize);
  lpOutput = mmap (0, fileSize, PROT_WRITE, MAP_SHARED, outputFile, 0);

#ifdef SINGLE_BYTE_PROCESSING
  lpInputPtr = (char *) lpInput;
  lpInputEnd = (char *) lpInput + fileSize;
  lpOutputPtr = (char *) lpOutput;
  while (lpInputPtr != lpInputEnd)
    {
      *lpOutputPtr++ = *lpInputPtr++ | 0x80;
    }
#else
  lpInputPtr = (unsigned int *) lpInput;
  lpInputEnd =
    (unsigned int *) ((unsigned int) lpInput + (fileSize & 0xFFFFFFFC));
  lpOutputPtr = (unsigned int *) lpOutput;
  while (lpInputPtr != lpInputEnd)
    {
      *lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
    }
  for (int i = fileSize & 0x03; i; i--)
    {
      *((char *) lpOutputPtr) = *(char *) lpInputPtr | 0x80;
      lpOutputPtr = (unsigned int *) ((char *) lpOutputPtr + 1);
      lpInputPtr = (unsigned int *) ((char *) lpInputPtr + 1);
    }
#endif
  munmap (lpInput, fileSize);
  munmap (lpOutput, fileSize);
  close (inputFile);
  close (outputFile);
  return 0;
}