billcch
asked on
Read binary file - efficiently
Please someone help me...
I need to read text file by each character in ascii
add 128 for the character (if it's ascii code smaller than 128)
<< don't change if the character is 1xxxxxxx,
do change 0xxxxxxx to 1xxxxxxx >>
then output the result to another file...
ifstream ifile;
ofstream ofile;
char c;
ifile.open("C:\\test1.txt" );
ofile.open("C:\\test2.txt" );
while(!ifile.eof())
{
ifile >> c;
if(c < 128)
c = c + 128;
ofile << c;
}
I would like find some way more efficient,
such like read byte directly ....
since some of file is really hurge (text file, but more than 6MB)
so I actually read 10,000 character a time,
convert them by each character
then just write into the file...
but I think there are lots of efficient way
please help me...
thanks
I need to read text file by each character in ascii
add 128 for the character (if it's ascii code smaller than 128)
<< don't change if the character is 1xxxxxxx,
do change 0xxxxxxx to 1xxxxxxx >>
then output the result to another file...
ifstream ifile;
ofstream ofile;
char c;
ifile.open("C:\\test1.txt"
ofile.open("C:\\test2.txt"
while(!ifile.eof())
{
ifile >> c;
if(c < 128)
c = c + 128;
ofile << c;
}
I would like find some way more efficient,
such like read byte directly ....
since some of file is really hurge (text file, but more than 6MB)
so I actually read 10,000 character a time,
convert them by each character
then just write into the file...
but I think there are lots of efficient way
please help me...
thanks
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Umm.. note that the code I listed isn't exactly the same as your code. The difference is your code skips over whitespaces (spaces, tabs, newline, etc..). My code processes every byte.
For example let's say your input file contains a space (ascii code 0x20) and the letter A (code 0x41). When you make the statement:
>> ifile >> c;
It skips over the space and gets 'A'. Adds 128 to 'A' (0x41) to give 0xC1 and sends it to the output file. So you have
Input file: {0x20, 0x41, 0x20}
Your codes output: {0xC1}
My codes output: {0xA0, 0xC1, 0xA0}
Not sure which you want but to change my code to do the same as your's just throw some if statements into the while loop.
b.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Oops..
Replace while loop with:
lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
while (lpInputPtr != lpInputEnd) {
*lpOutput++ = *lpInputPtr++ | 0x80808080;
}
for (i = dwSizeLow & 0x03;i;i--) {
*(char *)lpOutput = *(char *)lpInputPtr | 0x80;
(char *)lpOutput += 1;
(char *)lpInputPtr += 1;
}
Regardless of how you decide to read from the file, processing 4 bytes at a time will be faster than processing one at a time.
Mercantilum,
>> It has been the object of a looong discussion in another thread
It may be helpful to the OP (and others who are participating) if you provide a link.
b.
Replace while loop with:
lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
while (lpInputPtr != lpInputEnd) {
*lpOutput++ = *lpInputPtr++ | 0x80808080;
}
for (i = dwSizeLow & 0x03;i;i--) {
*(char *)lpOutput = *(char *)lpInputPtr | 0x80;
(char *)lpOutput += 1;
(char *)lpInputPtr += 1;
}
Regardless of how you decide to read from the file, processing 4 bytes at a time will be faster than processing one at a time.
Mercantilum,
>> It has been the object of a looong discussion in another thread
It may be helpful to the OP (and others who are participating) if you provide a link.
b.
Mercantilum,
... I ran your code against my code in a profiler...
OS: WinXP
Compiler: VC++7
Profiler: Vtune
Test file size: 7 MB
Time
Syscall open/read/write: 664,740
Memory Mapped files: 116,951
Memory mapped files seems about 6 times faster... so I tried again but this time I dumbed down the code to process one byte at a time.
Time
Syscall open/read/write: 717,742
Memory Mapped files w/byte processing: 259,052
Still about 3 times faster. I used your code as is.. just cut and paste. My code is as is also except for a couple (oops) syntax corrections. I placed your code in one function and my code in another function. My function is called first so if there is any advantage gained from having buffered the previous file opening, your code would stand to benefit.
Thoughts?
b.
... I ran your code against my code in a profiler...
OS: WinXP
Compiler: VC++7
Profiler: Vtune
Test file size: 7 MB
Time
Syscall open/read/write: 664,740
Memory Mapped files: 116,951
Memory mapped files seems about 6 times faster... so I tried again but this time I dumbed down the code to process one byte at a time.
Time
Syscall open/read/write: 717,742
Memory Mapped files w/byte processing: 259,052
Still about 3 times faster. I used your code as is.. just cut and paste. My code is as is also except for a couple (oops) syntax corrections. I placed your code in one function and my code in another function. My function is called first so if there is any advantage gained from having buffered the previous file opening, your code would stand to benefit.
Thoughts?
b.
When there are r/w ian optimization can help.
Can you please provide your (working) code, I'd like to test on g++, thanks.
Can you please provide your (working) code, I'd like to test on g++, thanks.
Mercantilum,
Hold on.. let me clean it up a bit..
b.
Hold on.. let me clean it up a bit..
b.
Mercantilum,
I forgot to mention all optimizations were off.
////////////////////////// ////////// ////////// ////////// ////////// ////////// ///
// Uncomment for single byte processing as opposed to processing 4 bytes at a time
//#define SINGLE_BYTE_PROCESSING
int MMTest() {
HANDLE hFile, hmFile; // handles for files and file mappings
LPVOID lpInput, lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
LPDWORD lpInputPtr, lpInputEnd, lpOutputPtr;
#endif
DWORD dwSizeLow, dwSizeHigh;
if ((hFile = CreateFile("E:\\test1.txt" ,GENERIC_R EAD,0,NULL ,OPEN_EXIS TING,FILE_ ATTRIBUTE_ NORMAL,NUL L)) == INVALID_HANDLE_VALUE)
return 0;
hmFile = CreateFileMapping(hFile,NU LL,PAGE_RE ADONLY,0,0 ,NULL);
dwSizeLow = GetFileSize(hFile, &dwSizeHigh);
CloseHandle(hFile);
if (!hmFile) return 0;
lpInput = (LPDWORD)MapViewOfFile (hmFile,FILE_MAP_READ,0,0, 0);
CloseHandle(hmFile);
if (dwSizeHigh) return 0; // too big..
if ((hFile = CreateFile("E:\\test2.txt" ,GENERIC_R EAD | GENERIC_WRITE,0,NULL,OPEN_ EXISTING,F ILE_ATTRIB UTE_NORMAL ,NULL)) == INVALID_HANDLE_VALUE)
return 0;
hmFile = CreateFileMapping(hFile,NU LL,PAGE_RE ADWRITE,dw SizeHigh,d wSizeLow,N ULL);
CloseHandle(hFile);
if (!hmFile) return 0;
lpOutput = MapViewOfFile (hmFile,FILE_MAP_ALL_ACCES S,0,0,0);
CloseHandle(hmFile);
#ifdef SINGLE_BYTE_PROCESSING
lpInputPtr = (char *)lpInput;
lpInputEnd = (char *)lpInput + dwSizeLow;
lpOutputPtr = (char *)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
}
#else
lpInputPtr = (LPDWORD)lpInput;
lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
lpOutputPtr = (LPDWORD)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
}
for (int i = dwSizeLow & 0x03;i;i--) {
*((char *)lpOutputPtr) = *(char *)lpInputPtr | 0x80;
lpOutputPtr = (LPDWORD)((char *)lpOutputPtr + 1);
lpInputPtr = (LPDWORD)((char *)lpInputPtr + 1);
}
#endif
UnmapViewOfFile (lpInput);
UnmapViewOfFile (lpOutput);
return 1;
}
////////////////////////// ////////// ////////// ////////// ////////// ////////// ///
Oo, and I just ran the test with "Maximize Speed" optimization....
Maximize for Speed Time
Syscall open/read/write: 282,247
Memory Mapped files w/byte processing: 415
b.
I forgot to mention all optimizations were off.
//////////////////////////
// Uncomment for single byte processing as opposed to processing 4 bytes at a time
//#define SINGLE_BYTE_PROCESSING
int MMTest() {
HANDLE hFile, hmFile; // handles for files and file mappings
LPVOID lpInput, lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
LPDWORD lpInputPtr, lpInputEnd, lpOutputPtr;
#endif
DWORD dwSizeLow, dwSizeHigh;
if ((hFile = CreateFile("E:\\test1.txt"
return 0;
hmFile = CreateFileMapping(hFile,NU
dwSizeLow = GetFileSize(hFile, &dwSizeHigh);
CloseHandle(hFile);
if (!hmFile) return 0;
lpInput = (LPDWORD)MapViewOfFile (hmFile,FILE_MAP_READ,0,0,
CloseHandle(hmFile);
if (dwSizeHigh) return 0; // too big..
if ((hFile = CreateFile("E:\\test2.txt"
return 0;
hmFile = CreateFileMapping(hFile,NU
CloseHandle(hFile);
if (!hmFile) return 0;
lpOutput = MapViewOfFile (hmFile,FILE_MAP_ALL_ACCES
CloseHandle(hmFile);
#ifdef SINGLE_BYTE_PROCESSING
lpInputPtr = (char *)lpInput;
lpInputEnd = (char *)lpInput + dwSizeLow;
lpOutputPtr = (char *)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
}
#else
lpInputPtr = (LPDWORD)lpInput;
lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
lpOutputPtr = (LPDWORD)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
}
for (int i = dwSizeLow & 0x03;i;i--) {
*((char *)lpOutputPtr) = *(char *)lpInputPtr | 0x80;
lpOutputPtr = (LPDWORD)((char *)lpOutputPtr + 1);
lpInputPtr = (LPDWORD)((char *)lpInputPtr + 1);
}
#endif
UnmapViewOfFile (lpInput);
UnmapViewOfFile (lpOutput);
return 1;
}
//////////////////////////
Oo, and I just ran the test with "Maximize Speed" optimization....
Maximize for Speed Time
Syscall open/read/write: 282,247
Memory Mapped files w/byte processing: 415
b.
Ok, just a question: in the timing, do you include the filemapping init etc... for Memory Mapped files w/byte processing ?
(since I guess it is what takes the most of the time, not the loop itself, done in memory :)
(since I guess it is what takes the most of the time, not the loop itself, done in memory :)
Mercantilum,
I posted the function that gets called from main. The times posted are the total time spent in the function MMTest(). So that means opening, mapping and processing.
I cut and paste your code (opening, reading, writing, processing, and closing.. everything) and put it in a function called void SYSTest(). The times posted are for the total time spent in SYSTest().
b.
I posted the function that gets called from main. The times posted are the total time spent in the function MMTest(). So that means opening, mapping and processing.
I cut and paste your code (opening, reading, writing, processing, and closing.. everything) and put it in a function called void SYSTest(). The times posted are for the total time spent in SYSTest().
b.
Do you mind posting the whole code, from line 1 to end, with includes, method for calc. time etc...
So that I can copy and paste as well and play :)
So that I can copy and paste as well and play :)
Mercantilum,
>> ..method for calc. time..
As I said earlier the timing is all done by the profiler. It runs the executable and shows how much time was spent in each function, therefore, I have no code for calculating the time. You'll have to write your own.
Anyway here is the 'whole code'.
////////////////////////// ////////// ////////// ////////// ////////
#include <fstream>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <io.h>
//#include <unistd.h>
#ifndef O_BINARY
#define O_BINARY 0
#endif
#define SIZE 1024
// Uncomment for single byte processing as opposed to processing 4 bytes at a time
#define SINGLE_BYTE_PROCESSING
int MMTest();
void SYSTest();
int main() {
MMTest();
SYSTest();
return 0;
}
int MMTest() {
HANDLE hFile, hmFile; // handles for files and file mappings
LPVOID lpInput, lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
LPDWORD lpInputPtr, lpInputEnd, lpOutputPtr;
#endif
DWORD dwSizeLow, dwSizeHigh;
if ((hFile = CreateFile("E:\\test1.txt" ,GENERIC_R EAD,0,NULL ,OPEN_EXIS TING,FILE_ ATTRIBUTE_ NORMAL,NUL L)) == INVALID_HANDLE_VALUE)
return 0;
hmFile = CreateFileMapping(hFile,NU LL,PAGE_RE ADONLY,0,0 ,NULL);
dwSizeLow = GetFileSize(hFile, &dwSizeHigh);
CloseHandle(hFile);
if (!hmFile) return 0;
lpInput = (LPDWORD)MapViewOfFile (hmFile,FILE_MAP_READ,0,0, 0);
CloseHandle(hmFile);
if (dwSizeHigh) return 0; // too big..
if ((hFile = CreateFile("E:\\test2.txt" ,GENERIC_R EAD | GENERIC_WRITE,0,NULL,OPEN_ EXISTING,F ILE_ATTRIB UTE_NORMAL ,NULL)) == INVALID_HANDLE_VALUE)
return 0;
hmFile = CreateFileMapping(hFile,NU LL,PAGE_RE ADWRITE,dw SizeHigh,d wSizeLow,N ULL);
CloseHandle(hFile);
if (!hmFile) return 0;
lpOutput = MapViewOfFile (hmFile,FILE_MAP_ALL_ACCES S,0,0,0);
CloseHandle(hmFile);
#ifdef SINGLE_BYTE_PROCESSING
lpInputPtr = (char *)lpInput;
lpInputEnd = (char *)lpInput + dwSizeLow;
lpOutputPtr = (char *)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
}
#else
lpInputPtr = (LPDWORD)lpInput;
lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
lpOutputPtr = (LPDWORD)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
}
for (int i = dwSizeLow & 0x03;i;i--) {
*((char *)lpOutputPtr) = *(char *)lpInputPtr | 0x80;
lpOutputPtr = (LPDWORD)((char *)lpOutputPtr + 1);
lpInputPtr = (LPDWORD)((char *)lpInputPtr + 1);
}
#endif
UnmapViewOfFile (lpInput);
UnmapViewOfFile (lpOutput);
return 0;
}
void SYSTest() {
char buffer[SIZE];
int i,len,hr,hw;
hr = open("E:\\test1.txt", O_RDONLY|O_BINARY);
hw = open("E:\\test3.txt", O_WRONLY|O_CREAT|O_TRUNC|O _BINARY);
while ((len=read (hr, buffer, SIZE)) > 0) {
for (i=0 ; i<len ; i++) if ( ! (buffer[i] & 0x80)) buffer[i] |= 0x80;
write (hw, buffer, len);
}
close (hr);
close (hw);
}
////////////////////////// ////////// ////////// ////////// ////////
have fun.
b.
>> ..method for calc. time..
As I said earlier the timing is all done by the profiler. It runs the executable and shows how much time was spent in each function, therefore, I have no code for calculating the time. You'll have to write your own.
Anyway here is the 'whole code'.
//////////////////////////
#include <fstream>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <io.h>
//#include <unistd.h>
#ifndef O_BINARY
#define O_BINARY 0
#endif
#define SIZE 1024
// Uncomment for single byte processing as opposed to processing 4 bytes at a time
#define SINGLE_BYTE_PROCESSING
int MMTest();
void SYSTest();
int main() {
MMTest();
SYSTest();
return 0;
}
int MMTest() {
HANDLE hFile, hmFile; // handles for files and file mappings
LPVOID lpInput, lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
LPDWORD lpInputPtr, lpInputEnd, lpOutputPtr;
#endif
DWORD dwSizeLow, dwSizeHigh;
if ((hFile = CreateFile("E:\\test1.txt"
return 0;
hmFile = CreateFileMapping(hFile,NU
dwSizeLow = GetFileSize(hFile, &dwSizeHigh);
CloseHandle(hFile);
if (!hmFile) return 0;
lpInput = (LPDWORD)MapViewOfFile (hmFile,FILE_MAP_READ,0,0,
CloseHandle(hmFile);
if (dwSizeHigh) return 0; // too big..
if ((hFile = CreateFile("E:\\test2.txt"
return 0;
hmFile = CreateFileMapping(hFile,NU
CloseHandle(hFile);
if (!hmFile) return 0;
lpOutput = MapViewOfFile (hmFile,FILE_MAP_ALL_ACCES
CloseHandle(hmFile);
#ifdef SINGLE_BYTE_PROCESSING
lpInputPtr = (char *)lpInput;
lpInputEnd = (char *)lpInput + dwSizeLow;
lpOutputPtr = (char *)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
}
#else
lpInputPtr = (LPDWORD)lpInput;
lpInputEnd = (LPDWORD)((DWORD)lpInput + (dwSizeLow & 0xFFFFFFFC));
lpOutputPtr = (LPDWORD)lpOutput;
while (lpInputPtr != lpInputEnd) {
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
}
for (int i = dwSizeLow & 0x03;i;i--) {
*((char *)lpOutputPtr) = *(char *)lpInputPtr | 0x80;
lpOutputPtr = (LPDWORD)((char *)lpOutputPtr + 1);
lpInputPtr = (LPDWORD)((char *)lpInputPtr + 1);
}
#endif
UnmapViewOfFile (lpInput);
UnmapViewOfFile (lpOutput);
return 0;
}
void SYSTest() {
char buffer[SIZE];
int i,len,hr,hw;
hr = open("E:\\test1.txt", O_RDONLY|O_BINARY);
hw = open("E:\\test3.txt", O_WRONLY|O_CREAT|O_TRUNC|O
while ((len=read (hr, buffer, SIZE)) > 0) {
for (i=0 ; i<len ; i++) if ( ! (buffer[i] & 0x80)) buffer[i] |= 0x80;
write (hw, buffer, len);
}
close (hr);
close (hw);
}
//////////////////////////
have fun.
b.
Aaah didn't realize that before ; is it compiled with MS ?
So it's all but compatible.
Do you have a standard version ?? I.e. compilable with g++ or gcc ...
You know I believe you of course, open/read, system calls, may be slower than other stuff.
But the mapping has to read the file anyway at some point!
So, open/read may be slower ... ok ... but 680 times slower... I'd like to check that by myself :)
PS: here is a method to calculate the time
-----------add this code to your prog----------
#include <sys/time.h>
struct timeval tt;
void start() { gettimeofday(&tt, NULL); }
unsigned long end()
{
struct timeval t;
gettimeofday (&t, NULL);
return (t.tv_sec*1000 + t.tv_usec/1000) - (tt.tv_sec*1000 + tt.tv_usec/1000);
}------------------------- ---------- ---------- -------
Call it this way:
int main()
{
start();
MMTest();
printf ("MMTest took %ld ms\n", end());
start();
SYSTest();
printf ("SysTest took %ld ms\n", end());
return (0);
}
So it's all but compatible.
Do you have a standard version ?? I.e. compilable with g++ or gcc ...
You know I believe you of course, open/read, system calls, may be slower than other stuff.
But the mapping has to read the file anyway at some point!
So, open/read may be slower ... ok ... but 680 times slower... I'd like to check that by myself :)
PS: here is a method to calculate the time
-----------add this code to your prog----------
#include <sys/time.h>
struct timeval tt;
void start() { gettimeofday(&tt, NULL); }
unsigned long end()
{
struct timeval t;
gettimeofday (&t, NULL);
return (t.tv_sec*1000 + t.tv_usec/1000) - (tt.tv_sec*1000 + tt.tv_usec/1000);
}-------------------------
Call it this way:
int main()
{
start();
MMTest();
printf ("MMTest took %ld ms\n", end());
start();
SYSTest();
printf ("SysTest took %ld ms\n", end());
return (0);
}
Mercantilum,
>> Aaah didn't realize that before ; is it compiled with MS ?
Uh.. the OP is on a windows system. .. ifile.open("C:\\test1.txt" );
>> Do you have a standard version ??
No. Didn't bother coding for gnu compiler as the OP is not not using linux. Feel free to port yourself.
>> So, open/read may be slower ... ok ... but 680 times slower... I'd like to check that by myself :)
Please do so.
>> PS: here is a method to calculate the time
Yes, I am well aware of how to calculate time in both linux and windows. Thanks anyway. Also, the profiler works fine for me.
b.
>> Aaah didn't realize that before ; is it compiled with MS ?
Uh.. the OP is on a windows system. .. ifile.open("C:\\test1.txt"
>> Do you have a standard version ??
No. Didn't bother coding for gnu compiler as the OP is not not using linux. Feel free to port yourself.
>> So, open/read may be slower ... ok ... but 680 times slower... I'd like to check that by myself :)
Please do so.
>> PS: here is a method to calculate the time
Yes, I am well aware of how to calculate time in both linux and windows. Thanks anyway. Also, the profiler works fine for me.
b.
You can use mmap in Linux to accomplish the same (man mmap will tell you more).
ASKER
booki,
Thank you very much,
you are not only help me the question,
but also give me the idea where my bug is.
Thank you very much,
you are not only help me the question,
but also give me the idea where my bug is.
>>You can use mmap in Linux to accomplish the same (man mmap will tell you more).
It's been my experience that file mapping works much better on Windows and UNIX then it does on Linux.
I'm not sure why, but the file mapping on Linux seems to have poor optimization, and the mapping can actually be slower then using open/read function sets.
This is not the case in the UNIX platforms I've tested, and most certainly not the case in Windows.
File mapping in Windows is much faster the C/C++ file functions.
It's been my experience that file mapping works much better on Windows and UNIX then it does on Linux.
I'm not sure why, but the file mapping on Linux seems to have poor optimization, and the mapping can actually be slower then using open/read function sets.
This is not the case in the UNIX platforms I've tested, and most certainly not the case in Windows.
File mapping in Windows is much faster the C/C++ file functions.
ASKER
Hi, booki:
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
How about
if(c > 128) c = c - 128;
else if(c < 128) c = c + 128;
Thanks
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
How about
if(c > 128) c = c - 128;
else if(c < 128) c = c + 128;
Thanks
billcch,
Are you looking to toggle (if it's set then unset, if it's unset then set) the high bit?
Then you can use bitwise XOR.
*lpOutputPtr++ = *lpInputPtr++ ^ 0x80;
*lpOutputPtr++ = *lpInputPtr++ ^ 0x80808080;
But, what you posted will also work if processing one byte at a time. It won't work if you want to process 4 at a time.
b.
Are you looking to toggle (if it's set then unset, if it's unset then set) the high bit?
Then you can use bitwise XOR.
*lpOutputPtr++ = *lpInputPtr++ ^ 0x80;
*lpOutputPtr++ = *lpInputPtr++ ^ 0x80808080;
But, what you posted will also work if processing one byte at a time. It won't work if you want to process 4 at a time.
b.
This is with a 2.4.20 kernel:
# ./mmap
MMTest took 234 ms
SysTest took 594 ms
# ./mmap
MMTest took 239 ms
SysTest took 608 ms
# ./mmap
MMTest took 259 ms
SysTest took 607 ms
# ./mmap
MMTest took 245 ms
SysTest took 657 ms
This was with 4-byte read/write operations. Not the 6x factor you get with Windows, but still better than just open/read/write.
# ./mmap
MMTest took 234 ms
SysTest took 594 ms
# ./mmap
MMTest took 239 ms
SysTest took 608 ms
# ./mmap
MMTest took 259 ms
SysTest took 607 ms
# ./mmap
MMTest took 245 ms
SysTest took 657 ms
This was with 4-byte read/write operations. Not the 6x factor you get with Windows, but still better than just open/read/write.
>>This was with 4-byte read/write operations. Not the 6x factor you get with Windows, but still better than just open/read/write.
What version of Linux are you running? Is it RedHat?
I had a debate with another expert about mmap functions being faster then C/C++ functions, and on the Linux platform, he proved me wrong.
No matter what I did with his code, I couldn't get the mmap functions to perform faster then either fread or read on the Linux platform.
Furthermore, he claim that on his Linux machine the C-functions ran faster then the mmap functions.
He also claim that when he looked at the implementation for mmap on Linux, that it was just a wrapper for the C file functions, and that it didn't have any OS specific memory mapping code.
Could you post the code you're using for your test?
What version of Linux are you running? Is it RedHat?
I had a debate with another expert about mmap functions being faster then C/C++ functions, and on the Linux platform, he proved me wrong.
No matter what I did with his code, I couldn't get the mmap functions to perform faster then either fread or read on the Linux platform.
Furthermore, he claim that on his Linux machine the C-functions ran faster then the mmap functions.
He also claim that when he looked at the implementation for mmap on Linux, that it was just a wrapper for the C file functions, and that it didn't have any OS specific memory mapping code.
Could you post the code you're using for your test?
Correction:
I just found the previous question, and it's BSD that had the performance problem. And the problem was in shm_open, which is used with mmap for memory mapping.
Here's the implementation he posted for BSD's shm_open function.
int shm_open (name, flags, mode)
char *name;
int flags;
mode_t mode;
{
struct stat st;
register int fd;
if (! name) {
errno = EINVAL;
return (-1);
}
if ((fd = open (name, flags, mode)) < 0)
return (-1);
if (fstat (fd, &st) < 0)
return (-1);
/*
* The whole difference to the plain open(2) is here.
*
* We check that the file descriptor we just opened
* refers to a regular file. We also report invalid
* parameter, if the file name did not give us just
* a plain regular file.
* This is to guarantee that the opened descriptor
* really can be mmap()ed later on.
*/
if (! S_ISREG (st.st_mode)) {
close (fd);
errno = EINVAL;
return (-1);
}
return (fd);
}
I just found the previous question, and it's BSD that had the performance problem. And the problem was in shm_open, which is used with mmap for memory mapping.
Here's the implementation he posted for BSD's shm_open function.
int shm_open (name, flags, mode)
char *name;
int flags;
mode_t mode;
{
struct stat st;
register int fd;
if (! name) {
errno = EINVAL;
return (-1);
}
if ((fd = open (name, flags, mode)) < 0)
return (-1);
if (fstat (fd, &st) < 0)
return (-1);
/*
* The whole difference to the plain open(2) is here.
*
* We check that the file descriptor we just opened
* refers to a regular file. We also report invalid
* parameter, if the file name did not give us just
* a plain regular file.
* This is to guarantee that the opened descriptor
* really can be mmap()ed later on.
*/
if (! S_ISREG (st.st_mode)) {
close (fd);
errno = EINVAL;
return (-1);
}
return (fd);
}
Axter,
>> I had a debate with another expert about mmap functions being faster then C/C++ functions...
I'd interested in looking at the code, could you provide a link.
b.
>> I had a debate with another expert about mmap functions being faster then C/C++ functions...
I'd interested in looking at the code, could you provide a link.
b.
Axter,
nevermind. that was quick..
b.
nevermind. that was quick..
b.
I don't see the BSD implementation for shm_open as being a true POSIX compliant function.
It looks like they just did a quick hack to get a simulated functionality of a true shm_open function.
Anyway the primary point I was trying to make is that memory mapping is usually the most efficient way to read/write to a file, but if the OS has poor implementation, your code can actually be less efficient with memory mapping.
I see BSD as the exception, and not the rule.
It looks like they just did a quick hack to get a simulated functionality of a true shm_open function.
Anyway the primary point I was trying to make is that memory mapping is usually the most efficient way to read/write to a file, but if the OS has poor implementation, your code can actually be less efficient with memory mapping.
I see BSD as the exception, and not the rule.
Even though we now know it wasn't Linux, here is the code that I used. I took the original MMTest() function from above and Linuxified it. BTW I run the test on a SuSE 8.2.
int
MMTest ()
{
int inputFile, outputFile; // handles for files and file mappings
void *lpInput, *lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
unsigned int *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#endif
unsigned int fileSize, dwSizeHigh;
if ((inputFile = open ("/tmp/test1.txt", O_RDONLY)) == -1)
return 0;
// get the file size
struct stat sbuf;
fstat (inputFile, &sbuf);
fileSize = sbuf.st_size;
lpInput = mmap (0, fileSize, PROT_READ, MAP_SHARED, inputFile, 0);
if ((outputFile =
open ("/tmp/test2.txt", O_RDWR | O_CREAT | O_TRUNC, 0644)) == -1)
return 0;
ftruncate (outputFile, fileSize);
lpOutput = mmap (0, fileSize, PROT_WRITE, MAP_SHARED, outputFile, 0);
#ifdef SINGLE_BYTE_PROCESSING
lpInputPtr = (char *) lpInput;
lpInputEnd = (char *) lpInput + fileSize;
lpOutputPtr = (char *) lpOutput;
while (lpInputPtr != lpInputEnd)
{
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
}
#else
lpInputPtr = (unsigned int *) lpInput;
lpInputEnd =
(unsigned int *) ((unsigned int) lpInput + (fileSize & 0xFFFFFFFC));
lpOutputPtr = (unsigned int *) lpOutput;
while (lpInputPtr != lpInputEnd)
{
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
}
for (int i = fileSize & 0x03; i; i--)
{
*((char *) lpOutputPtr) = *(char *) lpInputPtr | 0x80;
lpOutputPtr = (unsigned int *) ((char *) lpOutputPtr + 1);
lpInputPtr = (unsigned int *) ((char *) lpInputPtr + 1);
}
#endif
munmap (lpInput, fileSize);
munmap (lpOutput, fileSize);
close (inputFile);
close (outputFile);
return 0;
}
int
MMTest ()
{
int inputFile, outputFile; // handles for files and file mappings
void *lpInput, *lpOutput;
#ifdef SINGLE_BYTE_PROCESSING
char *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#else
unsigned int *lpInputPtr, *lpInputEnd, *lpOutputPtr;
#endif
unsigned int fileSize, dwSizeHigh;
if ((inputFile = open ("/tmp/test1.txt", O_RDONLY)) == -1)
return 0;
// get the file size
struct stat sbuf;
fstat (inputFile, &sbuf);
fileSize = sbuf.st_size;
lpInput = mmap (0, fileSize, PROT_READ, MAP_SHARED, inputFile, 0);
if ((outputFile =
open ("/tmp/test2.txt", O_RDWR | O_CREAT | O_TRUNC, 0644)) == -1)
return 0;
ftruncate (outputFile, fileSize);
lpOutput = mmap (0, fileSize, PROT_WRITE, MAP_SHARED, outputFile, 0);
#ifdef SINGLE_BYTE_PROCESSING
lpInputPtr = (char *) lpInput;
lpInputEnd = (char *) lpInput + fileSize;
lpOutputPtr = (char *) lpOutput;
while (lpInputPtr != lpInputEnd)
{
*lpOutputPtr++ = *lpInputPtr++ | 0x80;
}
#else
lpInputPtr = (unsigned int *) lpInput;
lpInputEnd =
(unsigned int *) ((unsigned int) lpInput + (fileSize & 0xFFFFFFFC));
lpOutputPtr = (unsigned int *) lpOutput;
while (lpInputPtr != lpInputEnd)
{
*lpOutputPtr++ = *lpInputPtr++ | 0x80808080;
}
for (int i = fileSize & 0x03; i; i--)
{
*((char *) lpOutputPtr) = *(char *) lpInputPtr | 0x80;
lpOutputPtr = (unsigned int *) ((char *) lpOutputPtr + 1);
lpInputPtr = (unsigned int *) ((char *) lpInputPtr + 1);
}
#endif
munmap (lpInput, fileSize);
munmap (lpOutput, fileSize);
close (inputFile);
close (outputFile);
return 0;
}
Use memory mapped files and process 4 bytes at a time instead of 1.
b.