ifstream + Binary file + Input -- Problems

Basically I'm reading a file which is 1201*1201*2 bytes (2.8 megs) in size. I have several thousands of these files.
Upon inspection, these files are exactly this size.

These files contain short-values in big-endian format. My program converts them. But that matters little to this question.

When I attempt to read the entire file into memory, Many times in.read(..) will return false.
This isually happens around byte # 69632  and this isn't file specific.

I've tried reading the entire file in 1 shot  (in.read(BigBuffer, 1201*1201*2))  
I've tried reading it in via a normal buffer  (in.read(Buffer, 1201))   ... process buffer ...
I've tried reading it in one byte at a time (in.read(AChar, 1)) ... process AChar...

Here's the code using a normal sized buffer[1201]

                        char C; int Counter = 0; char buffer[1201]; int H;
                        while(Counter < 1201 * 1201 * 2){//in.read(&C, 1)){
                              if(!in.good()){
                                    MessageBox(NULL, "It's not good", "Nope", MB_OK);
                              }
                              in.read(buffer, 1201);
                              for(H = 0; H < 1201; H++){
                                    FileData[Counter]=buffer[H];
                                    Counter++;
                              }
                        }

Note: FileData is an array of Chars.    Char* FileData = new Char[1201*1201*2]

Note: I had an old system which would open a file, and seek around in that file for specific bytes that were needed... but now I pretty much need every byte in every file, so I'm hoping to just read it all in at once and access it in an array (the old seek seek seek seek strategy was pretty slow for this).

Sincerely,
Confused and Frustrated
oxygen_728Asked:
Who is Participating?
 
jkrCommented:
Stupid Q - how are you opening the file? Are you using 'ios_base::binary' as the open mode?

Also, if you are on windows, you might want to consider reading the file using Win32 APIs, e.g.

HANDLE hFile = CreateFile("data.bin",GENERIC_READ,0,NULL,OPEN_EXISTING,0,NULL);

DWORD dwSize = 1201 * 1201 *2;
DWORD dwRead;

ReadFile(hFile,(LPVOID FileData,dwSize,&dwRead,NULL);

CloseHandle(hFile);

if (dwSize != dwRead) {

  // error
}
0
 
oxygen_728Author Commented:
I'm sorry, I thought i included that.
in.open(Path.c_str(),ios::in | ios::binary);

I got pretty frustrated looking through the archived questions on EE - almost EVERYBODY that had a problem similar to mine wasn't using the ios::binary flag.


I'm trying to use the ReadFile technique now.... but I'm getting some strange error codes.

It may be a hardware issue... I'm using an external USB hard drive... and It currently won't let me copy files from it to an internal hard drive via Windows Explorer.

It keeps giving me a message popup in the bottom right of my taskbar:

"Delayed Write Error"
"Windows was unable to savea ll the data for the file G:.  ..... "

I've got to be social for a few hours, I'll start hacking away at it when I return.

Thanks for the tip about the windows ReadFile() thing... the error codes it provides are very informative.

0
 
waysideCommented:
If you don't open using binary mode, when you read a ^Z (control-Z, decimal byte 26) it will be interpreted as the end of file marker.  This might be why your read fails at a certain point - look at your data file in a hex editor and see if there is a ^Z near the spot where the read fails.

In addition, <CR><LF> pairs are translated to a single <LF>, which is probably not what you want either.

> I'm hoping to just read it all in at once and access it in an array

If what you want is to access your file as if it were an array, you might consider opening it as a memory-mapped file. This frees you from doing any input yourself, and you can treat the file exactly as an array.
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
oxygen_728Author Commented:
wayside: Sorry I wasn't clear above... I'm opening as binary
wayside: Thanks for the tip about a memory mapped file - I wonder if it's faster to use a memory mapped file or to simply read an entire file into an array and index it via an array?

0
 
jkrCommented:
There is no difference in reading the file with one call compared to opening it as a memory mapped file, the result is the same.
0
 
oxygen_728Author Commented:
ok thanks, as soon as I get back home I'll hack away at it
0
 
rstaveleyCommented:
> It may be a hardware issue

You might want to check out the Microsoft patch recommended in the following thread: http://www.short-media.com/forum/archive/index.php/t-10625
0
 
oxygen_728Author Commented:
Hey i'll try that out when I get home. It'd be nice if that fixed the problem... I also suspect the directory the files are stored in is corrupt after reading that thread. I will have to "scan-disk" it.

0
 
waysideCommented:
>  There is no difference in reading the file with one call compared to opening it as a memory mapped file,

I don't think this is necessarily true, it can be faster, it depends on what else is going on in the program, how the memory is accessed, and how much can go on in the background. At worst, it will be no different than reading the entire file in one read, but it may be faster. You'd have to benchmark the app to see.

0
 
oxygen_728Author Commented:
I kinda figured they'd be about equal
0
 
oxygen_728Author Commented:
Ok I solved the problem by enabling write caching on my hard drive... which seems counterintuitive... but when you're doing with millions of lines of code and two separate things like USB and HDD transfer are expected to work properly on the first try, who knows what works?

Thanks for the tips
0
 
oxygen_728Author Commented:
Again, to restate for anybody stumbling on this thread:

My problem was an incorrectly working hard drive.

The most common solution to this problem is to verify that you are opening the file with ios::binary

0
 
rstaveleyCommented:
Enabling write caching sounds counterintuitive as you say. If I was you (and could spare the time to do it), I'd follow this up in the hardware TAs here at EE, and see what the search facility comes up with on "delayed write error" - i.e.

http://search.experts-exchange.com/search.jsp?query=%22delayed+write+error%22&searchType=all

Here's my guess: Write-caching is now causing your HDD to need to do fewer seeks for writing and whatever the fundamental problem is that you have, write caching is making it occur less often. Defragmentation may help too. My guess is that during the delayed write error event, I/O is blocked and reads fail. It would be better if you could address the fundamental problem. The hardware experts are the ones to ask, but certainly you should take a look at your event log to see if you are still getting delayed write errors albeit less often.
0
 
oxygen_728Author Commented:
Ok will do thanks rstaveley, much appreciated
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.