Solved

ifstream + Binary file + Input -- Problems

Posted on 2006-06-20
14
818 Views
Last Modified: 2013-11-15
Basically I'm reading a file which is 1201*1201*2 bytes (2.8 megs) in size. I have several thousands of these files.
Upon inspection, these files are exactly this size.

These files contain short-values in big-endian format. My program converts them. But that matters little to this question.

When I attempt to read the entire file into memory, Many times in.read(..) will return false.
This isually happens around byte # 69632  and this isn't file specific.

I've tried reading the entire file in 1 shot  (in.read(BigBuffer, 1201*1201*2))  
I've tried reading it in via a normal buffer  (in.read(Buffer, 1201))   ... process buffer ...
I've tried reading it in one byte at a time (in.read(AChar, 1)) ... process AChar...

Here's the code using a normal sized buffer[1201]

                        char C; int Counter = 0; char buffer[1201]; int H;
                        while(Counter < 1201 * 1201 * 2){//in.read(&C, 1)){
                              if(!in.good()){
                                    MessageBox(NULL, "It's not good", "Nope", MB_OK);
                              }
                              in.read(buffer, 1201);
                              for(H = 0; H < 1201; H++){
                                    FileData[Counter]=buffer[H];
                                    Counter++;
                              }
                        }

Note: FileData is an array of Chars.    Char* FileData = new Char[1201*1201*2]

Note: I had an old system which would open a file, and seek around in that file for specific bytes that were needed... but now I pretty much need every byte in every file, so I'm hoping to just read it all in at once and access it in an array (the old seek seek seek seek strategy was pretty slow for this).

Sincerely,
Confused and Frustrated
0
Comment
Question by:oxygen_728
  • 8
  • 2
  • 2
  • +1
14 Comments
 
LVL 86

Accepted Solution

by:
jkr earned 450 total points
ID: 16945946
Stupid Q - how are you opening the file? Are you using 'ios_base::binary' as the open mode?

Also, if you are on windows, you might want to consider reading the file using Win32 APIs, e.g.

HANDLE hFile = CreateFile("data.bin",GENERIC_READ,0,NULL,OPEN_EXISTING,0,NULL);

DWORD dwSize = 1201 * 1201 *2;
DWORD dwRead;

ReadFile(hFile,(LPVOID FileData,dwSize,&dwRead,NULL);

CloseHandle(hFile);

if (dwSize != dwRead) {

  // error
}
0
 

Author Comment

by:oxygen_728
ID: 16946239
I'm sorry, I thought i included that.
in.open(Path.c_str(),ios::in | ios::binary);

I got pretty frustrated looking through the archived questions on EE - almost EVERYBODY that had a problem similar to mine wasn't using the ios::binary flag.


I'm trying to use the ReadFile technique now.... but I'm getting some strange error codes.

It may be a hardware issue... I'm using an external USB hard drive... and It currently won't let me copy files from it to an internal hard drive via Windows Explorer.

It keeps giving me a message popup in the bottom right of my taskbar:

"Delayed Write Error"
"Windows was unable to savea ll the data for the file G:.  ..... "

I've got to be social for a few hours, I'll start hacking away at it when I return.

Thanks for the tip about the windows ReadFile() thing... the error codes it provides are very informative.

0
 
LVL 14

Assisted Solution

by:wayside
wayside earned 50 total points
ID: 16946320
If you don't open using binary mode, when you read a ^Z (control-Z, decimal byte 26) it will be interpreted as the end of file marker.  This might be why your read fails at a certain point - look at your data file in a hex editor and see if there is a ^Z near the spot where the read fails.

In addition, <CR><LF> pairs are translated to a single <LF>, which is probably not what you want either.

> I'm hoping to just read it all in at once and access it in an array

If what you want is to access your file as if it were an array, you might consider opening it as a memory-mapped file. This frees you from doing any input yourself, and you can treat the file exactly as an array.
0
 

Author Comment

by:oxygen_728
ID: 16946537
wayside: Sorry I wasn't clear above... I'm opening as binary
wayside: Thanks for the tip about a memory mapped file - I wonder if it's faster to use a memory mapped file or to simply read an entire file into an array and index it via an array?

0
 
LVL 86

Expert Comment

by:jkr
ID: 16946561
There is no difference in reading the file with one call compared to opening it as a memory mapped file, the result is the same.
0
 

Author Comment

by:oxygen_728
ID: 16946693
ok thanks, as soon as I get back home I'll hack away at it
0
 
LVL 17

Expert Comment

by:rstaveley
ID: 16947018
> It may be a hardware issue

You might want to check out the Microsoft patch recommended in the following thread: http://www.short-media.com/forum/archive/index.php/t-10625
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 

Author Comment

by:oxygen_728
ID: 16947074
Hey i'll try that out when I get home. It'd be nice if that fixed the problem... I also suspect the directory the files are stored in is corrupt after reading that thread. I will have to "scan-disk" it.

0
 
LVL 14

Expert Comment

by:wayside
ID: 16947719
>  There is no difference in reading the file with one call compared to opening it as a memory mapped file,

I don't think this is necessarily true, it can be faster, it depends on what else is going on in the program, how the memory is accessed, and how much can go on in the background. At worst, it will be no different than reading the entire file in one read, but it may be faster. You'd have to benchmark the app to see.

0
 

Author Comment

by:oxygen_728
ID: 16947972
I kinda figured they'd be about equal
0
 

Author Comment

by:oxygen_728
ID: 16948232
Ok I solved the problem by enabling write caching on my hard drive... which seems counterintuitive... but when you're doing with millions of lines of code and two separate things like USB and HDD transfer are expected to work properly on the first try, who knows what works?

Thanks for the tips
0
 

Author Comment

by:oxygen_728
ID: 16948235
Again, to restate for anybody stumbling on this thread:

My problem was an incorrectly working hard drive.

The most common solution to this problem is to verify that you are opening the file with ios::binary

0
 
LVL 17

Expert Comment

by:rstaveley
ID: 16948948
Enabling write caching sounds counterintuitive as you say. If I was you (and could spare the time to do it), I'd follow this up in the hardware TAs here at EE, and see what the search facility comes up with on "delayed write error" - i.e.

http://search.experts-exchange.com/search.jsp?query=%22delayed+write+error%22&searchType=all

Here's my guess: Write-caching is now causing your HDD to need to do fewer seeks for writing and whatever the fundamental problem is that you have, write caching is making it occur less often. Defragmentation may help too. My guess is that during the delayed write error event, I/O is blocked and reads fail. It would be better if you could address the fundamental problem. The hardware experts are the ones to ask, but certainly you should take a look at your event log to see if you are still getting delayed write errors albeit less often.
0
 

Author Comment

by:oxygen_728
ID: 16949598
Ok will do thanks rstaveley, much appreciated
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

This article covers general Notes 8.5 troubleshooting information including recreating the Notes\Data folder.
If your app took Google’s lash recently, here are the 5 most likely reasons.
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now