Solved

ifstream + Binary file + Input -- Problems

Posted on 2006-06-20
14
822 Views
Last Modified: 2013-11-15
Basically I'm reading a file which is 1201*1201*2 bytes (2.8 megs) in size. I have several thousands of these files.
Upon inspection, these files are exactly this size.

These files contain short-values in big-endian format. My program converts them. But that matters little to this question.

When I attempt to read the entire file into memory, Many times in.read(..) will return false.
This isually happens around byte # 69632  and this isn't file specific.

I've tried reading the entire file in 1 shot  (in.read(BigBuffer, 1201*1201*2))  
I've tried reading it in via a normal buffer  (in.read(Buffer, 1201))   ... process buffer ...
I've tried reading it in one byte at a time (in.read(AChar, 1)) ... process AChar...

Here's the code using a normal sized buffer[1201]

                        char C; int Counter = 0; char buffer[1201]; int H;
                        while(Counter < 1201 * 1201 * 2){//in.read(&C, 1)){
                              if(!in.good()){
                                    MessageBox(NULL, "It's not good", "Nope", MB_OK);
                              }
                              in.read(buffer, 1201);
                              for(H = 0; H < 1201; H++){
                                    FileData[Counter]=buffer[H];
                                    Counter++;
                              }
                        }

Note: FileData is an array of Chars.    Char* FileData = new Char[1201*1201*2]

Note: I had an old system which would open a file, and seek around in that file for specific bytes that were needed... but now I pretty much need every byte in every file, so I'm hoping to just read it all in at once and access it in an array (the old seek seek seek seek strategy was pretty slow for this).

Sincerely,
Confused and Frustrated
0
Comment
Question by:oxygen_728
  • 8
  • 2
  • 2
  • +1
14 Comments
 
LVL 86

Accepted Solution

by:
jkr earned 450 total points
ID: 16945946
Stupid Q - how are you opening the file? Are you using 'ios_base::binary' as the open mode?

Also, if you are on windows, you might want to consider reading the file using Win32 APIs, e.g.

HANDLE hFile = CreateFile("data.bin",GENERIC_READ,0,NULL,OPEN_EXISTING,0,NULL);

DWORD dwSize = 1201 * 1201 *2;
DWORD dwRead;

ReadFile(hFile,(LPVOID FileData,dwSize,&dwRead,NULL);

CloseHandle(hFile);

if (dwSize != dwRead) {

  // error
}
0
 

Author Comment

by:oxygen_728
ID: 16946239
I'm sorry, I thought i included that.
in.open(Path.c_str(),ios::in | ios::binary);

I got pretty frustrated looking through the archived questions on EE - almost EVERYBODY that had a problem similar to mine wasn't using the ios::binary flag.


I'm trying to use the ReadFile technique now.... but I'm getting some strange error codes.

It may be a hardware issue... I'm using an external USB hard drive... and It currently won't let me copy files from it to an internal hard drive via Windows Explorer.

It keeps giving me a message popup in the bottom right of my taskbar:

"Delayed Write Error"
"Windows was unable to savea ll the data for the file G:.  ..... "

I've got to be social for a few hours, I'll start hacking away at it when I return.

Thanks for the tip about the windows ReadFile() thing... the error codes it provides are very informative.

0
 
LVL 14

Assisted Solution

by:wayside
wayside earned 50 total points
ID: 16946320
If you don't open using binary mode, when you read a ^Z (control-Z, decimal byte 26) it will be interpreted as the end of file marker.  This might be why your read fails at a certain point - look at your data file in a hex editor and see if there is a ^Z near the spot where the read fails.

In addition, <CR><LF> pairs are translated to a single <LF>, which is probably not what you want either.

> I'm hoping to just read it all in at once and access it in an array

If what you want is to access your file as if it were an array, you might consider opening it as a memory-mapped file. This frees you from doing any input yourself, and you can treat the file exactly as an array.
0
 

Author Comment

by:oxygen_728
ID: 16946537
wayside: Sorry I wasn't clear above... I'm opening as binary
wayside: Thanks for the tip about a memory mapped file - I wonder if it's faster to use a memory mapped file or to simply read an entire file into an array and index it via an array?

0
 
LVL 86

Expert Comment

by:jkr
ID: 16946561
There is no difference in reading the file with one call compared to opening it as a memory mapped file, the result is the same.
0
 

Author Comment

by:oxygen_728
ID: 16946693
ok thanks, as soon as I get back home I'll hack away at it
0
 
LVL 17

Expert Comment

by:rstaveley
ID: 16947018
> It may be a hardware issue

You might want to check out the Microsoft patch recommended in the following thread: http://www.short-media.com/forum/archive/index.php/t-10625
0
Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

 

Author Comment

by:oxygen_728
ID: 16947074
Hey i'll try that out when I get home. It'd be nice if that fixed the problem... I also suspect the directory the files are stored in is corrupt after reading that thread. I will have to "scan-disk" it.

0
 
LVL 14

Expert Comment

by:wayside
ID: 16947719
>  There is no difference in reading the file with one call compared to opening it as a memory mapped file,

I don't think this is necessarily true, it can be faster, it depends on what else is going on in the program, how the memory is accessed, and how much can go on in the background. At worst, it will be no different than reading the entire file in one read, but it may be faster. You'd have to benchmark the app to see.

0
 

Author Comment

by:oxygen_728
ID: 16947972
I kinda figured they'd be about equal
0
 

Author Comment

by:oxygen_728
ID: 16948232
Ok I solved the problem by enabling write caching on my hard drive... which seems counterintuitive... but when you're doing with millions of lines of code and two separate things like USB and HDD transfer are expected to work properly on the first try, who knows what works?

Thanks for the tips
0
 

Author Comment

by:oxygen_728
ID: 16948235
Again, to restate for anybody stumbling on this thread:

My problem was an incorrectly working hard drive.

The most common solution to this problem is to verify that you are opening the file with ios::binary

0
 
LVL 17

Expert Comment

by:rstaveley
ID: 16948948
Enabling write caching sounds counterintuitive as you say. If I was you (and could spare the time to do it), I'd follow this up in the hardware TAs here at EE, and see what the search facility comes up with on "delayed write error" - i.e.

http://search.experts-exchange.com/search.jsp?query=%22delayed+write+error%22&searchType=all

Here's my guess: Write-caching is now causing your HDD to need to do fewer seeks for writing and whatever the fundamental problem is that you have, write caching is making it occur less often. Defragmentation may help too. My guess is that during the delayed write error event, I/O is blocked and reads fail. It would be better if you could address the fundamental problem. The hardware experts are the ones to ask, but certainly you should take a look at your event log to see if you are still getting delayed write errors albeit less often.
0
 

Author Comment

by:oxygen_728
ID: 16949598
Ok will do thanks rstaveley, much appreciated
0

Featured Post

Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
C++ Class Serialization, Encapsulation and Formatted Output 8 54
JItbit AD intergration 4 76
Denali Software 2 45
Safe conversion? 4 58
A list of useful business intelligence software.
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.

895 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now