Solved

Bytes read during an Asychronous File Read?

Posted on 2004-04-09
3
543 Views
Last Modified: 2008-02-01
I want to start processing on a data from a file while it is being loaded. I figure the only way for me to do this is with an asynchronous file read, but I can't seem to find a way to determine how much of the file has been loaded. If I know how much is loaded, I can limit my processing to the data that is loaded and not worry about processing data locations before they are filled. MSDN says that GetOverlappedResult() will return the number of bytes transferred, but in practice the call fails during the read and returns a byte value only when the read is completed.

unsigned int array[40000000];
HANDLE inputFile;

inputFile = CreateFile("test.file", GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN | FILE_FLAG_OVERLAPPED, NULL);
ReadFileEx(inputFile, &array[0], 40000000, &asyncSettings, NULL);

if(GetOverlappedResult(inputFile, &asyncSettings, &bytesRead, FALSE)) {
     AfxMessageBox("It worked!");   //it never works...
}
//Process Data in a Loop based on the number of bytes read      

Anyone have any ideas how to do this?
0
Comment
Question by:noble24
3 Comments
 
LVL 23

Expert Comment

by:brettmjohnson
ID: 10794667
ReadFileEx() will not call your callback routine until the entire 40MB has been read.

The easier way to do this is to use multiple threads.   One thread reading the file
in smaller chunks, while the other thread processes the data that has been read.

I have also done this in the past using file mapping, with one thread skipping along
the file's address space reading one word of each VM page, and the other thread
processing the data that has been mapped in.

0
 
LVL 22

Assisted Solution

by:grg99
grg99 earned 150 total points
ID: 10794737
Simple, read the file as you're doing, in async mode, but read smaller chunks.
Say read in one megabyte per read request, but do 40 read requests, all in parallel.
You'll get 40 callbacks, one for each one megabyte chunk.

Or if you want to minimize the number of async requests in the air, jut have two requests going, then in the callback schedule a new one.
That way you'll always have a read request going.

0
 
LVL 4

Accepted Solution

by:
caner_elci earned 200 total points
ID: 10795141
The last parameter of ReadFileEx() is the callback function which will be called when async read operation is completed. So you should not just give NULL as a callback function. Define one something like this:

void WINAPI ReadCompleteCallBack( DWORD dwError, DWORD dwBytesRead, LPOVERLAPPED lpOverlapped )

and as you can see, this callback function has all information you need. Number of bytes read (which is the same with the one you give in ReadFileEx unless it's end of file), any error code and your OVERLAPPED structure.

You should create a buffer with proper size because you'll have to wait until it's filled even this is a async operation. Then you can re-ReadFileEx in your callback function for the rest of data. But be careful here, you should advance Offset field of your OVERLAPPED structure before calling further ReadFileEx() because it won't be incremented automatically. Also you should check if it's end of file, and close your handle and inform your main loop that it's over.

When you call ReadFileEx() first time (not in the callback function) you should use a alertable function such as SleepEx or MsgWaitForMultipleObjectsEx (there are more of them)...

One more tip, there is a hEvent field in OVERLAPPED structure which is free to use. You can create a simple structure which has some fields like hWnd of your parent window, EOF state etc... And you can pass it by OVERLAPPED structure. That will help you to send messages to your main window to inform the status of your async read. For example, WM_USER + 1 may be a read operation having data in LPARAM of the message and data length in WPARAM; or you can send WM_USER + 3 to inform that it's EOF... etc

Anyway, I also recommend you to use multiple threads if possible.. it would be easier and looks better...

Caner ELCI
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Errors will happen. It is a fact of life for the programmer. How and when errors are detected have a great impact on quality and cost of a product. It is better to detect errors at compile time, when possible and practical. Errors that make their wa…
Templates For Beginners Or How To Encourage The Compiler To Work For You Introduction This tutorial is targeted at the reader who is, perhaps, familiar with the basics of C++ but would prefer a little slower introduction to the more ad…
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
The goal of the video will be to teach the user the difference and consequence of passing data by value vs passing data by reference in C++. An example of passing data by value as well as an example of passing data by reference will be be given. Bot…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now