• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 574
  • Last Modified:

Bytes read during an Asychronous File Read?

I want to start processing on a data from a file while it is being loaded. I figure the only way for me to do this is with an asynchronous file read, but I can't seem to find a way to determine how much of the file has been loaded. If I know how much is loaded, I can limit my processing to the data that is loaded and not worry about processing data locations before they are filled. MSDN says that GetOverlappedResult() will return the number of bytes transferred, but in practice the call fails during the read and returns a byte value only when the read is completed.

unsigned int array[40000000];
HANDLE inputFile;

ReadFileEx(inputFile, &array[0], 40000000, &asyncSettings, NULL);

if(GetOverlappedResult(inputFile, &asyncSettings, &bytesRead, FALSE)) {
     AfxMessageBox("It worked!");   //it never works...
//Process Data in a Loop based on the number of bytes read      

Anyone have any ideas how to do this?
2 Solutions
ReadFileEx() will not call your callback routine until the entire 40MB has been read.

The easier way to do this is to use multiple threads.   One thread reading the file
in smaller chunks, while the other thread processes the data that has been read.

I have also done this in the past using file mapping, with one thread skipping along
the file's address space reading one word of each VM page, and the other thread
processing the data that has been mapped in.

Simple, read the file as you're doing, in async mode, but read smaller chunks.
Say read in one megabyte per read request, but do 40 read requests, all in parallel.
You'll get 40 callbacks, one for each one megabyte chunk.

Or if you want to minimize the number of async requests in the air, jut have two requests going, then in the callback schedule a new one.
That way you'll always have a read request going.

The last parameter of ReadFileEx() is the callback function which will be called when async read operation is completed. So you should not just give NULL as a callback function. Define one something like this:

void WINAPI ReadCompleteCallBack( DWORD dwError, DWORD dwBytesRead, LPOVERLAPPED lpOverlapped )

and as you can see, this callback function has all information you need. Number of bytes read (which is the same with the one you give in ReadFileEx unless it's end of file), any error code and your OVERLAPPED structure.

You should create a buffer with proper size because you'll have to wait until it's filled even this is a async operation. Then you can re-ReadFileEx in your callback function for the rest of data. But be careful here, you should advance Offset field of your OVERLAPPED structure before calling further ReadFileEx() because it won't be incremented automatically. Also you should check if it's end of file, and close your handle and inform your main loop that it's over.

When you call ReadFileEx() first time (not in the callback function) you should use a alertable function such as SleepEx or MsgWaitForMultipleObjectsEx (there are more of them)...

One more tip, there is a hEvent field in OVERLAPPED structure which is free to use. You can create a simple structure which has some fields like hWnd of your parent window, EOF state etc... And you can pass it by OVERLAPPED structure. That will help you to send messages to your main window to inform the status of your async read. For example, WM_USER + 1 may be a read operation having data in LPARAM of the message and data length in WPARAM; or you can send WM_USER + 3 to inform that it's EOF... etc

Anyway, I also recommend you to use multiple threads if possible.. it would be easier and looks better...

Caner ELCI
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now