Data not being written to disk - carefully placed breakpoints fix the behavior

Posted on 2009-04-24
Last Modified: 2013-12-14
I have a small program written in C++ with Win32 that writes data to the slack space of a file; it does this by writing to the data to the file with WriteFile() and then shrinking the file with SetEndOfFile() to leave the original file in tact, but still have the written data remain on disk.  By examining my hard drive with Winhex, I've confirmed that this approach works, but only in rather unusual circumstances.

Basically, whenever I run my program normally, either by itself or in the Visual Studio debugger, I can't see any changes in the data on the disk after it finishes; Windows confirms that the file itself was modified, but Winhex doesn't show the new slack space data I wrote.  However, if I start the program in the debugger with a breakpoint right after my call to WriteFile(), when the breakpoint is hit, I can see the changes on the hard drive in Winhex immediately.  I figured that the buffered data to write wasn't getting flushed out in time, so I put in a call to FlushFileBuffers() right after WriteFile(), but I was still getting the same behavior.  However, by putting in other meaningless calls to take up time, like a call to system("pause") after WriteFile(), the program successfully wrote the data to disk no matter how I ran it.

So, does anyone know what's really going on here?  How can I ensure (without using hacks like system("pause")) that my program always writes its data to disk?  Any help would be appreciated.

The code in question (that requires a breakpoint on the call to SetFilePointer()) is attached.
SetFilePointer(current_file, 0, NULL, FILE_END);

for (int i = 0; i < cluster_size / 2; ++i)

      WriteFile(current_file, "01", 2, &bytes_returned, NULL);

SetFilePointer(current_file, -cluster_size, NULL, FILE_END);


Open in new window

Question by:millerdog
    LVL 86

    Accepted Solution

    You're probably missing 'FlushFileBuffers()' - or not setting FILE_FLAG_WRITE_THROUGH with FILE_FLAG_NO_BUFFERING when calling 'CreateFile()'. Either of these should solve this issue.

    Author Comment

    I've already tried both of those separately - although I'll try them together and see if that helps.
    LVL 49

    Expert Comment

    The required pause is the clue:
    I think that your device driver is just optimized.  Before its caching logic does the physical write-to-disk, it checks down the queue a bit and sees that the file will soon be truncated, making the disk write unnecessary.
    That would be not unlike the way an optimizing complier will "optimize away" intermediate results in a calculation series, and in fact discard the entrie code sequence,  when it sees that the next statement assigns a literal value to the variable, overwriting it anyway.
    There might be an IOCTL function, or other mechanism to override the standard operations of the cache control logic, or there might not be.
    You might be able to experiment on a thumb drive in which you have turned off write-behind option ("optimize for quick removal") to see if it affects the issue.

    Author Comment

    Thanks for the advice Dan; I had disregarded optimizations as the source of the problem when I saw that optimizations were disabled in my VS project settings.

    I tried running the program on a thumb drive with "optimize for quick removal" checked, but the same things happened.  I'll take your advice and continue to search for any methods that might help me override this behavior.  In the worst case scenario, I can just have my program pause for a moment before truncating the file, but this will be a pretty big performance hit, since it operates on a lot of files in one run.

    I'll leave the question open until I come to a final solution.
    LVL 49

    Expert Comment

    Of course disk caching optimization is totally unrelated to compiler code optimization...
    For more diagnostic info, you could try:


    Author Closing Comment

    Calling FlushFileBuffers() worked.  Before, I was calling it on the handle to the file, but when I called on a handle to the drive holding the file, it worked.  Thanks for the help.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    6 Surprising Benefits of Threat Intelligence

    All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

    Several part series to implement Internet Explorer 11 Enterprise Mode
    Container Orchestration platforms empower organizations to scale their apps at an exceptional rate. This is the reason numerous innovation-driven companies are moving apps to an appropriated datacenter wide platform that empowers them to scale at a …
    The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

    794 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now