Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

How do you write to end of files opened with FILE_FLAG_NO_BUFFERING?

Posted on 2003-03-12
8
Medium Priority
?
1,399 Views
Last Modified: 2013-12-03
Files opened with FiLE_FLAG_NO_BUFFERING need to be written in chunks that are multiples of the sector size.  I have to write a huge file, but the data size is not a multiple of the sector size.  How can I write the last handful of bytes?  I've looked into write a full block and then truncating the file, but I don't see how to do that either since SetFilePointer requires that you set the file pointer to sector boundaries.
0
Comment
Question by:danteleone
8 Comments
 
LVL 49

Expert Comment

by:DanRollins
ID: 8126858
1) Write the file, including the slack at the end.  But make a note of the 'actual' end of file.
2) Close it.  
3) Reopen it normally (without FiLE_FLAG_NO_BUFFERING).
4) Seek to the remembered 'actual' end of file.
5) Call SetEndOfFile

There ya do!
-- Dan
0
 

Author Comment

by:danteleone
ID: 8128067
Thanks, but the problem is that the files that I'm working with are huge (this one is 180 GB), and I'm hitting a hard limit in Windows on the total size of files opened with buffering (150 GB), because the kernel runs out of memory maintaining its cache data structure.  So I can't open the file normally to truncate it.
0
 
LVL 49

Expert Comment

by:DanRollins
ID: 8131468
You are writing a 150 GB file and worrying about the max extra 511 bytes at the end?

What if you access if for GENERIC_WRITE?  Does the system choke in that case?

-- Dan
0
Learn Veeam advantages over legacy backup

Every day, more and more legacy backup customers switch to Veeam. Technologies designed for the client-server era cannot restore any IT service running in the hybrid cloud within seconds. Learn top Veeam advantages over legacy backup and get Veeam for the price of your renewal

 
LVL 1

Expert Comment

by:Hermetic
ID: 8141761
If you aren't reading from the file, then you can
use FILE_FLAG_WRITE_THROUGH, this disables write buffering
but I think lifts the sector write limitation.
0
 

Author Comment

by:danteleone
ID: 8155035
Thanks, but the problem is that the files that I'm working with are huge (this one is 180 GB), and I'm hitting a hard limit in Windows on the total size of files opened with buffering (150 GB), because the kernel runs out of memory maintaining its cache data structure.  So I can't open the file normally to truncate it.
0
 

Author Comment

by:danteleone
ID: 8155054
Ignore my duplicate last comment...

I didn't get a chance to try the FILE_FLAG_WRITE_THROUGH flag, but the documentation isn't promising, since it does say that caching can still take place.

In any event, I believe I've found a solution: the ntdll call NtSetInformationFile allows you to explicitly set the file size.  So I can write a final full sector, and then truncate using that.
0
 
LVL 49

Expert Comment

by:DanRollins
ID: 8155300
It sound as if you have solved this yourself.  Please look here for instructions on how to procede:
   http://www.apollois.com/EE/Help/Closing_Questions.htm#Refund
-- Dan
0
 

Accepted Solution

by:
SpideyMod earned 0 total points
ID: 8157103
PAQ'd and all 75 points refunded.

SpideyMod
Community Support Moderator @Experts Exchange
0

Featured Post

Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

zlib is a free compression library (a DLL) on which the popular gzip utility is built.  In this article, we'll see how to use the zlib functions to compress and decompress data in memory; that is, without needing to use a temporary file.  We'll be c…
For most people, the WrapPanel seems like a magic when they switch from WinForms to WPF. Most of us will think that the code that is used to write a control like that would be difficult. However, most of the work is done by the WPF engine, and the W…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
Screencast - Getting to Know the Pipeline

926 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question