Solved

How do you write to end of files opened with FILE_FLAG_NO_BUFFERING?

Posted on 2003-03-12
8
1,306 Views
Last Modified: 2013-12-03
Files opened with FiLE_FLAG_NO_BUFFERING need to be written in chunks that are multiples of the sector size.  I have to write a huge file, but the data size is not a multiple of the sector size.  How can I write the last handful of bytes?  I've looked into write a full block and then truncating the file, but I don't see how to do that either since SetFilePointer requires that you set the file pointer to sector boundaries.
0
Comment
Question by:danteleone
8 Comments
 
LVL 49

Expert Comment

by:DanRollins
ID: 8126858
1) Write the file, including the slack at the end.  But make a note of the 'actual' end of file.
2) Close it.  
3) Reopen it normally (without FiLE_FLAG_NO_BUFFERING).
4) Seek to the remembered 'actual' end of file.
5) Call SetEndOfFile

There ya do!
-- Dan
0
 

Author Comment

by:danteleone
ID: 8128067
Thanks, but the problem is that the files that I'm working with are huge (this one is 180 GB), and I'm hitting a hard limit in Windows on the total size of files opened with buffering (150 GB), because the kernel runs out of memory maintaining its cache data structure.  So I can't open the file normally to truncate it.
0
 
LVL 49

Expert Comment

by:DanRollins
ID: 8131468
You are writing a 150 GB file and worrying about the max extra 511 bytes at the end?

What if you access if for GENERIC_WRITE?  Does the system choke in that case?

-- Dan
0
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

 
LVL 1

Expert Comment

by:Hermetic
ID: 8141761
If you aren't reading from the file, then you can
use FILE_FLAG_WRITE_THROUGH, this disables write buffering
but I think lifts the sector write limitation.
0
 

Author Comment

by:danteleone
ID: 8155035
Thanks, but the problem is that the files that I'm working with are huge (this one is 180 GB), and I'm hitting a hard limit in Windows on the total size of files opened with buffering (150 GB), because the kernel runs out of memory maintaining its cache data structure.  So I can't open the file normally to truncate it.
0
 

Author Comment

by:danteleone
ID: 8155054
Ignore my duplicate last comment...

I didn't get a chance to try the FILE_FLAG_WRITE_THROUGH flag, but the documentation isn't promising, since it does say that caching can still take place.

In any event, I believe I've found a solution: the ntdll call NtSetInformationFile allows you to explicitly set the file size.  So I can write a final full sector, and then truncate using that.
0
 
LVL 49

Expert Comment

by:DanRollins
ID: 8155300
It sound as if you have solved this yourself.  Please look here for instructions on how to procede:
   http://www.apollois.com/EE/Help/Closing_Questions.htm#Refund
-- Dan
0
 

Accepted Solution

by:
SpideyMod earned 0 total points
ID: 8157103
PAQ'd and all 75 points refunded.

SpideyMod
Community Support Moderator @Experts Exchange
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

zlib is a free compression library (a DLL) on which the popular gzip utility is built.  In this article, we'll see how to use the zlib functions to compress and decompress data in memory; that is, without needing to use a temporary file.  We'll be c…
A theme is a collection of property settings that allow you to define the look of pages and controls, and then apply the look consistently across pages in an application. Themes can be made up of a set of elements: skins, style sheets, images, and o…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question