We help IT Professionals succeed at work.

Problem When Reading Large Files With MapViewOfFile

ismarosmanovic
on
6,227 Views
Last Modified: 2008-06-02
Hello,
I have problems when trying to read files with MapViewOfFile API function. Actually I only have problems when I try to read very large file. Here is what I am doing;

long offsetHight = 0;
long offsetLow = 0;
long positionAdjustment = 0;
int startpointer;
NativeMethods.SYSTEM_INFO syInfo = new NativeMethods.SYSTEM_INFO();
NativeMethods.GetSystemInfo(ref syInfo);
offSettemp = (100000);
positionAdjustment = (offSettemp % syInfo.dwAllocationGranularity);
if (positionAdjustment != 0)
offSettemp = offSettemp - positionAdjustment;
offsetHight = ((offSettemp >> 32) & 0xFFFFFFFF);
offsetLow = (offSettemp & 0xFFFFFFFF);

IntPtr mapFile = NativeMethods.CreateFileMapping(objectHandle.DangerousGetHandle(), IntPtr.Zero, NativeMethods.PageProtection.ReadWrite, 0, 0, string.Empty);
IntPtr viewFile = NativeMethods.MapViewOfFile(mapFile, 0x004, (int)offsetHight, (int)offsetLow, 8192);

Above code returns the value for "viewFile" handle only when I map up to 1.9 GB but when I try to set the offset at (i.e 10 milion) the file handle always return zero and GetLastError is 8 ERROR_NOT_ENOUGH_MEMORY

Please advice....

Please advice.....
Comment
Watch Question

>>>> file handle always return zero and GetLastError is 8 ERROR_NOT_ENOUGH_MEMORY

MapViewOfFile creates shared memory which is limited to physical memory as far as I know. Even if it would be virtual memory it is limited to 4GB or the size of the page file - memory already used by Windows, other processes and the current process.

Regards, Alex
>>>> Even if it would be virtual memory it is limited to 4GB

moreover it needs to be contiguous memory, so the limitation is far below 4GB. The offset needs to be counted to the amount cause it must be reserved for further allocations.

Regards, Alex

Author

Commented:
Hello Alex,
Thank you for quick replay, but I could not get you? Are you saying there is no way to read files above 4 GB or?
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Hello again,
maybe it was not clear from my question, what I am trying to do is to read only small segment of the very large file, for example to read only 8 kb at 10 millionth offset. Can I do it from the above code?
jkr
CERTIFIED EXPERT
Top Expert 2012

Commented:
In your call to 'CresteFileMapping()', specify that small segment (or multiples of it) as the mapping object size. Using '0' here means a mapping object of the size of the whole fuile. E.g. like

IntPtr mapFile = NativeMethods.CreateFileMapping(objectHandle.DangerousGetHandle(), IntPtr.Zero, NativeMethods.PageProtection.ReadWrite, 0, 4 * 8192, string.Empty);

Author

Commented:
Hello jrk,
It does not work result is agian 0 (zero).....This is what I did;

IntPtr mapFile = NativeMethods.CreateFileMapping(objectHandle.DangerousGetHandle(), IntPtr.Zero, NativeMethods.PageProtection.ReadWrite, 0, 8192, string.Empty);
IntPtr viewFile = NativeMethods.MapViewOfFile(mapFile, 0x004, offsetHight, offsetLow, 8192);

here offsetLow and offsetHight points to the size above 2^32 and if I multiply 8 kb with desirable address then again error(8) is thrown. Simply I want to read last 8KB of file that sizes 45 GB....

Thank you
                              
>>>> Simply I want to read last 8KB of file that sizes 45 GB....

You could try the following though you would need to translate it to managed C++.

- open the file using fstream
- set position to file end - 8kb

#include <iostream>
#include <fstream>
using namespace std;

int main ( )
{
   fstream file;
   file.open( "huge.dat", ios_base::in);
   // set read position
   file.seekg( -8192, ios_base::end );
   char buffer[8192];
   if (file.read(buffer, 8192))
   {
        // ok
   }

Regards, Alex
AxterSenior Software Engineer
CERTIFIED EXPERT

Commented:
What happens if you try the following:
IntPtr mapFile = NativeMethods.CreateFileMapping(objectHandle.DangerousGetHandle(), IntPtr.Zero, NativeMethods.PageProtection.ReadWrite, 0, 0, string.Empty);
IntPtr viewFile = NativeMethods.MapViewOfFile(mapFile, 0x004, offsetHight, offsetLow, 0);
AxterSenior Software Engineer
CERTIFIED EXPERT

Commented:
Do you have enough disk space for this operation?
What's the size of your virtual memory?
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Commented:
Found this just now:

http://www.brianmadden.com/content/content.asp?ID=69

Just don't map the file.

Author

Commented:
Ok here is what I am doing;
I have application that is in the process with another running process. The other process open some files exclusively and my application needs to read some data from these files. What I do is; first duplicate the file handle and then map it to the file view and then read the buffer. I can do this until I need to map huge files and then the error is shown. CreateFileMapping function always gives me some handle(IntPtr) but MapViewOfFile does not. Can anybody advice if there is any other way to handle and exctract the buffer from these duplicated file handles?

After some more research it comes that Alex and mugman21 are right about MapViewOfFile? So I will need to accept their advices....

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.