ismarosmanovic
asked on
Problem When Reading Large Files With MapViewOfFile
Hello,
I have problems when trying to read files with MapViewOfFile API function. Actually I only have problems when I try to read very large file. Here is what I am doing;
long offsetHight = 0;
long offsetLow = 0;
long positionAdjustment = 0;
int startpointer;
NativeMethods.SYSTEM_INFO syInfo = new NativeMethods.SYSTEM_INFO(
NativeMethods.GetSystemInf
offSettemp = (100000);
positionAdjustment = (offSettemp % syInfo.dwAllocationGranula
if (positionAdjustment != 0)
offSettemp = offSettemp - positionAdjustment;
offsetHight = ((offSettemp >> 32) & 0xFFFFFFFF);
offsetLow = (offSettemp & 0xFFFFFFFF);
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr viewFile = NativeMethods.MapViewOfFil
Above code returns the value for "viewFile" handle only when I map up to 1.9 GB but when I try to set the offset at (i.e 10 milion) the file handle always return zero and GetLastError is 8 ERROR_NOT_ENOUGH_MEMORY
Please advice....
Please advice.....
I have problems when trying to read files with MapViewOfFile API function. Actually I only have problems when I try to read very large file. Here is what I am doing;
long offsetHight = 0;
long offsetLow = 0;
long positionAdjustment = 0;
int startpointer;
NativeMethods.SYSTEM_INFO syInfo = new NativeMethods.SYSTEM_INFO(
NativeMethods.GetSystemInf
offSettemp = (100000);
positionAdjustment = (offSettemp % syInfo.dwAllocationGranula
if (positionAdjustment != 0)
offSettemp = offSettemp - positionAdjustment;
offsetHight = ((offSettemp >> 32) & 0xFFFFFFFF);
offsetLow = (offSettemp & 0xFFFFFFFF);
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr viewFile = NativeMethods.MapViewOfFil
Above code returns the value for "viewFile" handle only when I map up to 1.9 GB but when I try to set the offset at (i.e 10 milion) the file handle always return zero and GetLastError is 8 ERROR_NOT_ENOUGH_MEMORY
Please advice....
Please advice.....
>>>> Even if it would be virtual memory it is limited to 4GB
moreover it needs to be contiguous memory, so the limitation is far below 4GB. The offset needs to be counted to the amount cause it must be reserved for further allocations.
Regards, Alex
moreover it needs to be contiguous memory, so the limitation is far below 4GB. The offset needs to be counted to the amount cause it must be reserved for further allocations.
Regards, Alex
ASKER
Hello Alex,
Thank you for quick replay, but I could not get you? Are you saying there is no way to read files above 4 GB or?
Thank you for quick replay, but I could not get you? Are you saying there is no way to read files above 4 GB or?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hello again,
maybe it was not clear from my question, what I am trying to do is to read only small segment of the very large file, for example to read only 8 kb at 10 millionth offset. Can I do it from the above code?
maybe it was not clear from my question, what I am trying to do is to read only small segment of the very large file, for example to read only 8 kb at 10 millionth offset. Can I do it from the above code?
In your call to 'CresteFileMapping()', specify that small segment (or multiples of it) as the mapping object size. Using '0' here means a mapping object of the size of the whole fuile. E.g. like
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr mapFile = NativeMethods.CreateFileMa
ASKER
Hello jrk,
It does not work result is agian 0 (zero).....This is what I did;
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr viewFile = NativeMethods.MapViewOfFil
here offsetLow and offsetHight points to the size above 2^32 and if I multiply 8 kb with desirable address then again error(8) is thrown. Simply I want to read last 8KB of file that sizes 45 GB....
Thank you
It does not work result is agian 0 (zero).....This is what I did;
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr viewFile = NativeMethods.MapViewOfFil
here offsetLow and offsetHight points to the size above 2^32 and if I multiply 8 kb with desirable address then again error(8) is thrown. Simply I want to read last 8KB of file that sizes 45 GB....
Thank you
>>>> Simply I want to read last 8KB of file that sizes 45 GB....
You could try the following though you would need to translate it to managed C++.
- open the file using fstream
- set position to file end - 8kb
#include <iostream>
#include <fstream>
using namespace std;
int main ( )
{
fstream file;
file.open( "huge.dat", ios_base::in);
// set read position
file.seekg( -8192, ios_base::end );
char buffer[8192];
if (file.read(buffer, 8192))
{
// ok
}
Regards, Alex
You could try the following though you would need to translate it to managed C++.
- open the file using fstream
- set position to file end - 8kb
#include <iostream>
#include <fstream>
using namespace std;
int main ( )
{
fstream file;
file.open( "huge.dat", ios_base::in);
// set read position
file.seekg( -8192, ios_base::end );
char buffer[8192];
if (file.read(buffer, 8192))
{
// ok
}
Regards, Alex
What happens if you try the following:
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr viewFile = NativeMethods.MapViewOfFil
IntPtr mapFile = NativeMethods.CreateFileMa
IntPtr viewFile = NativeMethods.MapViewOfFil
Do you have enough disk space for this operation?
What's the size of your virtual memory?
What's the size of your virtual memory?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ok here is what I am doing;
I have application that is in the process with another running process. The other process open some files exclusively and my application needs to read some data from these files. What I do is; first duplicate the file handle and then map it to the file view and then read the buffer. I can do this until I need to map huge files and then the error is shown. CreateFileMapping function always gives me some handle(IntPtr) but MapViewOfFile does not. Can anybody advice if there is any other way to handle and exctract the buffer from these duplicated file handles?
After some more research it comes that Alex and mugman21 are right about MapViewOfFile? So I will need to accept their advices....
I have application that is in the process with another running process. The other process open some files exclusively and my application needs to read some data from these files. What I do is; first duplicate the file handle and then map it to the file view and then read the buffer. I can do this until I need to map huge files and then the error is shown. CreateFileMapping function always gives me some handle(IntPtr) but MapViewOfFile does not. Can anybody advice if there is any other way to handle and exctract the buffer from these duplicated file handles?
After some more research it comes that Alex and mugman21 are right about MapViewOfFile? So I will need to accept their advices....
MapViewOfFile creates shared memory which is limited to physical memory as far as I know. Even if it would be virtual memory it is limited to 4GB or the size of the page file - memory already used by Windows, other processes and the current process.
Regards, Alex