JohnSantaFe
asked on
Problem using seekg with a large data file
I have a 6GB data file that I need to read. Many times I do not need to read from the begining so I use seekg to jump to the byte that I need. This save a huge amount of time. The bad news is it seems seekg only accepts an int, not even a long int. I tried using a two jump method where I first jum to INT_MAX and then try to jump using the current position with an additional amount. Any idea why the second method doesn't seem to work? Any ideas on how to get around this problem and seek to a value greater than INT_MAX?
<snip>
if ( startIndex > INT_MAX ){
laserTagStream.seekg(INT_M AX);
if ( laserTagStream.fail() ){
cout << "Error - could not seek forward using int_max " << endl;
return false;
}
laserTagStream.seekg((int) (startInde x - INT_MAX), ios_base::cur);
if ( laserTagStream.fail() ){
cout << "Error - could not seek forward using int_max second part" << endl;
return false;
}
<snip>
When my starting index is greater than 2,147,483,647 I get the message:
Error - could not seek forward using int_max second part
Thanks.
<snip>
if ( startIndex > INT_MAX ){
laserTagStream.seekg(INT_M
if ( laserTagStream.fail() ){
cout << "Error - could not seek forward using int_max " << endl;
return false;
}
laserTagStream.seekg((int)
if ( laserTagStream.fail() ){
cout << "Error - could not seek forward using int_max second part" << endl;
return false;
}
<snip>
When my starting index is greater than 2,147,483,647 I get the message:
Error - could not seek forward using int_max second part
Thanks.
The problem is as annoying as simple - you simply won't be able to handle files that are larger than 2,147,483,647 bytes with a 32bit STL. If you need to do that, you could use the Win32 file API (if you are on Windows), which can handle files up to 4TB.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Of course you could get a 64 bit OS, if money is no object
>>>> I dont think it is possible.
Hi Manish, you made valid suggestions for Windows and Linux. Why do you think it is *not* possible?
Regards, Alex
Hi Manish, you made valid suggestions for Windows and Linux. Why do you think it is *not* possible?
Regards, Alex
sorry for my bad english.
i meant not possible using fstream.
regards
Manish Regmi
i meant not possible using fstream.
regards
Manish Regmi
use seekpos() rather than seekg()
seekpos uses long
in STL library <iosfwd>
1. typedef long streamoff;
2. typedef streamoff off_type;
<fstream> has
3. virtual pos_type seekoff(off_type _O, ios_base::seekdir _Way,
ios_base::openmode =
(ios_base::openmode)(ios_b ase::in | ios_base::out))
{fpos_t _Fp;
if (_File == 0 || fseek(_File, _O, _Way) != 0
|| fgetpos(_File, &_Fp) != 0)
return (pos_type(_BADOFF));
return (pos_type(_State, _Fp)); }
so u can definitely use seekoff() using STL
since off_type is long i hope u can use it .
seekpos uses long
in STL library <iosfwd>
1. typedef long streamoff;
2. typedef streamoff off_type;
<fstream> has
3. virtual pos_type seekoff(off_type _O, ios_base::seekdir _Way,
ios_base::openmode =
(ios_base::openmode)(ios_b
{fpos_t _Fp;
if (_File == 0 || fseek(_File, _O, _Way) != 0
|| fgetpos(_File, &_Fp) != 0)
return (pos_type(_BADOFF));
return (pos_type(_State, _Fp)); }
so u can definitely use seekoff() using STL
since off_type is long i hope u can use it .
seekg has two overloaded functions one with pos_type and another with off_type
i feel u used one.
// basic_istream<charT,traits
use this
// basic_istream<charT,traits
>>use seekpos() rather than seekg() -mystatement read it as
use seekoff() than seekg()
else use seekg() with off stype
use seekoff() than seekg()
else use seekg() with off stype
Alas sizeof(off_type) == sizeof(long) == sizeof(int) == 4 in Win32 and 32-bit Linux.
You want sizeof(off_type) == sizeof (long long)... or simply to use a 64 bit system.
You want sizeof(off_type) == sizeof (long long)... or simply to use a 64 bit system.
ASKER
Thanks for the responses.
It looks like seekg and seekpos have the limitation because off_type is a long (not even an usigned long)
I found some C code on CodeProject that looks like it could solve the problem using a function called fseeki64.c
http://www.codeproject.com/file/64-bit_fileio.asp
I think there is a unix function called fseeko64 that also uses a different and larger data type. Unfortunately I'm using Windows.
My problem changed so that the file I'm dealing with is now slightly smaller than 4GB. So now if the position I want to seek to is less than 2GB into the file I use seekg(position). If the starting point I'm looking for is at greater than 2GB I use
seekg((position - filesize), ios_base::end)
(note, when using ios_base::end, the other argument should be negative as its the location in the file counting backward from the end)
Thanks.
It looks like seekg and seekpos have the limitation because off_type is a long (not even an usigned long)
I found some C code on CodeProject that looks like it could solve the problem using a function called fseeki64.c
http://www.codeproject.com/file/64-bit_fileio.asp
I think there is a unix function called fseeko64 that also uses a different and larger data type. Unfortunately I'm using Windows.
My problem changed so that the file I'm dealing with is now slightly smaller than 4GB. So now if the position I want to seek to is less than 2GB into the file I use seekg(position). If the starting point I'm looking for is at greater than 2GB I use
seekg((position - filesize), ios_base::end)
(note, when using ios_base::end, the other argument should be negative as its the location in the file counting backward from the end)
Thanks.
i feel winnt.h has a macro u can define it as WIN64 . check it out