Solved

Problem using seekg with a large data file

Posted on 2006-07-13
12
1,013 Views
Last Modified: 2008-02-07
I have a 6GB data file that I need to read.  Many times I do not need to read from the begining so I use seekg to jump to the byte that I need.  This save a huge amount of time. The bad news is it seems seekg only accepts an int, not even a long int.  I tried using a two jump method where I first jum to INT_MAX and then try to jump using the current position with an additional amount.  Any idea why the second method doesn't seem to work?  Any ideas on how to get around this problem and seek to a value greater than INT_MAX?


<snip>
            if ( startIndex > INT_MAX ){
                  laserTagStream.seekg(INT_MAX);
                  if ( laserTagStream.fail() ){
                        cout << "Error - could not seek forward using int_max " << endl;
                        return false;
                  }
                  laserTagStream.seekg((int)(startIndex - INT_MAX), ios_base::cur);
                  if ( laserTagStream.fail() ){
                        cout << "Error - could not seek forward using int_max second part" << endl;
                        return false;
                  }
<snip>

When my starting index is greater than 2,147,483,647 I get the message:
Error - could not seek forward using int_max second part

Thanks.
0
Comment
Question by:JohnSantaFe
  • 4
  • 3
  • 2
  • +3
12 Comments
 
LVL 86

Expert Comment

by:jkr
ID: 17100748
The problem is as annoying as simple - you simply won't be able to handle files that are larger than 2,147,483,647 bytes with a 32bit STL. If you need to do that, you could use the Win32 file API (if you are on Windows), which can handle files up to 4TB.
0
 
LVL 17

Accepted Solution

by:
rstaveley earned 175 total points
ID: 17100791
If you want a custom off_type (e.g. so that sizeof(off_type) == sizeof(long long)), I think you need a custom streambuf, but I'm not sure how you'd do that because streambuf takes its traits from the character type... but don't go expecting wstreams to provide a larger off_type.
0
 
LVL 8

Assisted Solution

by:manish_regmi
manish_regmi earned 75 total points
ID: 17100858
I dont think it is possible.

you have to use api like
__int64 _lseeki64(
   int fd,
   __int64 offset,
   int origin
);

in windows

or
_llseek() system call in Linux

regards
Manish Regmi

0
 
LVL 17

Expert Comment

by:rstaveley
ID: 17100970
Of course you could get a 64 bit OS, if money is no object
0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
ID: 17101636
>>>> I dont think it is possible.

Hi Manish, you made valid suggestions for Windows and Linux. Why do you think it is *not* possible?

Regards, Alex


0
 
LVL 8

Expert Comment

by:manish_regmi
ID: 17105214
sorry for my bad english.
i meant not possible using fstream.

regards
Manish Regmi


0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 4

Expert Comment

by:havman56
ID: 17108334
use seekpos() rather than seekg()

seekpos uses long

in STL library  <iosfwd>
1. typedef long streamoff;
2. typedef streamoff off_type;

<fstream> has
3.       virtual pos_type seekoff(off_type _O, ios_base::seekdir _Way,
            ios_base::openmode =
                  (ios_base::openmode)(ios_base::in | ios_base::out))
            {fpos_t _Fp;
            if (_File == 0 || fseek(_File, _O, _Way) != 0
                  || fgetpos(_File, &_Fp) != 0)
                  return (pos_type(_BADOFF));
            return (pos_type(_State, _Fp)); }


so u can definitely use seekoff() using STL
since off_type is long i hope u can use it .



0
 
LVL 4

Expert Comment

by:havman56
ID: 17108376

seekg has two overloaded functions one with pos_type and another with off_type

i feel u used one.

// basic_istream<charT,traits>& seekg(pos_type);

use this
// basic_istream<charT,traits>& seekg(off_type, ios_base::seekdir);

0
 
LVL 4

Expert Comment

by:havman56
ID: 17108410
>>use seekpos() rather than seekg()  -mystatement read it as

use seekoff() than seekg()

else use seekg() with off stype

0
 
LVL 17

Expert Comment

by:rstaveley
ID: 17108427
Alas sizeof(off_type) == sizeof(long) == sizeof(int) == 4 in Win32 and 32-bit Linux.

You want sizeof(off_type) == sizeof (long long)... or simply to use a 64 bit system.
0
 

Author Comment

by:JohnSantaFe
ID: 17123699
Thanks for the responses.

It looks like seekg and seekpos have the limitation because off_type is a long (not even an usigned long)

I found some C code on CodeProject that looks like it could solve the problem using a function called fseeki64.c
http://www.codeproject.com/file/64-bit_fileio.asp

I think there is a unix function called fseeko64 that also uses a different and larger data type.  Unfortunately I'm using Windows.


My problem changed so that the file I'm dealing with is now slightly smaller than 4GB.  So now if the position I want to seek to is less than 2GB into the file I use  seekg(position).  If the starting point I'm looking for is at greater than 2GB I use
seekg((position - filesize), ios_base::end)

(note, when using ios_base::end, the other argument should be negative as its the location in the file counting backward from the end)

Thanks.
0
 
LVL 4

Expert Comment

by:havman56
ID: 17146889


i feel winnt.h has a macro u can define it as WIN64 . check it out

0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Unlike C#, C++ doesn't have native support for sealing classes (so they cannot be sub-classed). At the cost of a virtual base class pointer it is possible to implement a pseudo sealing mechanism The trick is to virtually inherit from a base class…
Often, when implementing a feature, you won't know how certain events should be handled at the point where they occur and you'd rather defer to the user of your function or class. For example, a XML parser will extract a tag from the source code, wh…
The goal of the video will be to teach the user the difference and consequence of passing data by value vs passing data by reference in C++. An example of passing data by value as well as an example of passing data by reference will be be given. Bot…
The viewer will be introduced to the member functions push_back and pop_back of the vector class. The video will teach the difference between the two as well as how to use each one along with its functionality.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now