Solved

lseek64 - 8 gigabyte limit?

Posted on 2003-12-04
16
841 Views
Last Modified: 2007-12-19
How can I lseek64 to 8 gigabytes or beyond?

I am opening my file (well, raw device) with open64, and with the O_LARGEFILE flag.  My lseek64's are being called with SEEK_SET.  I am using full-fledged off64_t's, not off_t's.

lseek64 calls seem to be alright up to an 8 gigabyte barrier, at which point, the position I try to seek to seems to be getting modulused by 8 gigabytes.  i.e. 8 gigs seeks to 0, 8 gigs + 1k seeks to 1k, and so on.  At least, that is what the return values of lseek64 tell me.

I have googled all over, and google-groups'ed for solutions, but any slightly relevant discussion is about the 2- or 4-gig limit of regular lseek.

This is on a Red Hat 7 hybrid, running a modified 2.4.18 kernel (with SMP), compiled with "cc (GCC) 3.2.2 20030222".  My program runs on a reiserfs filesystem.
0
Comment
Question by:PistosChristou
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 4
  • 2
  • +1
16 Comments
 
LVL 24

Expert Comment

by:shivsa
ID: 9875535
u can check this link and see if u get any hint.
http://www.suse.de/~aj/linux_lfs.html
0
 

Author Comment

by:PistosChristou
ID: 9875969
shivsa: I've half-read, half-skimmed it, but it does not help my problem.
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 9880369
I never heard/encountred this milit for lseek64 ... may be because I never had to seek so far ...
If you are sure this is the problem, then a solution can be

while ( seek_length > 8 gb )
{
        lseek64 ( 8gb );
        seek_length = seek_length - 8gb;
}

lseek64 ( seek_length );  
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:PistosChristou
ID: 9882700
sunnycoder: I will try incremental seeking as you suggest.  For the moment, I am doing:

off64_t position_desired = (some number > 8 gigs);
off64_t actual_position = lseek64( fd, position_desired, SEEK_SET );

and then (actual_position != position_position).
0
 

Author Comment

by:PistosChristou
ID: 9885137
Okay, incremental seeking doesn't help.  If I increment in 512 MB increments, it seeks fine at 512, 1024 and 1536 MB, but hitting 2 gigs causes it to return -2 gigs as a return value, leading me to believe that off64_t is being typed as a 32-bit signed integer, as opposed to the long long that it is supposed to be.
0
 
LVL 24

Expert Comment

by:shivsa
ID: 9885289
Dear  PistosChristou,

there are lots of projects goin in LARGEFILE support areas, and this area is not yet stable.
till now linux has good stable support for files upto 4GB and but not above than that.

please check the site for new developements and all.
http://www.scyld.com/products/beowulf/software/lfs-5.html
http://www.suse.de/~aj/linux_lfs.html


good luck !
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 9886577
In that case, I am afraid you have not either used 64 interface consistently or you have not compiled the program with appropriate flags

you need to Compile your programs with "gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE".
also make sure that you use the O_LARGEFILE flag with open to operate on large files
0
 

Author Comment

by:PistosChristou
ID: 9896958
sunnycoder: I have done a search through my source code, and every relevant open is an open64, and every lseek is an lseek64.  Each open has the O_LARGEFILE flag, and my compiler flags include: "-D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64".

shivsa: Thank you for the links.

In case it is of relevance, here is my glibc version:

$ /lib/libc.so.6
GNU C Library stable release version 2.3.2, by Roland McGrath et al.
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.2.2 20030222 (Red Hat Linux 3.2.2-5).
Compiled on a Linux 2.4.20 system on 2003-02-27.
0
 

Author Comment

by:PistosChristou
ID: 9898152
Further possibly relevant information:

The program is being compiled on a different machine than the one it's being executed on.
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 9902950
>The program is being compiled on a different machine than the one it's being executed on.
try compiling and running on the same machine and post the results

> causes it to return -2 gigs as a return value,
is suspect ... I am not so sure about lseek64 but lseek returns -1 on failure and a positive value otherwise ... I believe lseek64 must not be too much different ....
0
 

Author Comment

by:PistosChristou
ID: 9904334
Compiling on the same machine doesn't bear any fruit.

> > causes it to return -2 gigs as a return value,
> is suspect ... I am not so sure about lseek64 but lseek returns -1 on failure and a positive value otherwise ... I believe lseek64 must not be too much different ....

Yes, I know, so this makes things rather, as you say, suspect.  Perhaps some high-order or low-order bits are being shifted, truncated, zeroed... who knows.  I've tried casting every relevant operand of every relevant operation, but it doesn't help.  The number being passed in is validly > 2 gigs, but the number being spit out is not.

Is it a possibility that my 64-bit calls and data types are being transformed at compile time to 32-bit versions, due to some header or library being missing or corrupt?
0
 

Author Comment

by:PistosChristou
ID: 9905630
Okay, someone else working on the project has come to the conclusion that the problem is a bug in glibc, wherein lseek64, though performing the seek correctly, does not return more than 32 bits.  A thread that we found:

http://mail.gnu.org/archive/html/bug-glibc/2003-10/msg00046.html

I guess this means my problem is unsolvable, short of a change in glibc.  :)  We are proceeding with the assumption that any non-negative-one return value of lseek64 indicates a successful seek.  I am proceeding to investigate if the seek really is successful by writing to my device in a controlled manner.
0
 

Author Comment

by:PistosChristou
ID: 9906481
Solved the problem.  I was just missing a #include <unistd.h>, and apparently, the compiler allowed it to compile anyway, presumably using function and type definitions from some other mysterious source.
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 9909853
PAQ/Refund
0
 
LVL 1

Accepted Solution

by:
Computer101 earned 0 total points
ID: 9931753
PAQed, with points refunded (200)

Computer101
E-E Admin
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…

724 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question