Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 472
  • Last Modified:

>2 Gb filesize

I have written a program that uses ~3.3 Gb of ram. It takes a long long time to create the lookup table and I would like to save this to disk! Except I can't at the moment.

How do I get my program to write files greater than two 2Gb. I have written my program in C/C++. Is there an easy way to do this or do I need to patch the kernal ?

Please feel free to help! I do need an idiots guide (sorry).

Thanks

Will
0
wjryder
Asked:
wjryder
  • 6
  • 5
  • 4
  • +1
1 Solution
 
packratt_jkCommented:
it's not a kernel thing - it's a filesystem thing.
what filesystem are you using?.
0
 
ahoffmannCommented:
I'm just fiddling arround with a similar problem. I'm trying to upgrade with kernel 2.4.9.
Will be back if I've it working ...
0
 
wjryderAuthor Commented:
The filesystem is ext2
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

 
bryanhCommented:
Just some terminology corrections:  "it's not a kernel thing it's a filesystem thing".  It's a kernel thing.  Filesystem drivers are part of the kernel.  Even if they're added to the kernel after boot (with insmod).  Even if they're distributed separately from Linux.

And the filesystem isn't ext2.  The filesystem _type_ is ext2.  A filesystem is an ordered collection of stored bytes.  As in "I put the filesystem on a partition on my Seagate drive."

I know it's very common in Linux circles to use the word "filesystem" to mean about 5 different things, including filesystem type, but it's less confusing for everyone if we don't.

I can never remember the current state of >2GB file capability, but  it definitely exists if you get the right level of Linux , compile options, open options, or whatever.  So I hope ahofffman finds the correct incantation and gets back to us.

At work, I'm developing a Linux filesystem driver for huge filesystems, so I need to know this cold eventually!

0
 
ahoffmannCommented:
just to avoid that someone waits for useless comments:
  - my problems occour on filesystem of type ext2
  - the problem is that I cannot delete files > 2GB
    and for exaple Samba (smbd) hangs when performing such files
.. still on work ..
0
 
bryanhCommented:
I checked, and Linux shouldn't have any problem writing >2GB files; you write them the same way you write <2GB files.  I presume from the question that you tried and failed, so if you could give details on the failure, maybe I could figure it out.

In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB (it apparently does this because it isn't sure you know how to handle a >2GB file).  To open such a file, you use open64().  But there shouldn't be any problem creating a >2GB file with a regular open() or on Linux 2.2.

ext2 has a file size restriction based on the block size, as for 1K blocks it's a little over 16GB.
0
 
ahoffmannCommented:
> In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB  ..
Does this mean that a simple patch of libc.* should work?
if so, which version?
0
 
ahoffmannCommented:
0
 
bryanhCommented:
"Work" to accomplish what?  You mean to use programs written and compiled for <2GB files on >2GB files?

In that case, a simple patch to libc would do it.  You just have to add the O_LARGEFILE flag to the open flags.  All versions of the GNU C library would need this patch.

But of course, this would only work insofar as the program in question can deal with file sizes and offsets that don't fit in a 32 bit signed integer.  A program that depends on <2GB files will fail in strange ways, since it will be using library functions with 32 bit parameters/return values.

And if you have the source code and know that your program can handle 64 bit file positions/sizes, then you don't need a patch -- just code the program with open64() instead of open().
0
 
ahoffmannCommented:
hmm, yes that's what I understand in the link povided in my previous comment, SO this should be a solution for wjryder
's question.
0
 
bryanhCommented:
Thanks, that was an interesting read.  It doesn't do a very good job of explaining how to use >2GB files, though.  

First of all, it mentions using -D gcc options.  Unless you're playing around, this is the wrong way to make your program work with >2GB files.  The macros in question should be set with #define statements in the source code (before the C library #include statements).  They declare characteristics of the source code, not of the compilation.  

Also, _LARGE_FILE_SOURCE isn't related to this.  You don't need it.

Finally it doesn't make it clear that there are two distinct methods:

  1) #define _FILE_OFFSET_BITS 64.  With this, you can use all the traditionally named functions (open, fseek, etc), but get the new 64 bit versions of them instead of the 32 bit versions they used to be.

  2) #define _LARGE_FILE_SOURCE.  With this, new 64-bit versions of the functions (open64, fseek64, etc.) exist, but the old functions remain 32 bits.  This might be useful if you had some files that need the 32 bit interface and some that need the 64 bit interface.

Since the web page implied >2GB files are impossible in Linux 2.2, even though I saw plenty of source code to do it, I investigated a little further and found that while the general file system is capable, the ext2 filesystem driver rejects seeks past 2GB (and presumably other attempts to work with >2GB files).

2.4 is all OK, though.
0
 
ahoffmannCommented:
hey bryanh, should post some points for you, thanks for these explanations ;-)
0
 
wjryderAuthor Commented:
I have tried to use fopen64. But am failing :

Hence have written an example program - (Hopefully error free!) - Can you please explain how to compile this ?

The program will create a file 1 byte bigger than the file limit size (I think)

Hence if it does not crash problem solved!!

#include <stdio.h>

int main()
{

FILE *fp;

fp = fopen64("test.dat","wt");

short int Value;
short int *pValue;

pValue = &Value;

Value = 2;

for (int x = 0; x < 1073741825; x++)
{
fwrite(pvalue, sizeof(short int), 1, fp);
}

fclose(fp);

return 0;

}
0
 
wjryderAuthor Commented:
I have spotted an error :

fwrite(pvalue, sizeof(short int), 1, fp);

should read:

fwrite(pValue, sizeof(short int), 1, fp);
0
 
bryanhCommented:
I had some serious typos in my earlier posting.  The macro you don't need is _LARGEFILE_SOURCE and the macro you do need is _LARGEFILE64_SOURCE  .

If you add #define _LARGEFILE64_SOURCE as the first line of your program, it should work.  I used the command

  gcc -x c++ -Wall -Werror bigfile

By the way, if you want a faster test, don't write 2GB of data.  Just write one byte after the 2GB position, with fseeko64(fp, 1ULL<<31); fwrite("X",1,1,fp);  The system will fill in the earlier bytes with zeroes (and won't use 2GB of disk space for them, since it can represent an entire block of zeroes with a single bit).
0
 
wjryderAuthor Commented:
It works for me now.

BTW I am using RED HAT 7.1 download from the web
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 6
  • 5
  • 4
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now