Solved

>2 Gb filesize

Posted on 2001-09-05
16
460 Views
Last Modified: 2010-04-21
I have written a program that uses ~3.3 Gb of ram. It takes a long long time to create the lookup table and I would like to save this to disk! Except I can't at the moment.

How do I get my program to write files greater than two 2Gb. I have written my program in C/C++. Is there an easy way to do this or do I need to patch the kernal ?

Please feel free to help! I do need an idiots guide (sorry).

Thanks

Will
0
Comment
Question by:wjryder
  • 6
  • 5
  • 4
  • +1
16 Comments
 
LVL 3

Expert Comment

by:packratt_jk
Comment Utility
it's not a kernel thing - it's a filesystem thing.
what filesystem are you using?.
0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
I'm just fiddling arround with a similar problem. I'm trying to upgrade with kernel 2.4.9.
Will be back if I've it working ...
0
 

Author Comment

by:wjryder
Comment Utility
The filesystem is ext2
0
 
LVL 5

Expert Comment

by:bryanh
Comment Utility
Just some terminology corrections:  "it's not a kernel thing it's a filesystem thing".  It's a kernel thing.  Filesystem drivers are part of the kernel.  Even if they're added to the kernel after boot (with insmod).  Even if they're distributed separately from Linux.

And the filesystem isn't ext2.  The filesystem _type_ is ext2.  A filesystem is an ordered collection of stored bytes.  As in "I put the filesystem on a partition on my Seagate drive."

I know it's very common in Linux circles to use the word "filesystem" to mean about 5 different things, including filesystem type, but it's less confusing for everyone if we don't.

I can never remember the current state of >2GB file capability, but  it definitely exists if you get the right level of Linux , compile options, open options, or whatever.  So I hope ahofffman finds the correct incantation and gets back to us.

At work, I'm developing a Linux filesystem driver for huge filesystems, so I need to know this cold eventually!

0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
just to avoid that someone waits for useless comments:
  - my problems occour on filesystem of type ext2
  - the problem is that I cannot delete files > 2GB
    and for exaple Samba (smbd) hangs when performing such files
.. still on work ..
0
 
LVL 5

Expert Comment

by:bryanh
Comment Utility
I checked, and Linux shouldn't have any problem writing >2GB files; you write them the same way you write <2GB files.  I presume from the question that you tried and failed, so if you could give details on the failure, maybe I could figure it out.

In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB (it apparently does this because it isn't sure you know how to handle a >2GB file).  To open such a file, you use open64().  But there shouldn't be any problem creating a >2GB file with a regular open() or on Linux 2.2.

ext2 has a file size restriction based on the block size, as for 1K blocks it's a little over 16GB.
0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
> In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB  ..
Does this mean that a simple patch of libc.* should work?
if so, which version?
0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 5

Expert Comment

by:bryanh
Comment Utility
"Work" to accomplish what?  You mean to use programs written and compiled for <2GB files on >2GB files?

In that case, a simple patch to libc would do it.  You just have to add the O_LARGEFILE flag to the open flags.  All versions of the GNU C library would need this patch.

But of course, this would only work insofar as the program in question can deal with file sizes and offsets that don't fit in a 32 bit signed integer.  A program that depends on <2GB files will fail in strange ways, since it will be using library functions with 32 bit parameters/return values.

And if you have the source code and know that your program can handle 64 bit file positions/sizes, then you don't need a patch -- just code the program with open64() instead of open().
0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
hmm, yes that's what I understand in the link povided in my previous comment, SO this should be a solution for wjryder
's question.
0
 
LVL 5

Expert Comment

by:bryanh
Comment Utility
Thanks, that was an interesting read.  It doesn't do a very good job of explaining how to use >2GB files, though.  

First of all, it mentions using -D gcc options.  Unless you're playing around, this is the wrong way to make your program work with >2GB files.  The macros in question should be set with #define statements in the source code (before the C library #include statements).  They declare characteristics of the source code, not of the compilation.  

Also, _LARGE_FILE_SOURCE isn't related to this.  You don't need it.

Finally it doesn't make it clear that there are two distinct methods:

  1) #define _FILE_OFFSET_BITS 64.  With this, you can use all the traditionally named functions (open, fseek, etc), but get the new 64 bit versions of them instead of the 32 bit versions they used to be.

  2) #define _LARGE_FILE_SOURCE.  With this, new 64-bit versions of the functions (open64, fseek64, etc.) exist, but the old functions remain 32 bits.  This might be useful if you had some files that need the 32 bit interface and some that need the 64 bit interface.

Since the web page implied >2GB files are impossible in Linux 2.2, even though I saw plenty of source code to do it, I investigated a little further and found that while the general file system is capable, the ext2 filesystem driver rejects seeks past 2GB (and presumably other attempts to work with >2GB files).

2.4 is all OK, though.
0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
hey bryanh, should post some points for you, thanks for these explanations ;-)
0
 

Author Comment

by:wjryder
Comment Utility
I have tried to use fopen64. But am failing :

Hence have written an example program - (Hopefully error free!) - Can you please explain how to compile this ?

The program will create a file 1 byte bigger than the file limit size (I think)

Hence if it does not crash problem solved!!

#include <stdio.h>

int main()
{

FILE *fp;

fp = fopen64("test.dat","wt");

short int Value;
short int *pValue;

pValue = &Value;

Value = 2;

for (int x = 0; x < 1073741825; x++)
{
fwrite(pvalue, sizeof(short int), 1, fp);
}

fclose(fp);

return 0;

}
0
 

Author Comment

by:wjryder
Comment Utility
I have spotted an error :

fwrite(pvalue, sizeof(short int), 1, fp);

should read:

fwrite(pValue, sizeof(short int), 1, fp);
0
 
LVL 5

Accepted Solution

by:
bryanh earned 250 total points
Comment Utility
I had some serious typos in my earlier posting.  The macro you don't need is _LARGEFILE_SOURCE and the macro you do need is _LARGEFILE64_SOURCE  .

If you add #define _LARGEFILE64_SOURCE as the first line of your program, it should work.  I used the command

  gcc -x c++ -Wall -Werror bigfile

By the way, if you want a faster test, don't write 2GB of data.  Just write one byte after the 2GB position, with fseeko64(fp, 1ULL<<31); fwrite("X",1,1,fp);  The system will fill in the earlier bytes with zeroes (and won't use 2GB of disk space for them, since it can represent an entire block of zeroes with a single bit).
0
 

Author Comment

by:wjryder
Comment Utility
It works for me now.

BTW I am using RED HAT 7.1 download from the web
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now