Solved

>2 Gb filesize

Posted on 2001-09-05
16
464 Views
Last Modified: 2010-04-21
I have written a program that uses ~3.3 Gb of ram. It takes a long long time to create the lookup table and I would like to save this to disk! Except I can't at the moment.

How do I get my program to write files greater than two 2Gb. I have written my program in C/C++. Is there an easy way to do this or do I need to patch the kernal ?

Please feel free to help! I do need an idiots guide (sorry).

Thanks

Will
0
Comment
Question by:wjryder
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
  • 4
  • +1
16 Comments
 
LVL 3

Expert Comment

by:packratt_jk
ID: 6458772
it's not a kernel thing - it's a filesystem thing.
what filesystem are you using?.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6458950
I'm just fiddling arround with a similar problem. I'm trying to upgrade with kernel 2.4.9.
Will be back if I've it working ...
0
 

Author Comment

by:wjryder
ID: 6460048
The filesystem is ext2
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 5

Expert Comment

by:bryanh
ID: 6468965
Just some terminology corrections:  "it's not a kernel thing it's a filesystem thing".  It's a kernel thing.  Filesystem drivers are part of the kernel.  Even if they're added to the kernel after boot (with insmod).  Even if they're distributed separately from Linux.

And the filesystem isn't ext2.  The filesystem _type_ is ext2.  A filesystem is an ordered collection of stored bytes.  As in "I put the filesystem on a partition on my Seagate drive."

I know it's very common in Linux circles to use the word "filesystem" to mean about 5 different things, including filesystem type, but it's less confusing for everyone if we don't.

I can never remember the current state of >2GB file capability, but  it definitely exists if you get the right level of Linux , compile options, open options, or whatever.  So I hope ahofffman finds the correct incantation and gets back to us.

At work, I'm developing a Linux filesystem driver for huge filesystems, so I need to know this cold eventually!

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6469011
just to avoid that someone waits for useless comments:
  - my problems occour on filesystem of type ext2
  - the problem is that I cannot delete files > 2GB
    and for exaple Samba (smbd) hangs when performing such files
.. still on work ..
0
 
LVL 5

Expert Comment

by:bryanh
ID: 6472392
I checked, and Linux shouldn't have any problem writing >2GB files; you write them the same way you write <2GB files.  I presume from the question that you tried and failed, so if you could give details on the failure, maybe I could figure it out.

In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB (it apparently does this because it isn't sure you know how to handle a >2GB file).  To open such a file, you use open64().  But there shouldn't be any problem creating a >2GB file with a regular open() or on Linux 2.2.

ext2 has a file size restriction based on the block size, as for 1K blocks it's a little over 16GB.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6472489
> In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB  ..
Does this mean that a simple patch of libc.* should work?
if so, which version?
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6472518
0
 
LVL 5

Expert Comment

by:bryanh
ID: 6472525
"Work" to accomplish what?  You mean to use programs written and compiled for <2GB files on >2GB files?

In that case, a simple patch to libc would do it.  You just have to add the O_LARGEFILE flag to the open flags.  All versions of the GNU C library would need this patch.

But of course, this would only work insofar as the program in question can deal with file sizes and offsets that don't fit in a 32 bit signed integer.  A program that depends on <2GB files will fail in strange ways, since it will be using library functions with 32 bit parameters/return values.

And if you have the source code and know that your program can handle 64 bit file positions/sizes, then you don't need a patch -- just code the program with open64() instead of open().
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6472536
hmm, yes that's what I understand in the link povided in my previous comment, SO this should be a solution for wjryder
's question.
0
 
LVL 5

Expert Comment

by:bryanh
ID: 6472698
Thanks, that was an interesting read.  It doesn't do a very good job of explaining how to use >2GB files, though.  

First of all, it mentions using -D gcc options.  Unless you're playing around, this is the wrong way to make your program work with >2GB files.  The macros in question should be set with #define statements in the source code (before the C library #include statements).  They declare characteristics of the source code, not of the compilation.  

Also, _LARGE_FILE_SOURCE isn't related to this.  You don't need it.

Finally it doesn't make it clear that there are two distinct methods:

  1) #define _FILE_OFFSET_BITS 64.  With this, you can use all the traditionally named functions (open, fseek, etc), but get the new 64 bit versions of them instead of the 32 bit versions they used to be.

  2) #define _LARGE_FILE_SOURCE.  With this, new 64-bit versions of the functions (open64, fseek64, etc.) exist, but the old functions remain 32 bits.  This might be useful if you had some files that need the 32 bit interface and some that need the 64 bit interface.

Since the web page implied >2GB files are impossible in Linux 2.2, even though I saw plenty of source code to do it, I investigated a little further and found that while the general file system is capable, the ext2 filesystem driver rejects seeks past 2GB (and presumably other attempts to work with >2GB files).

2.4 is all OK, though.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6473870
hey bryanh, should post some points for you, thanks for these explanations ;-)
0
 

Author Comment

by:wjryder
ID: 6474013
I have tried to use fopen64. But am failing :

Hence have written an example program - (Hopefully error free!) - Can you please explain how to compile this ?

The program will create a file 1 byte bigger than the file limit size (I think)

Hence if it does not crash problem solved!!

#include <stdio.h>

int main()
{

FILE *fp;

fp = fopen64("test.dat","wt");

short int Value;
short int *pValue;

pValue = &Value;

Value = 2;

for (int x = 0; x < 1073741825; x++)
{
fwrite(pvalue, sizeof(short int), 1, fp);
}

fclose(fp);

return 0;

}
0
 

Author Comment

by:wjryder
ID: 6474029
I have spotted an error :

fwrite(pvalue, sizeof(short int), 1, fp);

should read:

fwrite(pValue, sizeof(short int), 1, fp);
0
 
LVL 5

Accepted Solution

by:
bryanh earned 250 total points
ID: 6474717
I had some serious typos in my earlier posting.  The macro you don't need is _LARGEFILE_SOURCE and the macro you do need is _LARGEFILE64_SOURCE  .

If you add #define _LARGEFILE64_SOURCE as the first line of your program, it should work.  I used the command

  gcc -x c++ -Wall -Werror bigfile

By the way, if you want a faster test, don't write 2GB of data.  Just write one byte after the 2GB position, with fseeko64(fp, 1ULL<<31); fwrite("X",1,1,fp);  The system will fill in the earlier bytes with zeroes (and won't use 2GB of disk space for them, since it can represent an entire block of zeroes with a single bit).
0
 

Author Comment

by:wjryder
ID: 6474879
It works for me now.

BTW I am using RED HAT 7.1 download from the web
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
There are cases when e.g. an IT administrator wants to have full access and view into selected mailboxes on Exchange server, directly from his own email account in Outlook or Outlook Web Access. This proves useful when for example administrator want…
In this video, viewers are given an introduction to using the Windows 10 Snipping Tool, how to quickly locate it when it's needed and also how make it always available with a single click of a mouse button, by pinning it to the Desktop Task Bar. Int…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question