[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

>2 Gb filesize

Posted on 2001-09-05
16
Medium Priority
?
468 Views
Last Modified: 2010-04-21
I have written a program that uses ~3.3 Gb of ram. It takes a long long time to create the lookup table and I would like to save this to disk! Except I can't at the moment.

How do I get my program to write files greater than two 2Gb. I have written my program in C/C++. Is there an easy way to do this or do I need to patch the kernal ?

Please feel free to help! I do need an idiots guide (sorry).

Thanks

Will
0
Comment
Question by:wjryder
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
  • 4
  • +1
16 Comments
 
LVL 3

Expert Comment

by:packratt_jk
ID: 6458772
it's not a kernel thing - it's a filesystem thing.
what filesystem are you using?.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6458950
I'm just fiddling arround with a similar problem. I'm trying to upgrade with kernel 2.4.9.
Will be back if I've it working ...
0
 

Author Comment

by:wjryder
ID: 6460048
The filesystem is ext2
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 5

Expert Comment

by:bryanh
ID: 6468965
Just some terminology corrections:  "it's not a kernel thing it's a filesystem thing".  It's a kernel thing.  Filesystem drivers are part of the kernel.  Even if they're added to the kernel after boot (with insmod).  Even if they're distributed separately from Linux.

And the filesystem isn't ext2.  The filesystem _type_ is ext2.  A filesystem is an ordered collection of stored bytes.  As in "I put the filesystem on a partition on my Seagate drive."

I know it's very common in Linux circles to use the word "filesystem" to mean about 5 different things, including filesystem type, but it's less confusing for everyone if we don't.

I can never remember the current state of >2GB file capability, but  it definitely exists if you get the right level of Linux , compile options, open options, or whatever.  So I hope ahofffman finds the correct incantation and gets back to us.

At work, I'm developing a Linux filesystem driver for huge filesystems, so I need to know this cold eventually!

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6469011
just to avoid that someone waits for useless comments:
  - my problems occour on filesystem of type ext2
  - the problem is that I cannot delete files > 2GB
    and for exaple Samba (smbd) hangs when performing such files
.. still on work ..
0
 
LVL 5

Expert Comment

by:bryanh
ID: 6472392
I checked, and Linux shouldn't have any problem writing >2GB files; you write them the same way you write <2GB files.  I presume from the question that you tried and failed, so if you could give details on the failure, maybe I could figure it out.

In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB (it apparently does this because it isn't sure you know how to handle a >2GB file).  To open such a file, you use open64().  But there shouldn't be any problem creating a >2GB file with a regular open() or on Linux 2.2.

ext2 has a file size restriction based on the block size, as for 1K blocks it's a little over 16GB.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6472489
> In Linux 2.4, an open() on a 32 bit machine fails if the file is >2GB  ..
Does this mean that a simple patch of libc.* should work?
if so, which version?
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6472518
0
 
LVL 5

Expert Comment

by:bryanh
ID: 6472525
"Work" to accomplish what?  You mean to use programs written and compiled for <2GB files on >2GB files?

In that case, a simple patch to libc would do it.  You just have to add the O_LARGEFILE flag to the open flags.  All versions of the GNU C library would need this patch.

But of course, this would only work insofar as the program in question can deal with file sizes and offsets that don't fit in a 32 bit signed integer.  A program that depends on <2GB files will fail in strange ways, since it will be using library functions with 32 bit parameters/return values.

And if you have the source code and know that your program can handle 64 bit file positions/sizes, then you don't need a patch -- just code the program with open64() instead of open().
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6472536
hmm, yes that's what I understand in the link povided in my previous comment, SO this should be a solution for wjryder
's question.
0
 
LVL 5

Expert Comment

by:bryanh
ID: 6472698
Thanks, that was an interesting read.  It doesn't do a very good job of explaining how to use >2GB files, though.  

First of all, it mentions using -D gcc options.  Unless you're playing around, this is the wrong way to make your program work with >2GB files.  The macros in question should be set with #define statements in the source code (before the C library #include statements).  They declare characteristics of the source code, not of the compilation.  

Also, _LARGE_FILE_SOURCE isn't related to this.  You don't need it.

Finally it doesn't make it clear that there are two distinct methods:

  1) #define _FILE_OFFSET_BITS 64.  With this, you can use all the traditionally named functions (open, fseek, etc), but get the new 64 bit versions of them instead of the 32 bit versions they used to be.

  2) #define _LARGE_FILE_SOURCE.  With this, new 64-bit versions of the functions (open64, fseek64, etc.) exist, but the old functions remain 32 bits.  This might be useful if you had some files that need the 32 bit interface and some that need the 64 bit interface.

Since the web page implied >2GB files are impossible in Linux 2.2, even though I saw plenty of source code to do it, I investigated a little further and found that while the general file system is capable, the ext2 filesystem driver rejects seeks past 2GB (and presumably other attempts to work with >2GB files).

2.4 is all OK, though.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6473870
hey bryanh, should post some points for you, thanks for these explanations ;-)
0
 

Author Comment

by:wjryder
ID: 6474013
I have tried to use fopen64. But am failing :

Hence have written an example program - (Hopefully error free!) - Can you please explain how to compile this ?

The program will create a file 1 byte bigger than the file limit size (I think)

Hence if it does not crash problem solved!!

#include <stdio.h>

int main()
{

FILE *fp;

fp = fopen64("test.dat","wt");

short int Value;
short int *pValue;

pValue = &Value;

Value = 2;

for (int x = 0; x < 1073741825; x++)
{
fwrite(pvalue, sizeof(short int), 1, fp);
}

fclose(fp);

return 0;

}
0
 

Author Comment

by:wjryder
ID: 6474029
I have spotted an error :

fwrite(pvalue, sizeof(short int), 1, fp);

should read:

fwrite(pValue, sizeof(short int), 1, fp);
0
 
LVL 5

Accepted Solution

by:
bryanh earned 1000 total points
ID: 6474717
I had some serious typos in my earlier posting.  The macro you don't need is _LARGEFILE_SOURCE and the macro you do need is _LARGEFILE64_SOURCE  .

If you add #define _LARGEFILE64_SOURCE as the first line of your program, it should work.  I used the command

  gcc -x c++ -Wall -Werror bigfile

By the way, if you want a faster test, don't write 2GB of data.  Just write one byte after the 2GB position, with fseeko64(fp, 1ULL<<31); fwrite("X",1,1,fp);  The system will fill in the earlier bytes with zeroes (and won't use 2GB of disk space for them, since it can represent an entire block of zeroes with a single bit).
0
 

Author Comment

by:wjryder
ID: 6474879
It works for me now.

BTW I am using RED HAT 7.1 download from the web
0

Featured Post

Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…
Sometimes it takes a new vantage point, apart from our everyday security practices, to truly see our Active Directory (AD) vulnerabilities. We get used to implementing the same techniques and checking the same areas for a breach. This pattern can re…

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question