Solved

joining 5 LARGE text files

Posted on 2004-08-19
2
153 Views
Last Modified: 2013-12-25
Hi,

Please can anyone help me join 5 large (1.8gb) text files togeather
to create 1 very large file.

I have some code in PHP but it bombs out at 2gb (seems there is a limit
and php needs re compiling, something thats not an option for me)

I don't want to take up all the servers resources so something that uses
little memory would be very good indeed!

here's the php code if it help...

<?php
set_time_limit(14400000);

$file[0] = "file1.txt";
$file[1] = "file2.txt";
$file[2] = "file3.txt";
$file[3] = "file4.txt";
$file[4] = "file5.txt";

$target = "output.txt";
$handle = fopen($target, 'a');

foreach ($file as $var) {
  $c_handle = fopen($var, 'r');
  do {
       $content = fread($c_handle,1000000);
       fwrite($handle, $content);
  } while (!empty($content));
  fclose($c_handle);
}
?>


all the best

Stu
0
Comment
Question by:stuayre
2 Comments
 
LVL 22

Expert Comment

by:NovaDenizen
ID: 11844448

Can PHP handle files or streams> 2gb?  I don't know.  There's a good chance it doesn't.
Can your web server handle PHP streams >2gb?  Maybe.
Can your operating system handle streams >2gb?  Maybe.
What are the chances that 9gb is going to successfully pass over a single TCP connection without being interrupted?  Not so good.

Here, you're trying to send huge static datafiles through the overhead of a scripting engine and a web server.  If the transfer is interrupted, a new transfer would have to start all over again at the beginning.

Use a faster and simpler method to send the static files, like ftp or a simple static http file server.  These protocols are interruptable and automatically resumable if you are using the right client library.  You might consider concatenating all the data files into one huge file on disk, if that would make the transfer easier to implement via ftp or http.

0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 125 total points
ID: 11844946
<? passthru("cat file[12345].txt > output.txt"); ?>
should work if your filesystem can handle such huge files
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some…
Making a simple AJAX shopping cart Couple years ago I made my first shopping cart, I used iframe and JavaScript, it was very good at that time, there were no sessions or AJAX, I used cookies on clients machine. Today we have more advanced techno…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

792 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question