Solved

joining 5 LARGE text files

Posted on 2004-08-19
2
151 Views
Last Modified: 2013-12-25
Hi,

Please can anyone help me join 5 large (1.8gb) text files togeather
to create 1 very large file.

I have some code in PHP but it bombs out at 2gb (seems there is a limit
and php needs re compiling, something thats not an option for me)

I don't want to take up all the servers resources so something that uses
little memory would be very good indeed!

here's the php code if it help...

<?php
set_time_limit(14400000);

$file[0] = "file1.txt";
$file[1] = "file2.txt";
$file[2] = "file3.txt";
$file[3] = "file4.txt";
$file[4] = "file5.txt";

$target = "output.txt";
$handle = fopen($target, 'a');

foreach ($file as $var) {
  $c_handle = fopen($var, 'r');
  do {
       $content = fread($c_handle,1000000);
       fwrite($handle, $content);
  } while (!empty($content));
  fclose($c_handle);
}
?>


all the best

Stu
0
Comment
Question by:stuayre
2 Comments
 
LVL 22

Expert Comment

by:NovaDenizen
ID: 11844448

Can PHP handle files or streams> 2gb?  I don't know.  There's a good chance it doesn't.
Can your web server handle PHP streams >2gb?  Maybe.
Can your operating system handle streams >2gb?  Maybe.
What are the chances that 9gb is going to successfully pass over a single TCP connection without being interrupted?  Not so good.

Here, you're trying to send huge static datafiles through the overhead of a scripting engine and a web server.  If the transfer is interrupted, a new transfer would have to start all over again at the beginning.

Use a faster and simpler method to send the static files, like ftp or a simple static http file server.  These protocols are interruptable and automatically resumable if you are using the right client library.  You might consider concatenating all the data files into one huge file on disk, if that would make the transfer easier to implement via ftp or http.

0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 125 total points
ID: 11844946
<? passthru("cat file[12345].txt > output.txt"); ?>
should work if your filesystem can handle such huge files
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ever wondered how to display how many visitors you have online. In this tutorial I will show you an easy but effective way to display the number of online visitors in WhizBase. In this article I assume you have read my previous articles and know …
In this tutorial I will show you how to provide a dynamic RTF document on your website generated with data from your database. For this tutorial you will need Microsoft Word or WordPad, WhizBase and Microsoft Access. In this tutorial I will show …
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)

776 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question