Solved

joining 5 LARGE text files

Posted on 2004-08-19
2
149 Views
Last Modified: 2013-12-25
Hi,

Please can anyone help me join 5 large (1.8gb) text files togeather
to create 1 very large file.

I have some code in PHP but it bombs out at 2gb (seems there is a limit
and php needs re compiling, something thats not an option for me)

I don't want to take up all the servers resources so something that uses
little memory would be very good indeed!

here's the php code if it help...

<?php
set_time_limit(14400000);

$file[0] = "file1.txt";
$file[1] = "file2.txt";
$file[2] = "file3.txt";
$file[3] = "file4.txt";
$file[4] = "file5.txt";

$target = "output.txt";
$handle = fopen($target, 'a');

foreach ($file as $var) {
  $c_handle = fopen($var, 'r');
  do {
       $content = fread($c_handle,1000000);
       fwrite($handle, $content);
  } while (!empty($content));
  fclose($c_handle);
}
?>


all the best

Stu
0
Comment
Question by:stuayre
2 Comments
 
LVL 22

Expert Comment

by:NovaDenizen
ID: 11844448

Can PHP handle files or streams> 2gb?  I don't know.  There's a good chance it doesn't.
Can your web server handle PHP streams >2gb?  Maybe.
Can your operating system handle streams >2gb?  Maybe.
What are the chances that 9gb is going to successfully pass over a single TCP connection without being interrupted?  Not so good.

Here, you're trying to send huge static datafiles through the overhead of a scripting engine and a web server.  If the transfer is interrupted, a new transfer would have to start all over again at the beginning.

Use a faster and simpler method to send the static files, like ftp or a simple static http file server.  These protocols are interruptable and automatically resumable if you are using the right client library.  You might consider concatenating all the data files into one huge file on disk, if that would make the transfer easier to implement via ftp or http.

0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 125 total points
ID: 11844946
<? passthru("cat file[12345].txt > output.txt"); ?>
should work if your filesystem can handle such huge files
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

In this tutorial I will show you how to provide a dynamic RTF document on your website generated with data from your database. For this tutorial you will need Microsoft Word or WordPad, WhizBase and Microsoft Access. In this tutorial I will show …
This article will show, step by step, how to integrate R code into a R Sweave document
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now