Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

joining 5 LARGE text files

Posted on 2004-08-19
2
Medium Priority
?
162 Views
Last Modified: 2013-12-25
Hi,

Please can anyone help me join 5 large (1.8gb) text files togeather
to create 1 very large file.

I have some code in PHP but it bombs out at 2gb (seems there is a limit
and php needs re compiling, something thats not an option for me)

I don't want to take up all the servers resources so something that uses
little memory would be very good indeed!

here's the php code if it help...

<?php
set_time_limit(14400000);

$file[0] = "file1.txt";
$file[1] = "file2.txt";
$file[2] = "file3.txt";
$file[3] = "file4.txt";
$file[4] = "file5.txt";

$target = "output.txt";
$handle = fopen($target, 'a');

foreach ($file as $var) {
  $c_handle = fopen($var, 'r');
  do {
       $content = fread($c_handle,1000000);
       fwrite($handle, $content);
  } while (!empty($content));
  fclose($c_handle);
}
?>


all the best

Stu
0
Comment
Question by:stuayre
2 Comments
 
LVL 22

Expert Comment

by:NovaDenizen
ID: 11844448

Can PHP handle files or streams> 2gb?  I don't know.  There's a good chance it doesn't.
Can your web server handle PHP streams >2gb?  Maybe.
Can your operating system handle streams >2gb?  Maybe.
What are the chances that 9gb is going to successfully pass over a single TCP connection without being interrupted?  Not so good.

Here, you're trying to send huge static datafiles through the overhead of a scripting engine and a web server.  If the transfer is interrupted, a new transfer would have to start all over again at the beginning.

Use a faster and simpler method to send the static files, like ftp or a simple static http file server.  These protocols are interruptable and automatically resumable if you are using the right client library.  You might consider concatenating all the data files into one huge file on disk, if that would make the transfer easier to implement via ftp or http.

0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 500 total points
ID: 11844946
<? passthru("cat file[12345].txt > output.txt"); ?>
should work if your filesystem can handle such huge files
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Recently I have been answering a lot of questions like this in IT forums that I frequent. The question posed is usually something along the lines of "We have software X installed and need to uninstall it for reason Y" or some other variant of the sa…
Measuring Server's processing rate with a simple powershell command. The differences in processing rate also was recorded in different use-cases, when a server in free and busy states.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
Suggested Courses

564 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question