Solved

joining 5 LARGE text files

Posted on 2004-08-19
2
155 Views
Last Modified: 2013-12-25
Hi,

Please can anyone help me join 5 large (1.8gb) text files togeather
to create 1 very large file.

I have some code in PHP but it bombs out at 2gb (seems there is a limit
and php needs re compiling, something thats not an option for me)

I don't want to take up all the servers resources so something that uses
little memory would be very good indeed!

here's the php code if it help...

<?php
set_time_limit(14400000);

$file[0] = "file1.txt";
$file[1] = "file2.txt";
$file[2] = "file3.txt";
$file[3] = "file4.txt";
$file[4] = "file5.txt";

$target = "output.txt";
$handle = fopen($target, 'a');

foreach ($file as $var) {
  $c_handle = fopen($var, 'r');
  do {
       $content = fread($c_handle,1000000);
       fwrite($handle, $content);
  } while (!empty($content));
  fclose($c_handle);
}
?>


all the best

Stu
0
Comment
Question by:stuayre
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 22

Expert Comment

by:NovaDenizen
ID: 11844448

Can PHP handle files or streams> 2gb?  I don't know.  There's a good chance it doesn't.
Can your web server handle PHP streams >2gb?  Maybe.
Can your operating system handle streams >2gb?  Maybe.
What are the chances that 9gb is going to successfully pass over a single TCP connection without being interrupted?  Not so good.

Here, you're trying to send huge static datafiles through the overhead of a scripting engine and a web server.  If the transfer is interrupted, a new transfer would have to start all over again at the beginning.

Use a faster and simpler method to send the static files, like ftp or a simple static http file server.  These protocols are interruptable and automatically resumable if you are using the right client library.  You might consider concatenating all the data files into one huge file on disk, if that would make the transfer easier to implement via ftp or http.

0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 125 total points
ID: 11844946
<? passthru("cat file[12345].txt > output.txt"); ?>
should work if your filesystem can handle such huge files
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It is becoming increasingly popular to have a front-page slider on a web site. Nearly every TV website,  magazine or online news has one on their site, and even some e-commerce sites have one. Today you can use sliders with Joomla, WordPress or …
In threads here at EE, each comment has a unique Identifier (ID). It is easy to get the full path for an ID via the right-click context menu. However, we often want to post a short link within a thread rather than the full link. This article shows a…
The viewer will learn how to count occurrences of each item in an array.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question