File concat program

Posted on 2004-09-03
Last Modified: 2010-05-18
Hi There

I'm tryting to do the following:

I have all my webstats in one folder of mt Linux box ( /var/log/httpd)

They are listed as follow:


( note, dom1 and dom2 are only examples, there are no usable patterns in the part infront of the -access.log)

I would now like to have a script that can run through the folder to concat all the files for one particular dom into one file
ie, I want to combine dom1-access.log, dom1-access.log.1, dom1-access.log.2 , dom-access.log.3 etc into just dom1-access.log, ie, add the contents of the rolled logs to the "current" log file. The trick would also be to construct the new file as follow:

Since dom1-access.log.x (where x is the largest of whatever is present ( the log roller will go up to .4 I think)) is the "oldest" and it's data needs to be added to the "new" dom1-access.log file, then the dom1-access.log.3, then ...2, then ....1, and finaly the original dom1-access.log contents..

Can someone pls give me some pointers/examples since I am not very familiar with Perl, but Perl ( or Bash) is the only scripting language available to me on the server.

Question by:psimation
  • 3
  • 2

Expert Comment

ID: 11973540

opendir (INDIR,".");
closedir INDIR;

#combine files
foreach (sort {$b cmp $a} (@allfiles))
  if ( /(.*)-(access.log.*)/ )
        if ($sites{$1}++)
        {       open (OUT, ">>$1.log")}
        else {  open (OUT, ">$1.log")}
        open (IN, "$_");
        while (<IN>) {  print OUT }
        close IN;
        close OUT;

#now overwrite original with combined
foreach (keys %sites)
        rename "$_.access.log $_.access.log.0\n";
        rename "$_.dat $_.access.log\n";

NOTE: I backed up the original log. If you want to erase the files when you are done use "unlink". I also assumed you are running the perl program in your log directory

Accepted Solution

afong earned 250 total points
ID: 11973659
Sorry: the last couple of lines should be...
  rename "$_-access.log", "$_-access.log.0";
  rename "$_.log", "$_-access.log";
LVL 48

Expert Comment

ID: 11986313
This is a better/easier task for a shell script

cd /var/log/httpd

for log in *.log
   for domlog in `ls -r $domain-*[0-9]`
       cat $domlog >>$log
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

LVL 48

Expert Comment

ID: 12079953
psimation, just curious why you'd went for a more complex Perl solution over the easier shell solution.
LVL 17

Author Comment

ID: 12081633
It was the first solution posted to work  ;) By the time you posted, my problem was already solved, so it wouldn't be fair for me to then dismiss afong's solution. I do appreciate your input though!
LVL 48

Expert Comment

ID: 12091005
Which solution did you end up using though?  It's fine to give points to the first working solution, but remember that the first is not always the best.

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Uploading Binary Data using Perl 5 90
Expect & sshpass package for Solaris 10 x86 15 922
Migrate OTRS to newest version. 1 335
How to get all the API from website? 11 91
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This video shows how to use Hyena, from SystemTools Software, to bulk import 100 user accounts from an external text file. View in 1080p for best video quality.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question