Solved

File concat program

Posted on 2004-09-03
6
677 Views
Last Modified: 2010-05-18
Hi There

I'm tryting to do the following:

I have all my webstats in one folder of mt Linux box ( /var/log/httpd)

They are listed as follow:

dom1-access.log
dom1-access.log.1
dom1-access.log.2
dom2-access.log
dom2-access.log.1
dom2-access.log.2
dom3-access.log
dom3-access.log.1
etc.etc

( note, dom1 and dom2 are only examples, there are no usable patterns in the part infront of the -access.log)

I would now like to have a script that can run through the folder to concat all the files for one particular dom into one file
ie, I want to combine dom1-access.log, dom1-access.log.1, dom1-access.log.2 , dom-access.log.3 etc into just dom1-access.log, ie, add the contents of the rolled logs to the "current" log file. The trick would also be to construct the new file as follow:

Since dom1-access.log.x (where x is the largest of whatever is present ( the log roller will go up to .4 I think)) is the "oldest" and it's data needs to be added to the "new" dom1-access.log file, then the dom1-access.log.3, then ...2, then ....1, and finaly the original dom1-access.log contents..

Can someone pls give me some pointers/examples since I am not very familiar with Perl, but Perl ( or Bash) is the only scripting language available to me on the server.


0
Comment
Question by:psimation
  • 3
  • 2
6 Comments
 
LVL 1

Expert Comment

by:afong
ID: 11973540
#!/usr/bin/perl

opendir (INDIR,".");
@allfiles=readdir(INDIR);
closedir INDIR;

#combine files
foreach (sort {$b cmp $a} (@allfiles))
{
  if ( /(.*)-(access.log.*)/ )
  {
        if ($sites{$1}++)
        {       open (OUT, ">>$1.log")}
        else {  open (OUT, ">$1.log")}
        open (IN, "$_");
        while (<IN>) {  print OUT }
        close IN;
        close OUT;
   }
}

#now overwrite original with combined
foreach (keys %sites)
{
        rename "$_.access.log $_.access.log.0\n";
        rename "$_.dat $_.access.log\n";
}

NOTE: I backed up the original log. If you want to erase the files when you are done use "unlink". I also assumed you are running the perl program in your log directory
0
 
LVL 1

Accepted Solution

by:
afong earned 250 total points
ID: 11973659
Sorry: the last couple of lines should be...
{
  rename "$_-access.log", "$_-access.log.0";
  rename "$_.log", "$_-access.log";
}
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11986313
This is a better/easier task for a shell script

#!/bin/bash
cd /var/log/httpd

for log in *.log
do
   domain=${log%%-access.log}
   for domlog in `ls -r $domain-*[0-9]`
   do
       cat $domlog >>$log
    done
 done
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
LVL 48

Expert Comment

by:Tintin
ID: 12079953
psimation, just curious why you'd went for a more complex Perl solution over the easier shell solution.
0
 
LVL 17

Author Comment

by:psimation
ID: 12081633
It was the first solution posted to work  ;) By the time you posted, my problem was already solved, so it wouldn't be fair for me to then dismiss afong's solution. I do appreciate your input though!
0
 
LVL 48

Expert Comment

by:Tintin
ID: 12091005
Which solution did you end up using though?  It's fine to give points to the first working solution, but remember that the first is not always the best.
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Perl 101 11 81
rename outfile before writing 2 74
instmodsh - listing perl modules 2 86
add a syntax to a csv file 8 99
I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question