Solved

File concat program

Posted on 2004-09-03
6
669 Views
Last Modified: 2010-05-18
Hi There

I'm tryting to do the following:

I have all my webstats in one folder of mt Linux box ( /var/log/httpd)

They are listed as follow:

dom1-access.log
dom1-access.log.1
dom1-access.log.2
dom2-access.log
dom2-access.log.1
dom2-access.log.2
dom3-access.log
dom3-access.log.1
etc.etc

( note, dom1 and dom2 are only examples, there are no usable patterns in the part infront of the -access.log)

I would now like to have a script that can run through the folder to concat all the files for one particular dom into one file
ie, I want to combine dom1-access.log, dom1-access.log.1, dom1-access.log.2 , dom-access.log.3 etc into just dom1-access.log, ie, add the contents of the rolled logs to the "current" log file. The trick would also be to construct the new file as follow:

Since dom1-access.log.x (where x is the largest of whatever is present ( the log roller will go up to .4 I think)) is the "oldest" and it's data needs to be added to the "new" dom1-access.log file, then the dom1-access.log.3, then ...2, then ....1, and finaly the original dom1-access.log contents..

Can someone pls give me some pointers/examples since I am not very familiar with Perl, but Perl ( or Bash) is the only scripting language available to me on the server.


0
Comment
Question by:psimation
  • 3
  • 2
6 Comments
 
LVL 1

Expert Comment

by:afong
ID: 11973540
#!/usr/bin/perl

opendir (INDIR,".");
@allfiles=readdir(INDIR);
closedir INDIR;

#combine files
foreach (sort {$b cmp $a} (@allfiles))
{
  if ( /(.*)-(access.log.*)/ )
  {
        if ($sites{$1}++)
        {       open (OUT, ">>$1.log")}
        else {  open (OUT, ">$1.log")}
        open (IN, "$_");
        while (<IN>) {  print OUT }
        close IN;
        close OUT;
   }
}

#now overwrite original with combined
foreach (keys %sites)
{
        rename "$_.access.log $_.access.log.0\n";
        rename "$_.dat $_.access.log\n";
}

NOTE: I backed up the original log. If you want to erase the files when you are done use "unlink". I also assumed you are running the perl program in your log directory
0
 
LVL 1

Accepted Solution

by:
afong earned 250 total points
ID: 11973659
Sorry: the last couple of lines should be...
{
  rename "$_-access.log", "$_-access.log.0";
  rename "$_.log", "$_-access.log";
}
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11986313
This is a better/easier task for a shell script

#!/bin/bash
cd /var/log/httpd

for log in *.log
do
   domain=${log%%-access.log}
   for domlog in `ls -r $domain-*[0-9]`
   do
       cat $domlog >>$log
    done
 done
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 48

Expert Comment

by:Tintin
ID: 12079953
psimation, just curious why you'd went for a more complex Perl solution over the easier shell solution.
0
 
LVL 17

Author Comment

by:psimation
ID: 12081633
It was the first solution posted to work  ;) By the time you posted, my problem was already solved, so it wouldn't be fair for me to then dismiss afong's solution. I do appreciate your input though!
0
 
LVL 48

Expert Comment

by:Tintin
ID: 12091005
Which solution did you end up using though?  It's fine to give points to the first working solution, but remember that the first is not always the best.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

On Microsoft Windows, if  when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now