Solved

Concatenate log files across multiple directories

Posted on 2003-11-19
6
1,522 Views
Last Modified: 2007-02-13
I am looking for a batch file that will concatenate log files by date and put the resulting file in a new directory, date-format file name.

I have individual directories for several web server log files, and need to concatenate each file in each directory by date and put the new resulting file in a new directory, retaining the date format.

So, for example:
In Directory1, file ex031119.log needs to be concatenated with file ex031119.log from Directory2, Directory3, and Directory4.

File format is: "exYYMMDD.log"  (example: ex031119.log)

I thougt I had a solution, but can't get it to work right.
0
Comment
Question by:linesplice
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 16

Expert Comment

by:_nn_
ID: 9781790
Assuming Windows (you didn't mention...) and a structure like

\test
    \a
    \b
    \c
    \d
        \e
    \global

I would propose something like following (tested on W2K) :

@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (>>%TARGET%\%1 type %%a\%1)
:end

0
 

Author Comment

by:linesplice
ID: 9782068
Yes, sorry.  It's a windows environment.

That looks like it will work very well, except for whitespace.  Thank you!

As a follow-up, how would it be done to remove four specific strings at the beginning of three of the four files and get the outfile to be sorted by a timestamp within each file?

ex:
Remove these lines from all but the first file:

#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2003-04-25 14:47:06
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent)

And sort by the time stamp in the second column of data.

2003-04-25 14:47:06 192.168.168.10 - 192.168.168.20 80 GET /Content/css/mainStyles.css - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;+.NET+CLR+1.1.4322)
0
 
LVL 16

Accepted Solution

by:
_nn_ earned 400 total points
ID: 9782782
>> That looks like it will work very well, except for whitespace.  Thank you!

Welcome. What is the problem with the 'whitespace' ?

>> Remove these lines from all but the first file:

Can we simplify by removing all lines beginning with '#', but in the first file ?

>> And sort by the time stamp in the second column of data

If these are daily logs, dates are identical and times should sort fine with an alphabetic sorting I guess...

If my previous assumptions are correct, I'd propose :


@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (
  if not exist %TARGET%\temp_%1 ( >>%TARGET%\%1 findstr /R "^#" %%a\%1 )
  >>%TARGET%\temp_%1 findstr /R /V "^#" %%a\%1
)
>>%TARGET%\%1 sort %TARGET%\temp_%1
del /q %TARGET%\temp_%1
:end
0
Guide to Performance: Optimization & Monitoring

Nowadays, monitoring is a mixture of tools, systems, and codes—making it a very complex process. And with this complexity, comes variables for failure. Get DZone’s new Guide to Performance to learn how to proactively find these variables and solve them before a disruption occurs.

 

Author Comment

by:linesplice
ID: 9818435
Hmm... looks like the script works in about 90% of the cases, but just doesn't do some of the files, so I guess there is something special about those files that is preventing this from working properly on them.
Thanks again, _nn_ and if you had any other suggestions, it would be gratly appreciated.
0
 
LVL 16

Expert Comment

by:_nn_
ID: 9818928
First, thanks and glad I could help :)

Now, if you can find out what's the trouble with these files, please explain here, I'll keep monitoring this thread. Else, if you can zip a snapshot of the directory structure where it happens, please send it to : expert (you know what comes here) newvoice dot ch, and I'll take a look.
0
 

Author Comment

by:linesplice
ID: 9819520
Fantastic!

This happens in several files, but here is the line that it appears to hang on in one example.  It is the last line of the concatenated log, but only about half way through the first log, of four.

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ

Here is the full line from the original log file

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ–i01;+Windows+NT+5.0) -

On the server, the little arrow character that seems to be causing the problem is just a little square box, which I'll assume means it can't display the character.
0

Featured Post

How our DevOps Teams Maximize Uptime

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us. Read the use case whitepaper.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Change the background and font colors in Notepad++ 5 159
backup program with robocopy 6 76
WMIC Output Formatting 1 50
C# Windows Form Navigation - Total Beginner 9 54
In this post we will learn how to make Android Gesture Tutorial and give different functionality whenever a user Touch or Scroll android screen.
Computer science students often experience many of the same frustrations when going through their engineering courses. This article presents seven tips I found useful when completing a bachelors and masters degree in computing which I believe may he…
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question