Solved

Concatenate log files across multiple directories

Posted on 2003-11-19
6
1,512 Views
Last Modified: 2007-02-13
I am looking for a batch file that will concatenate log files by date and put the resulting file in a new directory, date-format file name.

I have individual directories for several web server log files, and need to concatenate each file in each directory by date and put the new resulting file in a new directory, retaining the date format.

So, for example:
In Directory1, file ex031119.log needs to be concatenated with file ex031119.log from Directory2, Directory3, and Directory4.

File format is: "exYYMMDD.log"  (example: ex031119.log)

I thougt I had a solution, but can't get it to work right.
0
Comment
Question by:linesplice
  • 3
  • 3
6 Comments
 
LVL 16

Expert Comment

by:_nn_
ID: 9781790
Assuming Windows (you didn't mention...) and a structure like

\test
    \a
    \b
    \c
    \d
        \e
    \global

I would propose something like following (tested on W2K) :

@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (>>%TARGET%\%1 type %%a\%1)
:end

0
 

Author Comment

by:linesplice
ID: 9782068
Yes, sorry.  It's a windows environment.

That looks like it will work very well, except for whitespace.  Thank you!

As a follow-up, how would it be done to remove four specific strings at the beginning of three of the four files and get the outfile to be sorted by a timestamp within each file?

ex:
Remove these lines from all but the first file:

#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2003-04-25 14:47:06
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent)

And sort by the time stamp in the second column of data.

2003-04-25 14:47:06 192.168.168.10 - 192.168.168.20 80 GET /Content/css/mainStyles.css - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;+.NET+CLR+1.1.4322)
0
 
LVL 16

Accepted Solution

by:
_nn_ earned 400 total points
ID: 9782782
>> That looks like it will work very well, except for whitespace.  Thank you!

Welcome. What is the problem with the 'whitespace' ?

>> Remove these lines from all but the first file:

Can we simplify by removing all lines beginning with '#', but in the first file ?

>> And sort by the time stamp in the second column of data

If these are daily logs, dates are identical and times should sort fine with an alphabetic sorting I guess...

If my previous assumptions are correct, I'd propose :


@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (
  if not exist %TARGET%\temp_%1 ( >>%TARGET%\%1 findstr /R "^#" %%a\%1 )
  >>%TARGET%\temp_%1 findstr /R /V "^#" %%a\%1
)
>>%TARGET%\%1 sort %TARGET%\temp_%1
del /q %TARGET%\temp_%1
:end
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 

Author Comment

by:linesplice
ID: 9818435
Hmm... looks like the script works in about 90% of the cases, but just doesn't do some of the files, so I guess there is something special about those files that is preventing this from working properly on them.
Thanks again, _nn_ and if you had any other suggestions, it would be gratly appreciated.
0
 
LVL 16

Expert Comment

by:_nn_
ID: 9818928
First, thanks and glad I could help :)

Now, if you can find out what's the trouble with these files, please explain here, I'll keep monitoring this thread. Else, if you can zip a snapshot of the directory structure where it happens, please send it to : expert (you know what comes here) newvoice dot ch, and I'll take a look.
0
 

Author Comment

by:linesplice
ID: 9819520
Fantastic!

This happens in several files, but here is the line that it appears to hang on in one example.  It is the last line of the concatenated log, but only about half way through the first log, of four.

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ

Here is the full line from the original log file

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ–i01;+Windows+NT+5.0) -

On the server, the little arrow character that seems to be causing the problem is just a little square box, which I'll assume means it can't display the character.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This article will show, step by step, how to integrate R code into a R Sweave document
Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now