• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1527
  • Last Modified:

Concatenate log files across multiple directories

I am looking for a batch file that will concatenate log files by date and put the resulting file in a new directory, date-format file name.

I have individual directories for several web server log files, and need to concatenate each file in each directory by date and put the new resulting file in a new directory, retaining the date format.

So, for example:
In Directory1, file ex031119.log needs to be concatenated with file ex031119.log from Directory2, Directory3, and Directory4.

File format is: "exYYMMDD.log"  (example: ex031119.log)

I thougt I had a solution, but can't get it to work right.
0
linesplice
Asked:
linesplice
  • 3
  • 3
1 Solution
 
_nn_Commented:
Assuming Windows (you didn't mention...) and a structure like

\test
    \a
    \b
    \c
    \d
        \e
    \global

I would propose something like following (tested on W2K) :

@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (>>%TARGET%\%1 type %%a\%1)
:end

0
 
linespliceAuthor Commented:
Yes, sorry.  It's a windows environment.

That looks like it will work very well, except for whitespace.  Thank you!

As a follow-up, how would it be done to remove four specific strings at the beginning of three of the four files and get the outfile to be sorted by a timestamp within each file?

ex:
Remove these lines from all but the first file:

#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2003-04-25 14:47:06
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent)

And sort by the time stamp in the second column of data.

2003-04-25 14:47:06 192.168.168.10 - 192.168.168.20 80 GET /Content/css/mainStyles.css - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;+.NET+CLR+1.1.4322)
0
 
_nn_Commented:
>> That looks like it will work very well, except for whitespace.  Thank you!

Welcome. What is the problem with the 'whitespace' ?

>> Remove these lines from all but the first file:

Can we simplify by removing all lines beginning with '#', but in the first file ?

>> And sort by the time stamp in the second column of data

If these are daily logs, dates are identical and times should sort fine with an alphabetic sorting I guess...

If my previous assumptions are correct, I'd propose :


@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (
  if not exist %TARGET%\temp_%1 ( >>%TARGET%\%1 findstr /R "^#" %%a\%1 )
  >>%TARGET%\temp_%1 findstr /R /V "^#" %%a\%1
)
>>%TARGET%\%1 sort %TARGET%\temp_%1
del /q %TARGET%\temp_%1
:end
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
linespliceAuthor Commented:
Hmm... looks like the script works in about 90% of the cases, but just doesn't do some of the files, so I guess there is something special about those files that is preventing this from working properly on them.
Thanks again, _nn_ and if you had any other suggestions, it would be gratly appreciated.
0
 
_nn_Commented:
First, thanks and glad I could help :)

Now, if you can find out what's the trouble with these files, please explain here, I'll keep monitoring this thread. Else, if you can zip a snapshot of the directory structure where it happens, please send it to : expert (you know what comes here) newvoice dot ch, and I'll take a look.
0
 
linespliceAuthor Commented:
Fantastic!

This happens in several files, but here is the line that it appears to hang on in one example.  It is the last line of the concatenated log, but only about half way through the first log, of four.

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ

Here is the full line from the original log file

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ–i01;+Windows+NT+5.0) -

On the server, the little arrow character that seems to be causing the problem is just a little square box, which I'll assume means it can't display the character.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now