Concatenate log files across multiple directories

I am looking for a batch file that will concatenate log files by date and put the resulting file in a new directory, date-format file name.

I have individual directories for several web server log files, and need to concatenate each file in each directory by date and put the new resulting file in a new directory, retaining the date format.

So, for example:
In Directory1, file ex031119.log needs to be concatenated with file ex031119.log from Directory2, Directory3, and Directory4.

File format is: "exYYMMDD.log"  (example: ex031119.log)

I thougt I had a solution, but can't get it to work right.
linespliceAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

_nn_Commented:
Assuming Windows (you didn't mention...) and a structure like

\test
    \a
    \b
    \c
    \d
        \e
    \global

I would propose something like following (tested on W2K) :

@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (>>%TARGET%\%1 type %%a\%1)
:end

0
linespliceAuthor Commented:
Yes, sorry.  It's a windows environment.

That looks like it will work very well, except for whitespace.  Thank you!

As a follow-up, how would it be done to remove four specific strings at the beginning of three of the four files and get the outfile to be sorted by a timestamp within each file?

ex:
Remove these lines from all but the first file:

#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2003-04-25 14:47:06
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent)

And sort by the time stamp in the second column of data.

2003-04-25 14:47:06 192.168.168.10 - 192.168.168.20 80 GET /Content/css/mainStyles.css - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;+.NET+CLR+1.1.4322)
0
_nn_Commented:
>> That looks like it will work very well, except for whitespace.  Thank you!

Welcome. What is the problem with the 'whitespace' ?

>> Remove these lines from all but the first file:

Can we simplify by removing all lines beginning with '#', but in the first file ?

>> And sort by the time stamp in the second column of data

If these are daily logs, dates are identical and times should sort fine with an alphabetic sorting I guess...

If my previous assumptions are correct, I'd propose :


@echo off
set BASE=\test\
set DIRS=a b c d\e
set TARGET=\test\global
set MASK=ex[0-9][0-9][0-9][0-9][0-9][0-9].log

set LAST=
pushd %BASE%
for /f "delims=" %%k in ('dir /b %DIRS% ^| findstr /R "%MASK%" ^| sort') do call :do_it %%k
popd
goto end

:do_it
if "%1" == "%LAST%" goto end
set LAST=%1
for %%a in (%DIRS%) do if exist %%a\%1 (
  if not exist %TARGET%\temp_%1 ( >>%TARGET%\%1 findstr /R "^#" %%a\%1 )
  >>%TARGET%\temp_%1 findstr /R /V "^#" %%a\%1
)
>>%TARGET%\%1 sort %TARGET%\temp_%1
del /q %TARGET%\temp_%1
:end
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

linespliceAuthor Commented:
Hmm... looks like the script works in about 90% of the cases, but just doesn't do some of the files, so I guess there is something special about those files that is preventing this from working properly on them.
Thanks again, _nn_ and if you had any other suggestions, it would be gratly appreciated.
0
_nn_Commented:
First, thanks and glad I could help :)

Now, if you can find out what's the trouble with these files, please explain here, I'll keep monitoring this thread. Else, if you can zip a snapshot of the directory structure where it happens, please send it to : expert (you know what comes here) newvoice dot ch, and I'll take a look.
0
linespliceAuthor Commented:
Fantastic!

This happens in several files, but here is the line that it appears to hang on in one example.  It is the last line of the concatenated log, but only about half way through the first log, of four.

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ

Here is the full line from the original log file

2003-09-29 12:39:09 151.196.37.36 - 192.168.10.15 80 GET /baltimore - 302 187 Mozilla/4.0+(compatible;+MSIÍ–i01;+Windows+NT+5.0) -

On the server, the little arrow character that seems to be causing the problem is just a little square box, which I'll assume means it can't display the character.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Programming

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.