Solved

Script to extract log from many SMTP log

Posted on 2010-09-09
41
600 Views
Last Modified: 2012-05-10
Hi,

I need a script powershel or other to extract from my HUB exchange 2007 SmtpReceive LOG all line containt this:

"421 4.3.2 Service not available, closing transmission channel"

I have many file log in same folder, the script must go through all file.

Thanks for your help
0
Comment
Question by:cawasaki
  • 19
  • 11
  • 10
  • +1
41 Comments
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33634554
use in script
for /f "tokens=*" %%I in (dir target_folder /b /A-D ) do FIND /I "421 4.3.2 Service not available, closing transmission channel" %%I >> logfile.txt
0
 

Author Comment

by:cawasaki
ID: 33634593
hi Tominov,

Its a script? can you explain your script and how i can use it?

Thanks
0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33634626
create batch file - smtplog.cmd with content:
for /f "tokens=*" %%I in (dir target_folder /b /A-D ) do FIND /I "421 4.3.2 Service not available, closing transmission channel" %%I >> logfile.txt

You must type correct target_ folder place and place for result file logfile.txt.
Explanation of the script:
for every line in output from command in bracket - dir..... - is done command FIND which print the whole line
where find the requested string - this output is directed to the file logfile.txt - line by line.
0
 

Author Comment

by:cawasaki
ID: 33634649
HUm, but i have many different log file and i need to find "421 4.3.2 Service not available, closing transmission channel" in all logs file at same times, your script can do this?

thanks
0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33634671
this script go throw all files in directory and find in all files the requested string.
Do you have the log files in the same directory ?
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33634788
PowerShell solution:
dir *.log | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) | ft filename,linenumber, line

Open in new window

0
 

Author Comment

by:cawasaki
ID: 33634796
yes all my smtp log is in same directory at e:\smtplog

I need add the directory to the script or just lunch the script in directory?

thanks
0
 

Author Comment

by:cawasaki
ID: 33634884
hi soostibi,

your script list just a name of file, i need a script to export all line contain 421 4.3.2 Service not available, closing transmission channel
0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33634904
if you launch it in this directory then you don't need to fill whole path.
But better is to fill full path. I correct my mistake in command - please copy and paste the following
part to your cmd file:
for /f "tokens=*" %%I in ('dir e:\smtplog /b /A-D') do FIND /I "421 4.3.2 Service not available, closing transmission channel" e:\smtplog\%%I >> e:\logfile.txt
The result file always save in another location.
0
 

Author Comment

by:cawasaki
ID: 33635389
Tominov

your script work perfectly, it is possible to add option to this script t export logfile with date, for exemple:

lofile09_09_10, because i need to schedule this script every 2 days.

Thanks
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33635415
No, my script extracts the filename, the linenumber and the line containing the given text. I updated your path in the script and left the formatting part, so now you'll see only the lines.

If you add this, you'll also see the other data:
| ft line, filename, linenumber
dir e:\smtplog\log | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel"))

Open in new window

0
 
LVL 13

Expert Comment

by:soostibi
ID: 33635417
Sorry the * is missing, so the corrected one:
dir e:\smtplog\*log | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel"))

Open in new window

0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33635427
Yes, I can, but be aware: in every export file will be the same data like in previous version + new data. It is really what you need ?
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33635433
This exports the data to yourpath\yourfile:
dir e:\smtplog\log | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) | select-object line, filename, linenumber | out-file e:\yourpath\yourfile.txt -append

Open in new window

0
 
LVL 13

Expert Comment

by:soostibi
ID: 33635438
Sorry, the * disappeared again, so:
dir e:\smtplog\*log | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) | 

select-object line, filename, linenumber | out-file e:\yourpath\yourfile.txt -append

Open in new window

0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33635739
New variant of your script include generation of export filename based on today's date.
The first line create variable currentdate based on current date (this must be done because
output from command date depend on international setting in operating system and one
of the varant is day/month/year and "/" is not allowed character in name of the file or directory.

for /f "tokens=2-4 delims=/ " %%i in ('date /t') do set currentdate=%%i-%%j-%%k
for /f "tokens=*" %%I in ('dir e:\smtplog /b /A-D') do FIND /I "421 4.3.2 Service not available, closing transmission channel" e:\smtplog\%%I >> e:\LOG-%currentdate%.txt
0
 

Author Comment

by:cawasaki
ID: 33637423
Hi,

tominov: Yes, I can, but be aware: in every export file will be the same data like in previous version + new data. It is really what you need ?

==> Yes

soostibi:

==> I will test your script
0
 

Author Comment

by:cawasaki
ID: 33637567
soostibi:

i just need to export all line in a txt or csv file not line number.

If the script find a line contain 421 4.3.2 Service not available, closing transmission channel, it copy all line to exported file.

thanks
0
 

Author Comment

by:cawasaki
ID: 33637604
soostibi:

Your script create a file like this: LOG-09-2010-.txt.

It is possible to add the day? because if i need to lunch a script every day

thanks
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33638782
This dumps all the filenames and line that include the given expression into a csv. If you run this from day to day, the new CSVs only include data from the logs, that were created after the previous CSV file. The name of the CSV file is like LOG-08-09-2010.csv.
Is that what you want?
$startdate = (Get-ChildItem e:\smtplog\*.csv | Sort-Object -Property lastwritetime | Select-Object -First 1).lastwritetime

dir e:\smtplog\*.log | ?{$_.lastwritetime -gt $startdate} | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) | 

	Select-Object filename, line | Export-Csv -Path "e:\smtplog\LOG-$(get-date -f 'dd-mm-yyy').log"

Open in new window

0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 

Author Comment

by:cawasaki
ID: 33640315
Tominov,


Your script create a file like this: LOG-09-2010-.txt.

It is possible to add the day? because if i need to lunch a script every day

thanks
0
 
LVL 16

Expert Comment

by:Dale Harris
ID: 33640325
Soostibi,

If the above script is executed, the 'dd-mm-yyy' will actually return day - minute - year.  Your best bet is:
(get-date).toshortdatestring

Which will return 9/9/2010 if it's ran today.

Regards,

Dale Harris
0
 
LVL 13

Accepted Solution

by:
soostibi earned 250 total points
ID: 33640373
Sorry, with MM it is correct.
$startdate = (Get-ChildItem e:\smtplog\*.csv | Sort-Object -Property lastwritetime | Select-Object -First 1).lastwritetime  

dir e:\smtplog\*.log | ?{$_.lastwritetime -gt $startdate} | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) |   

        Select-Object filename, line | Export-Csv -Path "e:\smtplog\LOG-$(get-date -f 'dd-MM-yyy').log"

Open in new window

0
 

Author Comment

by:cawasaki
ID: 33640400
Soostibi:

This dumps all the filenames and line that include the given expression into a csv. If you run this from day to day, the new CSVs only include data from the logs, that were created after the previous CSV file. The name of the CSV file is like LOG-08-09-2010.csv.
Is that what you want

==>i need jut line that include given expression un my csv, not need filename.
and ok for the rest

I have error when i execute your powershellscript
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33640492
What error do you get? (I took the filename out of the result)
$startdate = (Get-ChildItem e:\smtplog\*.csv | Sort-Object -Property lastwritetime | Select-Object -First 1).lastwritetime    

dir e:\smtplog\*.log | ?{$_.lastwritetime -gt $startdate} | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) |     

        Select-Object line | Export-Csv -Path "e:\smtplog\LOG-$(get-date -f 'dd-MM-yyy').log"

Open in new window

0
 

Author Comment

by:cawasaki
ID: 33640602
here error:


[PS] C:\Windows\System32>$startdate = (Get-ChildItem e:\test\*.csv | Sort-Object
 -Property lastwritetime | Select-Object -First 1).lastwritetime dir c:\test\*.l
og | ?{$_.lastwritetime -gt $startdate} | Select-String ([regex]::escape("421 4.
3.2 Service not available, closing transmission channel")) Select-Object line |
Export-Csv -Path "c:\test\LOG-$(get-date -f 'dd-MM-yyy').log"

Unexpected token 'dir' in expression or statement.
At line:1 char:124
+ $startdate = (Get-ChildItem e:\test\*.csv | Sort-Object -Property lastwriteti
me | Select-Object -First 1).lastwritetime dir  <<<< c:\test\*.log | ?{$_.lastw
ritetime -gt $startdate} | Select-String ([regex]::escape("421 4.3.2 Service no
t available, closing transmission channel")) Select-Object line | Export-Csv -P
ath "c:\test\LOG-$(get-date -f 'dd-MM-yyy').log"
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33640888
The newline is missing from the code you runs. There should be a new line before "dir", after lastwritetime.
0
 

Author Comment

by:cawasaki
ID: 33640892
ok i get it work but:

1- the output file is empty, it only list the line in powershell mmc.

2- the script list file name, and i not need it, i need jut the entire line that include given expression in my csv, not need filename.

thanks for your help
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33640986
I think you also missed the pipe character (|) from the following place:
... channel")) | Select-Object ...
(It is hard to help, if you do not copy the exact code...)
0
 

Author Comment

by:cawasaki
ID: 33641616
i will test this tomorow
0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33644041
What is output from command date /t ?
0
 

Author Comment

by:cawasaki
ID: 33644385
the output of date/t:

10/09/2010
0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33644453
Yeah. My output on english version Windows 2003 server is Fri 09/10/2010.
Correction is only in first tokens parameter. tokens parameter is used for choosing
which values command retrieve - delimiters between valueses is in delim parameter (there
is / and space. Bellow is full script which will work in your international setting. If you move this
script to another system simply type date/t and based on result you only change tokens and/or
delims parameter of the first FOR command.

for /f "tokens=* delims=/ " %%i in ('date /t') do set currentdate=%%i-%%j-%%k
for /f "tokens=*" %%I in ('dir e:\smtplog /b /A-D') do FIND /I "421 4.3.2 Service not available, closing transmission channel" e:\smtplog\%%I >> e:\LOG-%currentdate%.txt
0
 

Author Comment

by:cawasaki
ID: 33644457
soostibi:

Your script work now, but i have an error when i lunsh the script:

Select-String : The file can not be read: C:\test\LOG-10-09-2010.log
At C:\test\PLIP.ps1:2 char:71
+ dir c:\test\*.log | ?{$_.lastwritetime -gt $startdate} | Select-String  <<<<
([regex]::escape("421 4.3.2 Service not available, closing transmission channel
")) |
0
 

Author Comment

by:cawasaki
ID: 33644579
Tominov:

Not work with new script:

C:\test>FIND /I "421 4.3.2 Service not available, closing transmission channel"
c:\test\test3.bat  1>>e:\test\LOG-10/09/2010
The system cannot find the path specified.
0
 
LVL 9

Assisted Solution

by:Tomas Valenta
Tomas Valenta earned 250 total points
ID: 33644601
My mistake, sorry.
for /f "tokens=1-3 delims=/ " %%i in ('date /t') do set currentdate=%%i-%%j-%%k
for /f "tokens=*" %%I in ('dir e:\smtplog /b /A-D') do FIND /I "421 4.3.2 Service not available, closing transmission channel" e:\smtplog\%%I >> e:\LOG-%currentdate%.txt
0
 
LVL 13

Expert Comment

by:soostibi
ID: 33644610
Can you open the C:\test\LOG-10-09-2010.log file with notepad? Maybe it is locked by some other processes...
0
 

Author Comment

by:cawasaki
ID: 33644708
Tominov:

Its work now, its possible to delete this part from output file:

---------- C:\TEST\LOG.LOG
2010-09-08T22:06:31.121Z,,08CD09A6F872EB85,1,10.200.4.228:25,10.33.22.21:50174,>,"421 4.3.2 Service not available, closing transmission channel",

---------- C:\TEST\PLIP.PS1
dir c:\test\*.log | ?{$_.lastwritetime -gt $startdate} | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel")) |    

---------- C:\TEST\PLOP.PS1
dir c:\test\*.log | ?{$_.lastwritetime -gt $startdate} | Select-String ([regex]::escape("421 4.3.2 Service not available, closing transmission channel"))

---------- C:\TEST\TEST.BAT

---------- C:\TEST\TEST2.BAT
for /f "tokens=*" %%I in ('dir c:\test /b /A-D') do FIND /I "421 4.3.2 Service not available, closing transmission channel" c:\test\%%I >> c:\test\LOG-%currentdate%.txt

---------- C:\TEST\TEST3.BAT
for /f "tokens=*" %%I in ('dir c:\test /b /A-D') do FIND /I "421 4.3.2 Service not available, closing transmission channel" c:\test\%%I >> C:\test\LOG-%currentdate%.txt
0
 

Author Comment

by:cawasaki
ID: 33644715
Soostibi:

Can you open the C:\test\LOG-10-09-2010.log file with notepad? Maybe it is locked by some other processes...

==>LOG-10-09-2010.log is the output file of your script!
0
 
LVL 9

Expert Comment

by:Tomas Valenta
ID: 33644734
it is easy - you must have logfile and script saved in another location then smtplogs. Thats all.
0
 

Author Comment

by:cawasaki
ID: 33644838
Ok perfect,

The 2 script is OK.

Thanks tominov and Soostibi.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

We are happy to announce a brand new addition to our line of acclaimed email signature management products – CodeTwo Email Signatures for Office 365.
A procedure for exporting installed hotfix details of remote computers using powershell
In this video we show how to create an Accepted Domain in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Ac…
To add imagery to an HTML email signature, you have two options available to you. You can either add a logo/image by embedding it directly into the signature or hosting it externally and linking to it. The vast majority of email clients display l…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now