Exchange 2007 email tracking

Hey,

I did this for a few of the domains I look after on my own server. I ended up using the SMTP service logs (provided Verbose logging is enabled there) along with the Agent Log. I found message tracking to be too slow, not that doing it my way is exactly fast.

It's not pretty but I am able to show these for a given domain...

Total Inbound Messages
Total Outbound Messages
Total Recipients
Total Rejected
Top 10 Recipients
Top 10 Sendesr
Total Inbound from Local Network
Top 10 Recipients with Spam Probability over 70% (SCL 7)

Are you open to scripting this? I might need to optimise my code a bit, its a bit rough :)

Chris

Not really sure why GFI was suggested - I am not aware that they have a tool that will process the message tracking logs.

There are lots of tools that will - the top of the tree (both functionality and price) is Quest Message Stats. Promodag have a tool as well, plus they have a tool that will import the logs in to Access if you have Access skills.
Then there is Mail Detective, Appanalyzer, plus scripts.
http://www.amset.info/exchange/message-tracking.asp

Simon.

Chris-Dent -
I am not opposed to scripting at all. I have all the logging enabled - I tried extracting the info I was looking for from those files but failed to get the info I was looking for.
If you have a script that does this already (not really asking that you write one from scratch) that you wouldn't mind sharing that would be fantastic.

Mestha: Thanks for that application names and links, i will check into those asap.

Thanks both of you.
Richard

Hi Richard,

I do, I'll tidy it up in the morning and post it.

Chris

Okay so morning was a bit ambitious. This is the script. It needs a bit of explanation...

I advise you do not run this on your Exchange Server. The very first thing it does is copy a range of log files from the Exchange server onto whatever system is running the script. All work against the logs is done on that system, the Exchange server isn't necessary for this.

The system running the script needs PowerShell installed, but does not require the Exchange System Tools (or anything except the default PowerShell installation).

The three values at the very top of the script should be set. The script will only return results for "@domain.com". Everything else is ignored.

Depending on how many logs you have any want to analyse you may run into problems with memory usage (on the system running the script). This is most likely to occur when the script attempts to combine the SmtpReceive log with the AgentLog. Frankly the Agent Log is a pain in the proverbial and I've had trouble coming up with a neater way to deal with it.

The results produced, held in $Inbound and $Outbound at the end are pretty basic (in terms of formatting). Accessing the more complex sets can be done like this:

$Inbound | Select -expand "Top 10 Recipients for spam probability over 70%

Or like this:

$Inbound."Top 10 Recipients for spam probability over 70%"

If you can think of any other statistics you'd like I can add them (within reason of course :)).

Chris

# Server Name to process logs for
$ExchangeServer = "AnExchangeServer"
 
# Period to analyse in days
$Period = 7
 
# Domain Name to examine
$Domain = "@domain.com"
 
 
Function New-LogFolderObject($Source, $Destination) {
  $LogFolder = New-Object System.Object
  $LogFolder | Add-Member -Type NoteProperty -Name "Source" -Value $Source
  $LogFolder | Add-Member -Type NoteProperty -Name "Destination" -Value $Destination
  Return $LogFolder
}
 
Function Get-LogFolders {
  # These paths are hard-coded to the default path values here
 
  $LogFolders = @()
  $LogFolders += New-LogFolderObject `
    "\\$ExchangeServer\c$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive" `
    "SmtpReceive"
  $LogFolders += New-LogFolderObject `
    "\\$ExchangeServer\c$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpSend" `
    "SmtpSend"
  $LogFolders += New-LogFolderObject `
    "\\$ExchangeServer\c$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\AgentLog" `
    "AgentLog"
  Return $LogFolders
}
 
Function Get-LogFiles($Source, $Destination) {
  # Copy the log files off the Exchange server onto a system which can be used to analyse each
 
  If (!(Test-Path $Destination)) { New-Item -Type Directory -Path $Destination }
 
  Get-ChildItem $Destination | %{ $_.Delete() }
  Get-ChildItem $Source | ?{ $_.LastWriteTime -gt (Get-Date).Date.AddDays(-$Period) } | %{
    $_.CopyTo("$((Get-Location).Path)\$Destination\$($_.Name)")
  }
}
 
Function WriteLineMatchingPattern ($SourcePath, $DestinationPath, $Pattern, $Replace, $StopOnMatch) {
  # Faster than Get-Content so better for large amounts of data
 
  $StreamReader = New-Object System.IO.StreamReader($SourcePath)
  Do {
    $Line = $StreamReader.ReadLine()
    If ($Line -Like "*$Pattern*") {
      If ($Replace) {
        $Line -Replace $Pattern >> $DestinationPath
      } else {
        $Line >> $DestinationPath
      }
      If ($StopOnMatch) { $Match = $True }
    }
  } Until ($StreamReader.EndOfStream -eq $True -Or $Match)
}
 
Function WriteLineMatchingHashKey ($SourcePath, $DestinationPath, $Index, $Hash) {
  # Returns a line from a comma delimited file where the field at $Index matches a key in $Hash
 
  $StreamReader = New-Object System.IO.StreamReader($SourcePath)
  Do {
    $Line = $StreamReader.ReadLine()
    If ($Hash."$($Line.Split(',')[$Index])") {
      $Line >> $DestinationPath
    }
  } Until ($StreamReader.EndOfStream -eq $True)
}
 
Function GenerateInboundStatistics($InboundLog) {
  # Total Inbound
 
  $TotalInbound = ($InboundLog | Select-Object SessionId -Unique | Measure-Object).Count
 
  # Top 10 Recipients
 
  $RecipientCount = @()
  $UniqueRecipients = $InboundLog | Select-Object Recipient -Unique
  ForEach ($Recipient in $UniqueRecipients) {
    $RecipientCount += $Recipient | Select-Object Recipient, `
      @{n='Count';e={ ($InboundLog | ?{ $_.Recipient -eq $Recipient.Recipient } | Measure-Object).Count }}
  }
  $Top10Recipients = $RecipientCount | Sort-Object -Descending -Property Count | Select-Object -First 10
 
  # Total Inbound Rejected
 
  $TotalInboundRejected = ($InboundLog | ?{ $_.Action -ne 'AcceptMessage' } | Measure-Object).Count
 
  # Total Inbound from Private IP Ranges
 
  $InboundFromPrivateIP = $InboundLog | ?{ $_.RemoteIP -match "10`.*`.*`.*|192`.168`.*`.*|172`.[16..31]`.*`.*" }
  $InboundFromPrivateIPCount = ($InboundFromPrivateIP | Measure-Object).Count
 
  # Top 10 Internal SMTP Clients (does not include MAPI clients)
 
  $UniqueSystemsCount = @()
  $UniqueSystems = $InboundFromPrivateIP | Select-Object RemoteIP -Unique
  ForEach ($SourceIP in $UniqueSystems) {
    $UniqueSystemsCount += $SourceIP | Select-Object RemoteIP, `
      @{n='Count';e={ ($InboundFromPrivateIP | ?{ $_.RemoteIP  -eq $SourceIP.RemoteIP } | Measure-Object).Count }}
  }
  $Top10PrivateSMTPClients = $UniqueSystemsCount | Sort-Object -Descending -Property Count | Select-Object -First 10
 
  # Top 10 Recipients with spam probability over 70%
 
  $ProbablySpam = $InboundLog | ?{ $_.ReasonData -gt 6 }
  $UniqueRecipients = $ProbablySpam | Select-Object Recipient -Unique
  ForEach ($Recipient in $UniqueRecipients) {
    $RecipientCount += $Recipient | Select-Object Recipient, `
      @{n='Count';e={ ([Array]($ProbablySpam | ?{ $_.Recipient -eq $Recipient.Recipient })).Count }}
  }
  $Top10SpamRecipients = $RecipientCount | Sort-Object -Descending -Property Count | Select-Object -First 10
 
  $InboundStatistics = New-Object System.Object
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Total Inbound" -Value $TotalInbound
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Top 10 Recipients" -Value $Top10Recipients
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Total Inbound Rejected" -Value $TotalInboundRejected
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Total Inbound from Private IP Ranges" -Value $InboundFromPrivateIPCount
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Top 10 Internal SMTP Clients" -Value $Top10PrivateSMTPClients
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Top 10 Recipients for spam probability over 70%" -Value $Top10SpamRecipients
  Return $InboundStatistics
}
 
Function GenerateOutboundStatistics($OutboundLog) {
  # Total Outbound
 
  $TotalOutbound = ($OutboundLog | Select-Object SessionId -Unique | Measure-Object).Count
 
  # Top 10 Senders
 
  $SenderCount = @()
  $UniqueSenderse = $OutboundLog | Select-Object Sender -Unique
  ForEach ($Sender in $UniqueSenders) {
    $SenderCount += $Sender | Select-Object Sender, `
      @{n='Count';e={ ($OutboundLog | ?{ $_.Recipient -eq $Sender.Sender } | Measure-Object).Count }}
  }
  $Top10Senders = $SenderCount | Sort-Object -Descending -Property Count | Select-Object -First 10
 
  $OutboundStatistics = New-Object System.Object
  $OutboundStatistics | Add-Member -Type NoteProperty -Name "Total Outbound" -Value $TotalOutbound
  $OutboundStatistics | Add-Member -Type NoteProperty -Name "Top 10 Senders" -Value $Top10Senders
 
  Return $OutboundStatistics
}
 
#
# Main Code
#
 
# This portion of the script deals with removing unwanted information from the log files. 
# Once done the files should be small enough to read using standard PowerShell CmdLets.
 
# Copy each of the files down to the local system
 
Get-LogFolders | %{ Get-LogFiles $_.Source $_.Destination }
 
# Assume all the header line is consistent across all log files for each log type.
# The header is present on the 5th line of each file. The file reader will stop when it finds the line.
 
# Receive Log
[Void](New-Item -Name "SmtpReceive.log" -Type File -Force)
WriteLineMatchingPattern (Get-ChildItem "SmtpReceive")[0].FullName "SmtpReceive.log" "#Fields: " $True $True
 
# Send Log
[Void](New-Item -Name "SmtpSend.log" -Type File -Force)
WriteLineMatchingPattern (Get-ChildItem "SmtpSend")[0].FullName "SmtpSend.log" "#Fields: " $True $True
 
# Agent Log
[Void](New-Item -Name "AgentLog.log" -Type File -Force)
WriteLineMatchingPattern (Get-ChildItem "AgentLog")[0].FullName "AgentLog.log" "#Fields: " $True $True
 
# Pull out all relevant content from the source log files (SmtpReceive and SmtpSend)
 
Get-ChildItem "SmtpReceive" | %{
  WriteLineMatchingPattern $_.FullName "SmtpReceive.log" "RCPT TO*$Domain" }
 
Get-ChildItem "SmtpSend" | %{
  WriteLineMatchingPattern $_.FullName "SmtpSend.log" "MAIL FROM*$Domain" }
 
# Session IDs are required from the Receive Log to bring in the Agent Log. The ReceiveLog should be small enough at this stage
# to cope with Import-CSV
 
$SessionIds = @{}
Import-CSV "SmtpReceive.log" | Select-Object "Session-Id" -Unique | %{
  $SessionIds.Add($_."session-id", $True) }
 
# Get the Index of the SessionId from the AgentLog.log file
 
$i = 0; $j = 0
(Get-Content "AgentLog.log").Split(",") | %{
  If ($_ -eq "SessionId") { $i = $j } else { $j++ } }
 
# Pull out all relevant content from the source log files (AgentLog)
 
Get-ChildItem "AgentLog" | %{
  WriteLineMatchingHashKey $_.FullName "AgentLog.log" $i $SessionIds }
 
# Combining the AgentLog and ReceiveLog. No unique identifiers, only semi-unique combinations. Makes this hard work.
# Watch system memory usage during this section.
 
$SmtpReceive = Import-CSV "SmtpReceive.log"
$AgentLog = Import-CSV "AgentLog.log"
 
$InboundLog = @()
ForEach ($Entry in $SmtpReceive) {
 
  # Is there a specific entry in the Agent log for this recipient
  $AgentLogEntry = $AgentLog | ?{ $Entry."session-id" -eq $_.SessionId -And `
    $($Entry.data -Replace "RCPT TO:<|>").ToLower() -eq $_.Recipient.ToLower() }
 
  If (!$AgentLogEntry) {
    # Otherwise it might be captured under an AgentLog entry for multiple recipients
    $AgentLogEntry = $AgentLog | ?{ $Entry."session-id" -eq $_.SessionId -And $_.NumRecipients -gt 1 }
  }
 
  # Or maybe we just missed this altogether, may be a slight discrepancy between files times
  If (!$AgentLogEntry) { 
    Write-Host $Entry."session-id"
  } Else {
 
    # Otherwise the combined result is added to the log
 
    $InboundLog += $Entry | Select-Object `
      @{n='TimeStamp';e={ Get-Date($_."date-time") }}, `
      @{n='Recipient';e={ ($_.data -Replace "RCPT TO:<|>").ToLower() }}, `
      @{n='SessionId';e={ $AgentLogEntry.SessionId }}, `
      @{n='MessageId';e={ $AgentLogEntry.MessageId }}, `
      @{n='ConnectorId';e={ $_."connector-id" }}, `
      @{n='LocalIP';e={ $_."local-endpoint" -Replace ":.*" }}, `
      @{n='RemoteIP';e={ $_."remote-endpoint" -Replace ":.*" }}, `
      @{n='NumReceipients';e={ $AgentLogEntry.NumRecipients }}, `
      @{n='Event';e={ $AgentLogEntry.Event }}, `
      @{n='Action';e={ $AgentLogEntry.Action }}, `
      @{n='SmtpResponse';e={ $AgentLogEntry.SmtpResponse }}, `
      @{n='Reason';e={ $AgentLogEntry.Reason }}, `
      @{n='ReasonData';e={ $AgentLogEntry.ReasonData }}, `
      @{n='Diagnostics';e={ $AgentLogEntry.Diagnostics }}
  }
}
 
# Export it to a file
 
$InboundLog | Export-CSV "Inbound.csv"
 
# Tidy up the SmtpSend.log file
 
$OutboundLog = Import-CSV "SmtpSend.log"
$OutboundLog = $OutboundLog | Select-Object `
  @{n='TimeStamp';e={ Get-Date($_."date-time") }}, `
  @{n='Sender';e={ ($_.data -Replace "MAIL FROM:<|>.*").ToLower() }}, `
  @{n='SessionId';e={ $_."session-id" }}, `
  @{n='ConnectorId';e={ $_."connector-id" }}, `
  @{n='LocalIP';e={ $_."local-endpoint" -Replace ":.*" }}, `
  @{n='RemoteIP';e={ $_."remote-endpoint" -Replace ":.*" }}, `
  @{n='Size';e={ $_.data -Replace ".*SIZE`=" }}
$OutboundLog | Export-CSV "Outbound.csv"
 
# Generate statistics
 
$Inbound = GenerateInboundStatistics $InboundLog
$Outbound = GenerateOutboundStatistics $OutboundLog

Select allOpen in new window

Thank you for that script. I am going to read it over and test it out.
To make this clear if anyone else reads this posting I do have a few questions as I am not a programmer :)
1. What type of code this is?
2. Do I just call it from Powershell after saving in x format?
3. Does it accept switches or does it just return all the statistics?

I thank you very much for your time and effort on this.

Richard

Works great thanks.

That was quick :) Did you need it to do anything else?

Chris

No I think this about does it - I will play with it more as I think that it pulls some usefully information. What I really needed to know was simply if any emails were coming in on a specific domain.
I do get some errors but I am chalking that up to the fact that I just enabled logging in this particualar test environment. Going to try it in the live domain later today.
Thanks for the script and the help -
Richard

It actually isn't writing to the .csv files.

This is the error I get.

PS C:\Scripts\Powershell> C:\Scripts\Powershell\EmailDomainStats.ps1
 
 
    Directory: C:\Scripts\Powershell
 
 
Mode                LastWriteTime     Length Name                                                                                                                          
----                -------------     ------ ----                                                                                                                          
d----          7/9/2009  11:42 AM            SmtpReceive                                                                                                                   
-a---          7/9/2009  11:37 AM     142193 RECV20090709-1.LOG                                                                                                            
d----          7/9/2009  11:42 AM            SmtpSend                                                                                                                      
-a---          7/9/2009  11:40 AM       3791 SEND20090709-1.LOG                                                                                                            
d----          7/9/2009  11:42 AM            AgentLog                                                                                                                      
-a---         6/10/2009   5:54 PM      49394 AgentLog20090610-1.LOG                                                                                                        
-a---         6/11/2009   5:54 PM      45414 AgentLog20090611-1.LOG                                                                                                        
-a---         6/12/2009   6:24 PM      34728 AgentLog20090612-1.LOG                                                                                                        
-a---         6/13/2009   2:51 PM      11837 AgentLog20090613-1.LOG                                                                                                        
-a---         6/14/2009   4:48 PM       6994 AgentLog20090614-1.LOG                                                                                                        
-a---         6/15/2009   5:51 PM      34949 AgentLog20090615-1.LOG                                                                                                        
-a---         6/16/2009   5:55 PM      51597 AgentLog20090616-1.LOG                                                                                                        
-a---         6/17/2009   5:51 PM      48635 AgentLog20090617-1.LOG                                                                                                        
-a---         6/18/2009   5:54 PM      49534 AgentLog20090618-1.LOG                                                                                                        
-a---         6/19/2009   5:41 PM      36678 AgentLog20090619-1.LOG                                                                                                        
-a---         6/20/2009   2:45 PM      10355 AgentLog20090620-1.LOG                                                                                                        
-a---         6/21/2009   4:37 PM       8563 AgentLog20090621-1.LOG                                                                                                        
-a---         6/22/2009   5:56 PM      45221 AgentLog20090622-1.LOG                                                                                                        
-a---         6/23/2009   5:17 PM      57316 AgentLog20090623-1.LOG                                                                                                        
-a---         6/24/2009   5:21 PM      55932 AgentLog20090624-1.LOG                                                                                                        
-a---         6/25/2009   5:35 PM      49249 AgentLog20090625-1.LOG                                                                                                        
-a---         6/26/2009   3:35 PM      35661 AgentLog20090626-1.LOG                                                                                                        
-a---         6/29/2009   5:57 PM      73220 AgentLog20090629-1.LOG                                                                                                        
-a---         6/30/2009   5:46 PM      49788 AgentLog20090630-1.LOG                                                                                                        
-a---          7/1/2009   5:45 PM      58858 AgentLog20090701-1.LOG                                                                                                        
-a---          7/2/2009   5:13 PM      55992 AgentLog20090702-1.LOG                                                                                                        
-a---          7/3/2009   5:58 PM      16819 AgentLog20090703-1.LOG                                                                                                        
-a---          7/4/2009   2:26 PM       6512 AgentLog20090704-1.LOG                                                                                                        
-a---          7/5/2009   5:29 PM       7843 AgentLog20090705-1.LOG                                                                                                        
-a---          7/6/2009   5:44 PM      39777 AgentLog20090706-1.LOG                                                                                                        
-a---          7/7/2009   5:59 PM      42436 AgentLog20090707-1.LOG                                                                                                        
-a---          7/8/2009   5:59 PM      45826 AgentLog20090708-1.LOG                                                                                                        
-a---          7/9/2009  11:37 AM      32973 AgentLog20090709-1.LOG                                                                                                        
Unable to index into an object of type System.IO.FileInfo.
At C:\Scripts\Powershell\DomainStats.ps1:166 char:56
+ WriteLineMatchingPattern (Get-ChildItem "SmtpReceive")[ <<<< 0].FullName "SmtpReceive.log" "#Fields: " $True $True
    + CategoryInfo          : InvalidOperation: (0:Int32) [], RuntimeException
    + FullyQualifiedErrorId : CannotIndex
 
Unable to index into an object of type System.IO.FileInfo.
At C:\Scripts\Powershell\DomainStats.ps1:170 char:53
+ WriteLineMatchingPattern (Get-ChildItem "SmtpSend")[ <<<< 0].FullName "SmtpSend.log" "#Fields: " $True $True
    + CategoryInfo          : InvalidOperation: (0:Int32) [], RuntimeException
    + FullyQualifiedErrorId : CannotIndex
 
You cannot call a method on a null-valued expression.
At C:\Scripts\Powershell\DomainStats.ps1:213 char:77
+     $($Entry.data -Replace "RCPT TO:<|>").ToLower() -eq $_.Recipient.ToLower <<<< () }
    + CategoryInfo          : InvalidOperation: (ToLower:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull
 
 
Export-Csv : Cannot bind argument to parameter 'InputObject' because it is null.
At C:\Scripts\Powershell\DomainStats.ps1:260 char:26
+ $OutboundLog | Export-CSV <<<<  "Outbound.csv"
    + CategoryInfo          : InvalidData: (:) [Export-Csv], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ExportCsvCommand

Select allOpen in new window

I'll have a more robust version of this shortly. We'll revisit the errors with that one.

Chris

No problem no rush. If there is anything specific about my Exchange server that you need to know let me know. Though other then it being installed on E: and not C: it is a normal full install on a single machine.
Richard

Yeah that would have broken it :) Not to worry, this one is rather more flexible and should be quite a lot faster.

I think this command should do what you want:

$Results = ./ScriptName.ps1 -ExchangeServer "SomeServer" -ExchangeDrive "e" -InboundOnly -ExcludeAgentLogs
$Results

Or just this if you don't care about any of the other results:

./ScriptName.ps1 -ExchangeServer "SomeServer" -ExchangeDrive "e" -InboundOnly -ExcludeAgentLogs

Assuming you run the script from the same directory as you have open in PowerShell.

That's about the most basic run mode. Might want to check the paths for the logs though, it still expects this type:

      \\ExchangeServer\e$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive

If that's not going to be right I'll stick in an option for the Exchange installation path (rather than just drive).

Chris

# Simple Log File Analysis for Exchange 2007
 
# Usage:
#
# Download logs from the server and process those
# $Results = ./ScriptName.ps1 -DomainName "@domain.com" -ExchangeServer "someserver"
# Download logs from server (alternate drive) and process Inbound path only
# $Results = ./ScriptName.ps1 -ExchangeServer "SomeServer" -ExchangeDrive "e" -InboundOnly -ExcludeAgentLogs
# Run without downloading logs from the server (logs must exist)
# $Results = ./ScriptName.ps1 -DomainName "@domain.com" -UseExistingLogFiles
# Inbound path only, basic attribute (Total, Top10Recipients, SMTP Client details)
# $Results = ./ScriptName.ps1 -DomainName "@domain.com" -InboundOnly -ExcludeAgentLogs
 
# Parameters for this script
 
Param(
  [String]$DomainName = $( Throw("DomainName is required") ),  # Domain Name in the form "domain.com" or "@domain.com"
  [Int]$Period = 7,                                           # Extract logs for previous 7 days
  [String]$ExchangeServer,                                    # Exchange Server to pull log files from
  [String]$ExchangeDrive = "C",                               # Drive for Exchange Installation (used to access administrative share)
  [Switch]$UseExistingLogFiles = $False,                      # If log files have already been downloaded
  [Switch]$InboundOnly,                                       # Retrieve / Process only SmtpReceiveLogs
  [Switch]$OutboundOnly,                                      # Retrieve / Process only SmtpSendLogs
  [Switch]$ExcludeAgentLogs                                   # Ignore the AgentLogs
)
 
#
# Functions
#
 
Function New-LogFolderObject($Source, $Destination) {
  $LogFolder = New-Object Object
  $LogFolder | Add-Member -Type NoteProperty -Name "Source" -Value $Source
  $LogFolder | Add-Member -Type NoteProperty -Name "Destination" -Value $Destination
  Return $LogFolder
}
 
Function Get-LogFolders ([String]$ExchangeServer) {
  # These paths are hard-coded to the default path values here
  # Alternative requires Exchange System Tools as well
 
  If (!$ExchangeServer) { Throw("No Exchange Server specified") }
 
  $LogFolders = @()
 
  If (!$OutboundOnly) {
    $LogFolders += New-LogFolderObject `
      "\\$ExchangeServer\$ExchangeDrive`$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive" `
      "SmtpReceive"
  }
  If (!$InboundOnly) {
    $LogFolders += New-LogFolderObject `
      "\\$ExchangeServer\$ExchangeDrive`$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpSend" `
      "SmtpSend"
  }
  If (!$ExcludeAgentLogs -And !$OutboundOnly) {
    $LogFolders += New-LogFolderObject `
      "\\$ExchangeServer\$ExchangeDrive`$\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\AgentLog" `
      "AgentLog"
  }
  Return $LogFolders
}
 
Function Get-LogFiles($Source, $Destination) {
  # Copy the log files off the server onto a system which can be used to analyse each
 
  If (!(Test-Path $Destination)) { New-Item -Type Directory -Path $Destination }
  Get-ChildItem $Destination | %{ $_.Delete() }
 
  $Logs = Get-ChildItem $Source | ?{ $_.LastWriteTime -gt (Get-Date).Date.AddDays(-$Period) }
  $i = 0
  If (!$Logs) {
    # Terminate the script if no log files are present
    Throw("Error: No log files present in $Source for the requested Period ($Period days)")
  } Else {
    $Logs | %{ 
      $i++; $Progress = ($i / $Logs.Count) * 100
      Write-Progress -Activity "Copying Logs" -Status "Copying $($_.Name) to $Destination" -PercentComplete $Progress
      [Void]($_.CopyTo("$((Get-Location).Path)\$Destination\$($_.Name)"))
    }
    Write-Progress -Activity "Copying Logs" -Status "Complete" -Completed
  }
}
 
Function WriteLineMatchingPattern ($SourcePath, $DestinationPath, $Pattern, $Replace, $StopOnMatch) {
  # Faster than Get-Content so better for large amounts of data
 
  $StreamReader = New-Object IO.StreamReader($SourcePath)
  Do {
    $Line = $StreamReader.ReadLine()
    If ($Line -Like "*$Pattern*") {
      If ($Replace) {
        $Line -Replace $Pattern >> $DestinationPath
      } else {
        $Line >> $DestinationPath
      }
      If ($StopOnMatch) { $Match = $True }
    }
  } Until ($StreamReader.EndOfStream -eq $True -Or $Match)
}
 
Function WriteLineMatchingHashKey ($SourcePath, $DestinationPath, $Index, $Hash) {
  # Returns a line from a comma delimited file where the field at $Index matches a key in $Hash
 
  $StreamReader = New-Object IO.StreamReader($SourcePath)
  Do {
    $Line = $StreamReader.ReadLine()
    If ($Hash."$($Line.Split(',')[$Index])") {
      $Line >> $DestinationPath
    }
  } Until ($StreamReader.EndOfStream -eq $True)
  $StreamReader.Dispose()
}
 
Function CreateObject($Fields, $Data) {
  $Object = New-Object Object
  For ($i = 0; $i -lt $Fields.Count; $i++) {
    $Object | Add-Member -Type NoteProperty -Name $($Fields[$i]) -Value $($Data[$i])
  }
  Return $Object
}
 
Function ImportCSVToHash ($FilePath, $Key, $KeyIndex) {
  $StreamReader = New-Object IO.StreamReader($FilePath)
  # Read the Header
  $Fields = $StreamReader.ReadLine().Split(",")
 
  $Hash = @{}
 
  Write-Progress -Activity "Importing AgentLog.log" -Status "Sorting data into Hash"
  If (!$StreamReader.EndOfStream) {
    Do {
      $Data = $StreamReader.ReadLine().Split(",")
      $Object = CreateObject $Fields $Data
 
      If ($Hash.$($Data[$KeyIndex])) {
        $Hash.$($Data[$KeyIndex]) += $Object
      } Else {
        $Hash.Add($Data[$KeyIndex], @($Object))
      }
    } Until ($StreamReader.EndOfStream -eq $True)
  }
  Write-Progress -Activity "Importing AgentLog.log" -Status "Complete" -Completed
 
  $StreamReader.Dispose()
  Return $Hash
}
 
Function CombineLogs ($SmtpReceive, $AgentLog, $SessionIdIndex) {
  # Import the SmtpReceive Log
  $SmtpReceive = Import-CSV $SmtpReceive
  $AgentLog = ImportCSVToHash (Get-Item $AgentLog).FullName "SessionId" $SessionIdIndex
 
  $CombinedEntries = @(); $i = 0
  ForEach ($SmtpLogEntry in $SmtpReceive) {
    $SessionId = $SmtpLogEntry."session-id"
    $Recipient = ($SmtpLogEntry.data -Replace "RCPT TO:<|>").ToLower()
 
    $i++; $Progress = ($i / $SmtpReceive.Count) * 100
    Write-Progress -Activity "Matching SmtpReceive to AgentLog" -Status "[$i/$($SmtpReceive.Count)] Matching $SessionId and $Recipient" -PercentComplete $Progress
 
    # The set of entries that may relate to this mail
 
    If ($AgentLog.$SessionId) {
      $AgentLog.$SessionId | %{
        If ($_.Recipient.ToLower() -eq $Recipient) {
          # An entry for this recipient
          $AgentLogEntry = $_
        } ElseIf ($_.NumRecipients -gt 1) {
          # An entry covering multiple recipients
          $AgentLogEntry = $_
        }
      }
 
      $CombinedEntries += $SmtpLogEntry | Select-Object `
        @{n='TimeStamp';e={ Get-Date($_."date-time") }}, `
        @{n='Recipient';e={ $Recipient }}, `
        @{n='SessionId';e={ $AgentLogEntry.SessionId }}, `
        @{n='MessageId';e={ $AgentLogEntry.MessageId }}, `
        @{n='ConnectorId';e={ $_."connector-id" }}, `
        @{n='LocalIP';e={ $_."local-endpoint" -Replace ":.*" }}, `
        @{n='RemoteIP';e={ $_."remote-endpoint" -Replace ":.*" }}, `
        @{n='NumReceipients';e={ $AgentLogEntry.NumRecipients }}, `
        @{n='Event';e={ $AgentLogEntry.Event }}, `
        @{n='Action';e={ $AgentLogEntry.Action }}, `
        @{n='SmtpResponse';e={ $AgentLogEntry.SmtpResponse }}, `
        @{n='Reason';e={ $AgentLogEntry.Reason }}, `
        @{n='ReasonData';e={ $AgentLogEntry.ReasonData }}, `
        @{n='Diagnostics';e={ $AgentLogEntry.Diagnostics }}
    }
  }
  Write-Progress -Activity "Matching SmtpReceive to AgentLog" -Status "Complete" -Completed
 
  Return $CombinedEntries
}
 
Function GenerateInboundStatistics($InboundLog, $SessionIds) {
  $InboundStatistics = New-Object Object
 
  # Total Inbound
 
  Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "Total"
 
  $TotalInbound = $SessionIds.Count
  $InboundStatistics | Add-Member -Type NoteProperty -Name "TotalInbound" -Value $TotalInbound
 
  # Top 10 Recipients
 
  Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "Top 10 Recipients"
 
  $RecipientCount = @()
  $UniqueRecipients = $InboundLog | Select-Object Recipient -Unique
  ForEach ($Recipient in $UniqueRecipients) {
    $RecipientCount += $Recipient | Select-Object Recipient, `
      @{n='Count';e={ ($InboundLog | ?{ $_.Recipient -eq $Recipient.Recipient } | Measure-Object).Count }}
  }
  $Top10Recipients = $RecipientCount | Sort-Object -Descending -Property Count | Select-Object -First 10
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Top10Recipients" -Value $Top10Recipients
 
  # Total Inbound from Private IP Ranges
 
  Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "From Private IP ranges"
 
  $InboundFromPrivateIP = $InboundLog | ?{ $_.RemoteIP -match "10`.*`.*`.*|192`.168`.*`.*|172`.[16..31]`.*`.*" }
  $InboundFromPrivateIPCount = ($InboundFromPrivateIP | Measure-Object).Count
  $InboundStatistics | Add-Member -Type NoteProperty -Name "TotalInboundFromPrivateIP" -Value $InboundFromPrivateIPCount
 
  # Top 10 Internal SMTP Clients (does not include MAPI clients)
 
  Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "Top 10 Internal SMTP Clients"
 
  $UniqueSystemsCount = @()
  $UniqueSystems = $InboundFromPrivateIP | Select-Object RemoteIP -Unique
  ForEach ($SourceIP in $UniqueSystems) {
    $UniqueSystemsCount += $SourceIP | Select-Object RemoteIP, `
      @{n='Count';e={ ($InboundFromPrivateIP | ?{ $_.RemoteIP  -eq $SourceIP.RemoteIP } | Measure-Object).Count }}
  }
  $Top10PrivateSMTPClients = $UniqueSystemsCount | Sort-Object -Descending -Property Count | Select-Object -First 10
  $InboundStatistics | Add-Member -Type NoteProperty -Name "Top10PrivateSMTPClients" -Value $Top10PrivateSMTPClients
 
  If (!$ExcludeAgentLogs) {
    # Total Inbound Rejected
 
    Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "Total Rejected"
 
    $TotalInboundRejected = ($InboundLog | ?{ $_.Action -ne 'AcceptMessage' } | Measure-Object).Count
    $InboundStatistics | Add-Member -Type NoteProperty -Name "TotalInboundRejected" -Value $TotalInboundRejected
 
    # Top 10 Recipients with spam probability over 70%
 
    Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "Top 10 Spam Recipients"
 
    $ProbablySpam = $InboundLog | ?{ $_.ReasonData -gt 6 }
    $UniqueRecipients = $ProbablySpam | Select-Object Recipient -Unique
    ForEach ($Recipient in $UniqueRecipients) {
      $RecipientCount += $Recipient | Select-Object Recipient, `
        @{n='Count';e={ ([Array]($ProbablySpam | ?{ $_.Recipient -eq $Recipient.Recipient })).Count }}
    }
    $Top10SpamRecipients = $RecipientCount | Sort-Object -Descending -Property Count | Select-Object -First 10
    $InboundStatistics | Add-Member -Type NoteProperty -Name "Top10SpamRecipients-Over70%" -Value $Top10SpamRecipients
  }
 
  Write-Progress -Activity "Generating Statistics for Inbound Mail" -Status "Complete" -Completed
 
  Return $InboundStatistics
}
 
Function GenerateOutboundStatistics($OutboundLog) {
  $OutboundStatistics = New-Object Object
 
  # Total Outbound
 
  Write-Progress -Activity "Generating Statistics for Outbound Mail" -Status "Total"
 
  $TotalOutbound = ($OutboundLog | Select-Object SessionId -Unique | Measure-Object).Count
  $OutboundStatistics | Add-Member -Type NoteProperty -Name "TotalOutbound" -Value $TotalOutbound
 
  # Top 10 Senders
 
  Write-Progress -Activity "Generating Statistics for Outbound Mail" -Status "Top 10 Senders"
 
  $SenderCount = @()
  $UniqueSenders = $OutboundLog | Select-Object Sender -Unique
  ForEach ($Sender in $UniqueSenders) {
    $SenderCount += $Sender | Select-Object Sender, `
      @{n='Count';e={ ($OutboundLog | ?{ $_.Recipient -eq $Sender.Sender } | Measure-Object).Count }}
  }
  $Top10Senders = $SenderCount | Sort-Object -Descending -Property Count | Select-Object -First 10
  $OutboundStatistics | Add-Member -Type NoteProperty -Name "Top10Senders" -Value $Top10Senders
 
  Write-Progress -Activity "Generating Statistics for Outbound Mail" -Status "Complete" -Completed
 
  Return $OutboundStatistics
}
 
Function Join-Object ($BaseObject, $AdditionalObject) {
  ForEach ($Property in $($AdditionalObject | Get-Member -Type Property, NoteProperty)) {
    $BaseObject | Add-Member -MemberType NoteProperty -Name $Property.Name `
      -Value $AdditionalObject.$($Property.Name) -ErrorAction SilentlyContinue
  }
  Return $BaseObject
}
 
#
# Main Code
#
 
# This portion of the script deals with removing unwanted information from the log files. 
# Once done the files should be small enough to read using standard PowerShell CmdLets.
 
# Copy each of the files down to the local system
 
If (!$UseExistingLogFiles) {
  Get-LogFolders $ExchangeServer | %{ Get-LogFiles $_.Source $_.Destination }
}
 
# Test that some log files exist
 
If (!(Get-ChildItem "SmtpReceive") -And !$OutboundOnly) { Throw("SmtpReceive is empty") }
If (!(Get-ChildItem "SmtpSend") -And !$InboundOnly) { Throw("SmtpSend is empty") }
 
#
# Create and process the SmtpReceive.log file
#
 
If (!$OutboundOnly) {
 
  [Void](New-Item -Name "SmtpReceive.log" -Type File -Force)
  WriteLineMatchingPattern (Get-ChildItem "SmtpReceive")[0].FullName "SmtpReceive.log" "#Fields: " $True $True
 
  $Files = Get-ChildItem "SmtpReceive"; $i = 0
  $Files | %{
    $i++; $Progress = ($i / $Files.Count) * 100
    Write-Progress -Activity "Reading SmtpReceive Logs" -Status "Reading $($_.Name)" -PercentComplete $Progress
    WriteLineMatchingPattern $_.FullName "SmtpReceive.log" "RCPT TO*$DomainName"
  }
  Write-Progress -Activity "Reading SmtpReceive Logs" -Status "Complete" -Completed
 
  If (!$ExcludeAgentLogs) {
 
    # Combine the SmtpReceive.log file with the Agent.log file
 
    [Void](New-Item -Name "AgentLog.log" -Type File -Force)
    WriteLineMatchingPattern (Get-ChildItem "AgentLog")[0].FullName "AgentLog.log" "#Fields: " $True $True
 
    # Session IDs are required from the Receive Log to bring in the Agent Log. The ReceiveLog "should" be small enough at this stage
    # to cope with Import-CSV
 
    Write-Progress -Activity "Getting Unique Session Ids" -Status "Reading SmtpReceive.log"
    $SessionIds = @{}
    Import-CSV "SmtpReceive.log" | Select-Object "Session-Id" -Unique | %{
      $SessionIds.Add($_."session-id", $True) }
    Write-Progress -Activity "Getting Unique Session Ids" -Status "Reading SmtpReceive.log" -Complete
 
    If ($SessionsIds.Count -ne 0) {
 
      # Get the Index of the SessionId field from the AgentLog.log file
 
      $SessionIdIndex = 0; $i = 0
      (Get-Content "AgentLog.log").Split(",") | %{
        If ($_ -eq "SessionId") { $SessionIdIndex = $i } else { $i++ } }
 
      # Pull out all relevant content from the source log files (AgentLog)
 
      $Files = Get-ChildItem "AgentLog"; $i = 0
      $Files | %{
        $i++; $Progress = ($i / $Files.Count) * 100
        Write-Progress -Activity "Reading AgentLog Logs" -Status "Reading $($_.Name)" -PercentComplete $Progress
        WriteLineMatchingHashKey $_.FullName "AgentLog.log" $SessionIdIndex $SessionIds
      }
      Write-Progress -Activity "Reading AgentLog Logs" -Status "Complete" -Completed
 
      $InboundLog = CombineLogs "SmtpReceive.log" "AgentLog.log" $SessionIdIndex
    }
 
  } Else {
 
    # Tidy up the Inbound Log
 
    $InboundLog = Import-CSV "SmtpReceive.log"
    $InboundLog = $InboundLog | Select-Object `
      @{n='TimeStamp';e={ Get-Date($_."date-time") }}, `
      @{n='Sender';e={ ($_.data -Replace "RCPT TO:<|>").ToLower() }}, `
      @{n='SessionId';e={ $_."session-id" }}, `
      @{n='ConnectorId';e={ $_."connector-id" }}, `
      @{n='LocalIP';e={ $_."local-endpoint" -Replace ":.*" }}, `
      @{n='RemoteIP';e={ $_."remote-endpoint" -Replace ":.*" }}
  }
 
  If ($InboundLog -eq $Null) {
    Write-Warning "No matches found in SmtpReceive"
  } Else {
    $InboundLog | Export-CSV "Inbound.csv"
    $Inbound = GenerateInboundStatistics $InboundLog $SessionIds
  }
}
 
#
# Create and process the SmtpSend.log file
#
 
If (!$InboundOnly) {
 
  [Void](New-Item -Name "SmtpSend.log" -Type File -Force)
  WriteLineMatchingPattern (Get-ChildItem "SmtpSend")[0].FullName "SmtpSend.log" "#Fields: " $True $True
 
  $Files = Get-ChildItem "SmtpSend"; $i = 0
  $Files | %{
    $i++; $Progress = ($i / $Files.Count) * 100
    Write-Progress -Activity "Reading SmtpSend Logs" -Status "Reading $($_.Name)" -PercentComplete $Progress
    WriteLineMatchingPattern $_.FullName "SmtpSend.log" "MAIL FROM*$DomainName"
  }
  Write-Progress -Activity "Reading SmtpSend Logs" -Status "Complete" -Completed
 
  $OutboundLog = Import-CSV "SmtpSend.log"
  $OutboundLog = $OutboundLog | Select-Object `
    @{n='TimeStamp';e={ Get-Date($_."date-time") }}, `
    @{n='Sender';e={ ($_.data -Replace "MAIL FROM:<|>.*").ToLower() }}, `
    @{n='SessionId';e={ $_."session-id" }}, `
    @{n='ConnectorId';e={ $_."connector-id" }}, `
    @{n='LocalIP';e={ $_."local-endpoint" -Replace ":.*" }}, `
    @{n='RemoteIP';e={ $_."remote-endpoint" -Replace ":.*" }}, `
    @{n='Size';e={ $_.data -Replace ".*SIZE`=" }}
 
  If ($OutboundLog -eq $Null) {
    Write-Warning "No matches found in SmtpSend"
  } Else {
    $OutboundLog | Export-CSV "Outbound.csv"
    $Outbound = GenerateOutboundStatistics $OutboundLog
  }
}
 
# Return the results
 
If ($Inbound -ne $Null -And $Outbound -ne $Null) {
  Join-Object $Inbound $Outbound
} ElseIf ($Inbound) {
  $Inbound
} ElseIf ($Outbound) {
  $Outbound
}

Select allOpen in new window

Issue: The code you inserted displays then disapears up to line 207 then I can read it. Tried to two different locations, don't know if it is an IE8 Win 7 issue or a problem with the site.

since I cannot read all the code I do not know if it is different though I suspect that it is.
In the original code you sent I did make the changes to reflect the different install location of Exchange and the paths pertaining to the logs.

So not only to clear this up with me (but for anyone else if they should read this post), I should get the new code from you (was going to as that you email it to <Address Removed>) then use powershell to browse to the location (c:\scripts\powershell) and just run the above stated command

$Results = ./ScriptName.ps1 -ExchangeServer "SomeServer" -ExchangeDrive "e" -InboundOnly -ExcludeAgentLogs
$Results

from the command line.

Am I following you correctly?
Richard

Hmmm I can see it all, so I'd have to put it down to browser troubles. Lets see if it works as an attachment. Probably for the best, I needed to fix a bug I managed to insert into the code :)

So, if you save the script in "C:\scripts\Powershell" you can open up the PowerShell Prompt (from the start menu), run "cd \scripts\powershell" in the usual DOS style and run the script with the commands above.

The first command stores the results in $Results. Then you can access those with either $Results (on its own), or:

$Results | Format-List

Or for the two complex fields in there:

$Results | Select-Object -Expand Top10Recipients
$Results | Select-Object -Expand Top10PrivateSMTPClients

I think / hope I have all the bugs in it now.

Chris

PS Want your email address removing from the post above? It's a bit too easy to harvest from this site
LogParser.txt

I will test that out later today. Thanks for the help and clarification. I will post back on Monday success or failure. As of yet I have not received the email -
How does one remove an email address from the posting?

haha  nevermind on that I just noticed that you posted it here not to the email.

try this man:

Get-MessageTrackingLog -server ExcSrv01 -start 1/1/2010 -end 1/31/2010 -resultsize unlimited | group {$_.Sender.substring($_.sender.indexOf("@")+1)} -NoElement | sort count | Export-Csv "c:\top_domains.csv"

it owrks for me to sort out how many email received per domain.