Powershell - script to tail log file on multiple servers

It is a little bit more complex than that. You need to remote execute and/or access the log files via share.
I would not try to use parallel execution, because that needs some setup (to get PowerShell jobs running, you need to enable WSMan Remoting on the excuting machine).

So the simple setup is to indeed use loops to go thru all targets in a cycle.

$serviceB = "my application service"
$logfile2 = "F$\App\log\wrapper.log"
$waitFor = "[4.2.3.GA (build: SVNTag=JBoss_4_2]”
$hosts = 'server1', 'server2', 'server3'   # and so on

foreach ($host in $hosts) {
  Write-Host "Starting $serviceB on $host"
  Start-Service $serviceB -ComputerName $host
}
$hostsleft = $hosts
do {
  foreach ($host in $hostsleft) {
    Write-Host "Waiting for $waitFor in \\$host\$logfile2" -NoNewLine
    if (Get-Content $logfile2 -Tail 30 | ? {$_ -match $waitFor}) {
      write-Host " - found" -NoNewLine
      $hostsleft = $hostsleft | ? { $_ -notmatch $host }
    }
    Write-Host
    Start-Sleep 5
} while ($hostsleft)

Select allOpen in new window

This should perform reaonsably, as it checks the remote machines in a pseudo-parallel way.

do you have a monitoring system
, sitescope E Health sysedge Etc.
Nagios might be an option.

In short you gave a conditional dependent requirement, that can trigger events on systems...

To have multiple inputs to then react in unison would require that they all have access to a central/single point where each can check whether the condition has been met.
is there a timeframe after which if the event is not seen, the counter resets? I.e. The pattern on all five has to be seen within 10 minutes ...

Thank you Qlemo. Indeed, I believe remote PoSh capabilities are enabled on all the systems. A few questions to clarify.

1. So, the script above..it will run against one system at a time, correct?

2.  Will the script terminate on its own or run in a continues loop?

arnold: we have solarwinds. however, I can spin up just about any solution if need be. I've stood up and implemented Nagios before.

I believe there is no time out values. All 5 systems can take as long as they need to, to start the service. I've been also thinking about running 5 different instances of posh.

My code manages all remotes at nearly the same time, and terminate as soon as all have reported success. The progress output should show you details about which hosts did still not "finish" in each loop go.

Thank you Qlemo.

You've answered my question.

I am only trying to use tail because its what Ido when cranking up the application services. With that said, is this the most efficient way for me to know that the log entry is in the log? Does the tail aspect of this script "make it slower" than just "telling me" when the log entry exists?

Can I use some other logic like simply, look for string: [4.2.3.GA (build: SVNTag=JBoss_4_2]    ?

Using tail is fine, However, would be cool if we can have an alternative leaner/quicker/lighter script.

Tail tries to get only the last lines. Depending on how much is in the log, it might be unnecessary to use tail. But it shouldn't make any difference if the files are small, and the tail lines are enough to catch the search pattern.

You're not quite done yet, I'm afraid. There is one major issue left with this script, and that is the text search.
The pattern used will happily match pretty much anything, including this:

PS C:\> 'Serious error, your computer will self destruct in 3 seconds.' -match '[4.2.3.GA (build: SVNTag=JBoss_4_2]'
True
PS C:\>

Select allOpen in new window

Why?
Because you're using -match with a pattern of "[4.2.3.GA (build: SVNTag=JBoss_4_2]".
-match compares against a Regular Expression. In a regex, a group of characters enclosed in square brackets means "match any of the characters inside the square brackets".
The string "error", for example, will match against the "o" in "JBoss", a "3" or "." against the "4.2.3", and so on.
So either you "escape" the text you're searching for:

$waitFor = [regex]::Escape('[4.2.3.GA (build: SVNTag=JBoss_4_2]')

Select allOpen in new window

Or, since there's no real need for a regex here, you can just use the -like operator instead of -match (assuming the string is enclosed in surrounding characters, otherwise a simple -eq will do):

if (Get-Content $logfile2 -Tail 30 | ? {$_ -like "*$($waitFor)*"}) {

Select allOpen in new window

I think that looking at ONLY the last 30 lines for a given text string is... hmmm. Ridiculous.

Guaranteed to fail. Eventually.

If it's (about) the last line in the log after a successful service restart, it's totally legitimate.
Might even be necessary if the log appends to former logs, in order to not capture a successful earlier service restart.

Correct, we cannot decide if tail os required, superfluous or even wrong without knowing details about how the log file is populated ...
And of course we need to use -like instead of -match here.

Thanks for all the input folks. The new version of the application is supposed to generate the file so it will always be a new one. However, at the moment the application log entries are appending to the existing log file which is why I am leaning towards tail. Also, the logs files themselves don't get bigger than a few MB's.


Moreover, if it helps using a string, I can always simplify the search pattern by removing the brackets. Also, The reason I am only tailing the last 30 lines is to avoid looking at the entire line. I shouldn't have any issues there - I think.

oBdA, should I add your if statement to the foreach loop?

Just removing the square brackets won't help, because then you'll have an invalid regular expression.
-match is not the best operator here; just use -like and enclose the searchstring in "*"
So instead of
if (Get-Content $logfile2 -Tail 30 | ? {$_ -match $waitFor}) {"
use
if (Get-Content $logfile2 -Tail 30 | ? {$_ -like "*$($waitFor)*"}) {

The tail will only help if is guaranteed that there are at least 30 lines between the previous entry and the time the service is started up.

Sorry for the late response Qlemo. So Im trying to further use the script in your first reply where all the script is executed in a psuedo-parallel way.

However, the scripts seems to finish and not move on to the next server. Perhaps, I am executing it wrong? Am I supposed to use the invoke command to run it on "each server" rather than just execute the code from my domain PC?

I ran the script from my domain joined PC and it seems to be doing this:

PS C:\Scripts> C:\Scripts\roll-2.1.startUIsvcs.ps1
Starting bits on servername1
Starting bits on servername1
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\servername1\E$\all\log\wrapper.log - found
PS C:\Scripts> 
PS C:\Scripts> 

Select allOpen in new window

I am using 'bits' to test the script. It seems like its starting only the first server and not trying to move on to the others. I only have 2 server in the list for now. I will add 5 as soon as I can validate with 2 servers.  Any idea why it's only running the script on the first server in the list? I've tried both get-content from a txt as well as strings as shown in your script.

The only explanation I have 8s that you put the same server twice into the hosts list. Tue first loop is executed twice (with that same server), the second one only once because same-name entries in the hosts list get removed together.

Right on!

12 + hour day. Thanks for helping me purge the brain fog. Indeed I had the same hostname listed twice ::doh::

In any case, the script spits out the same result. It simply finishes.

Also, when I cp your code to the ISE it was complaining about a few things. One was the $host variable. So I change the name to $remotehost. Then it was complaining about missing brackets so I added brackets. However, I am not sure they are in the right place. Could this be the reason the script doesn't run "pseudo-parallel"?

$serviceB = "bits"
$logfile2 = "E$\log\wrapper.log"
$waitFor = "[4.2.3.GA (build: SVNTag=JBoss_4_2]”
#$remotehosts = 'server1','server2'   # and so on

foreach ($remotehost in $remotehosts) {
  Write-Host "Starting $serviceB on $remotehost"
  Start-Service $serviceB  #-ComputerName $remotehost
}


$remotehostsleft = $remotehost
do {
  foreach ($remotehost in $remotehostsleft) {
    Write-Host "Waiting for $waitFor in \\$remotehost\$logfile2" -NoNewLine
    if (Get-Content \\$remotehost\$logfile2 -Tail 30 | ? {$_ -match $waitFor}) {
      write-Host " - found" -NoNewLine
      $remotehostsleft = $remotehostsleft | ? { $_ -notmatch $remotehost } 
    }
    }
    }
    while ($remotehostsleft) 

Select allOpen in new window

Here is a new script

• incorporating the use of -like instead of -match,
• removing the override of the predefined  $host variable
• including the full path in the tail part (I missed to add the server name)
• added to ignore errors when trying to access the log file
• added the missing closing brace

This should work now.
Regarding your tests with bits, you won't be able to find anything in the logfiles if there aren't any, and if they exist, they won't change ;-).

$serviceB = "my application service"
$logfile2 = "F$\App\log\wrapper.log"
$waitFor = "[4.2.3.GA (build: SVNTag=JBoss_4_2]”
$hosts = 'server1', 'server2', 'server3'   # and so on

foreach ($actHost in $hosts) {
  Write-Host "Starting $serviceB on $actHost"
  Start-Service $serviceB -ComputerName $actHost
}
$hostsleft = $hosts
do {
  foreach ($actHost in $hostsleft) {
    Write-Host "Waiting for $waitFor in \\$actHost\$logfile2" -NoNewLine
    if (Get-Content "\\$actHost\$logfile2" -Tail 30 -ea SilentlyContinue | ? {$_ -like "*$($waitFor)*"}) {
      write-Host " - found" -NoNewLine
      $hostsleft = $hostsleft | ? { $_ -notmatch $host }
    }
    Write-Host
  }
  Start-Sleep 5
} while ($hostsleft)

Select allOpen in new window

Thanks Qlemo!

Sounds like the -computername parameter seems to be crying so removed it. Also, I have the wrapper logs in the same location on both servers. However, on 1 I have the "string"  in the log file while on 2 I do not have the entry - on purpose to test. I generally wait for the tail and then open the log file, manually add the entry and save it. If I don't ctrl-c the script it will go on generating the same "waiting for line"

Here is the output:

PS C:\Scripts> C:\Scripts\SDS-roll-2.2.startUIsvcs.ps1
Starting bits on Server1
Start-Service : A parameter cannot be found that matches parameter name 'ComputerName'.
At C:\Scripts\roll-2.2.startUIsvcs.ps1:8 char:27
+   Start-Service $serviceB -ComputerName $actHost
+                           ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Start-Service], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.StartServiceCommand
 
Starting bits on Server2
Start-Service : A parameter cannot be found that matches parameter name 'ComputerName'.
At C:\Scripts\roll-2.2.startUIsvcs.ps1:8 char:27
+   Start-Service $serviceB -ComputerName $actHost
+                           ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Start-Service], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.StartServiceCommand
 
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\Server1\E$\log\wrapper.log - found
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\Server2\E$\log\wrapper.log - found
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\server1\E$\log\wrapper.log - found
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\Server2\E$\log\wrapper.log - found
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\server1\E$\log\wrapper.log - found
Waiting for [4.2.3.GA (build: SVNTag=JBoss_4_2] in \\Server2\E$\log\wrapper.log - found

Select allOpen in new window

Thank for the continuing advise and support Qlemo!

Here is the new output. It seems to be crying about a missed closing. Can you please advise when you spare a moment?

Darn paste errors ...
Lines 6 and 7 need to be

Write-Host "Starting $serviceB on $($hosts -join ', ')"
Get-Service $serviceB -ComputerName $hosts -ea SilentlyContinue | Start-Service 

Select allOpen in new window

Thanks! The message is now gone. However, for some reason, the Tail doesn't seem to be picking up the $waitfor variable. It's almost as it is being ignored. However, when I modify the string and remove some of the brackets to leave it like so: "4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)]"  

..it works. Perhaps this is due to one of the methodologies regarding the regex mentioned earlier by oBdA. I might be able to use it like that. Thanks again for the big help. I will now test with multiple servers at the same time.

Using -like "*${waitFor}*" might work better without having to change the search pattern.

Thank you all for your help. Especially Qlemo. I have successfully tested the script. This will save us downtime and highly minimize human error.  You all will be honored in my department!

Hello Qlemo,

I was hoping to ask one last thing from this script. I'm willing to start another thread since this is a wish list item. However, would it be possible to append the "found" hosts to the title bar?

For example, every time the string is found on the remote host, append the host name to the console title so that we can easily see which hosts have been found.

Although it already displays found and the host name in the command output, it is moving quick so it's hard to reference a row while it's scrolling.

Thanks!

I really would do that differently: Only output found hosts, and kind of a progess info on a single line.

$serviceB = "my application service"
$logfile2 = "F$\App\log\wrapper.log"
$waitFor = "[4.2.3.GA (build: SVNTag=JBoss_4_2]”
$hosts = 'server1', 'server2', 'server3'   # and so on

Write-Host "Starting $serviceB on $($hosts -join ', '"
Get-Service $serviceB -ComputerName $osts -ea SilentlyContinue | Start-Service 

$hostsleft = $hosts
do {
  foreach ($actHost in $hostsleft) {
    Write-Host "`rWaiting for $actHost                   " -NoNewLine
    if (Get-Content "\\$actHost\$logfile2" -Tail 30 -ea SilentlyContinue | ? {$_ -like "*${waitFor}*"}) {
      write-Host " - found"
      $hostsleft = $hostsleft | ? { $_ -ne $actHost }
    }
  }
  Start-Sleep 5
} while ($hostsleft)

Select allOpen in new window

The last line always shows some "changing" info just to be sure the script really does something, and starts a new line whenever it found the search pattern on another server.