asked on

Copy portion of text in a log file with PS

Hey Experts. I've got the beginnings of a PS script that will copy text between two lines but I need one more variable to be involved. I need the script to find the lines AND yesterday's date, then copy the text between those lines. I can't figure out how to get it to do that second filter or where to specify where the output log file goes. Here is what I have so far so please feel free to modify what's there.

$arr = @()
$path = "c:\tools\logs\backup.log"
$pattern = "(?<=.SCHEDULEREC STATUS BEGIN)\w+?(?=SCHEDULEREC STATUS END.*)"

Get-Content $path | Foreach {if ([Regex]::IsMatch($_, $pattern)) {
           $arr += [Regex]::Match($_, $pattern)
            }
        }
$arr | Foreach {$_.Value}

Open in new window

Thanks Experts!

//If you need additional information or want to suggest a different PS script, I'm game. I'm new to PS and learning a lot of this so help is appreciated\\

samiam41

ASKER

Ok. So I'm finding bits and pieces of code. Help would still be appreciated as I try to make this happen.

[regex]::Match() only returns the first match. Use [regex]::Matches() to capture all matches:

$s = Get-Content 'C:\app.config'
[regex]::Matches($s, '{{([^/)]+)}}') |ForEach-Object { $_.Groups[1].Value }

SOLUTION

J0rtIT

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

samiam41

ASKER

Thanks Jose! How do I get that to match yesterday's date (the second filter)? Here is an example of the log file (this would be all in one log file) I need the script to search through and locate 1) yesterday's date and 2) the two fields:

12/30/2017 20:25:04 Subfile objects reduced by: 0%
12/30/2017 20:25:04 Elapsed processing time: 00:20:19
12/30/2017 20:25:04 --- SCHEDULEREC STATUS END
12/30/2017 20:25:04 --- SCHEDULEREC OBJECT END DAILYINC_8PM 12/30/2017 20:00:00
12/30/2017 20:25:04 Scheduled event 'DAILYINC_8PM' completed successfully.
12/30/2017 20:25:04 Sending results for scheduled event 'DAILYINC_8PM'.
12/30/2017 20:25:04 Results sent to server for scheduled event 'DAILYINC_8PM'.

12/30/2017 20:25:04 ANS1483I Schedule log pruning started.
12/30/2017 20:25:04 ANS1484I Schedule log pruning finished successfully.
12/30/2017 20:25:04 TSM Backup-Archive Client Version 6, Release 4, Level 0.11
12/30/2017 20:25:04 Querying server for next scheduled event.
12/30/2017 20:25:04 Node Name: W2K12FS01
12/30/2017 20:25:04 Session established with server xx: Linux/x86_64
12/30/2017 20:25:04 Server Version 7, Release 1, Level 3.0
12/30/2017 20:25:04 Server date/time: 12/30/2017 20:25:04 Last access: 12/30/2017 20:09:25

12/30/2017 20:25:04 --- SCHEDULEREC QUERY BEGIN
12/30/2017 20:25:04 --- SCHEDULEREC QUERY END
12/30/2017 20:25:04 Next operation scheduled:
12/30/2017 20:25:04 ------------------------------------------------------------
12/30/2017 20:25:04 Schedule Name: DAILYINC_8PM
12/30/2017 20:25:04 Action: Incremental
12/30/2017 20:25:04 Objects:
12/30/2017 20:25:04 Options:
12/30/2017 20:25:04 Server Window Start: 20:00:00 on 12/31/2017
12/30/2017 20:25:04 ------------------------------------------------------------
12/30/2017 20:25:04 Schedule will be refreshed in 12 hours.
12/31/2017 08:25:04 TSM Backup-Archive Client Version 6, Release 4, Level 0.11
12/31/2017 08:25:04 Querying server for next scheduled event.
12/31/2017 08:25:04 Node Name: W2K12FS01
12/31/2017 08:25:05 Session established with server xx: Linux/x86_64
12/31/2017 08:25:05 Server Version 7, Release 1, Level 3.0
12/31/2017 08:25:05 Server date/time: 12/31/2017 08:25:04 Last access: 12/30/2017 20:25:04

12/31/2017 08:25:05 --- SCHEDULEREC QUERY BEGIN
12/31/2017 08:25:05 --- SCHEDULEREC QUERY END
12/31/2017 08:25:05 Next operation scheduled:
12/31/2017 08:25:05 ------------------------------------------------------------
12/31/2017 08:25:05 Schedule Name: DAILYINC_8PM
12/31/2017 08:25:05 Action: Incremental
12/31/2017 08:25:05 Objects:
12/31/2017 08:25:05 Options:
12/31/2017 08:25:05 Server Window Start: 20:00:00 on 12/31/2017
12/31/2017 08:25:05 ------------------------------------------------------------
12/31/2017 08:25:05 Command will be executed in 11 hours and 36 minutes.
12/31/2017 20:01:05
Executing scheduled command now.
12/31/2017 20:01:06 Node Name: W2K12FS01
12/31/2017 20:01:06 Session established with server xx: Linux/x86_64
12/31/2017 20:01:06 Server Version 7, Release 1, Level 3.0
12/31/2017 20:01:06 Server date/time: 12/31/2017 20:01:06 Last access: 12/31/2017 08:25:05

12/31/2017 20:01:06 --- SCHEDULEREC OBJECT BEGIN DAILYINC_8PM 12/31/2017 20:00:00
12/31/2017 20:01:06 Incremental backup of volume '\\w2k12fs01\e$'
12/31/2017 20:03:28 ANS1898I ***** Processed 2,000 files *****
12/31/2017 20:03:29 ANS1898I ***** Processed 7,500 files *****
12/31/2017 20:03:30 ANS1898I ***** Processed 12,000 files *****

01/04/2018 23:43:40 --- SCHEDULEREC STATUS BEGIN
01/04/2018 23:43:40 Total number of objects inspected: 1,130,802
01/04/2018 23:43:40 Total number of objects backed up: 1,112
01/04/2018 23:43:40 Total number of objects updated: 0
01/04/2018 23:43:40 Total number of objects rebound: 0
01/04/2018 23:43:40 Total number of objects deleted: 0
01/04/2018 23:43:40 Total number of objects expired: 788
01/04/2018 23:43:40 Total number of objects failed: 0
01/04/2018 23:43:40 Total number of subfile objects: 0
01/04/2018 23:43:40 Total number of bytes inspected: 805.53 GB
01/04/2018 23:43:40 Total number of bytes transferred: 35.60 GB
01/04/2018 23:43:40 Data transfer time: 11,967.81 sec
01/04/2018 23:43:40 Network data transfer rate: 3,119.66 KB/sec
01/04/2018 23:43:40 Aggregate data transfer rate: 2,809.67 KB/sec
01/04/2018 23:43:40 Objects compressed by: 9%
01/04/2018 23:43:40 Total data reduction ratio: 95.58%
01/04/2018 23:43:40 Subfile objects reduced by: 0%
01/04/2018 23:43:40 Elapsed processing time: 03:41:28
01/04/2018 23:43:40 --- SCHEDULEREC STATUS END

So this script would have run in the morning of Jan 5th and would have captured the data I put in bold above (for ease of identifying it). How would this script accomplish this? Some sort of piping?

Qlemo

Bad example, as the snippet only contains one Begin/End pair ;-). BTW, it is better to post such into a CODE block instead of a quote.
If you use the following pattern, you'll filter for the past day:

$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"

Open in new window

You'll have to use $TheMatch.Groups[1].Value to get the result. Sadly, that will not preserve line breaks - they are gone.

samiam41

ASKER

Thanks Qlemo. Yeah that probably wasn't the best example. :) And I wondered if I should have put that into a code block or if it was more for just code. I'll use that going forward.

For the code portion, based on your other recommendation, something like this?

$arr = @()
$path = "c:\tools\logs\backup.log"
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"

Get-Content $path | Foreach {if ([Regex]::IsMatch($_, $pattern)) {
           $arr += [Regex]::Match($_, $pattern)
            }
        }
$arr | Foreach {$_.Value}

Open in new window

I've got a limited understanding of PS so I'm not sure what you mean by this or where it would go: $TheMatch.Groups[1].Value

Thanks for the continued help!

Qlemo

Based on Jose's code:

#The path is: 
$path = "c:\tools\logs\backup.log"
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
[System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches((Get-Content $path),$pattern)
if($TheMatch.Count) {
    Write-host "Match found" -foreground green
    write-host $($TheMatch.groups[1].value) -ForegroundColor Cyan
}
else{
     Write-host "No match were found" -foreground red
}

Open in new window

samiam41

ASKER

When I run it from PS ISE, it returns right to the prompt with nothing shown besides the coding. How do I get the output?

samiam41

ASKER

When I run this script

$script = Get-Content C:\tools\logs\dsmsched.log

$in = $false

$script | %{
    if ($_.Contains("SCHEDULEREC STATUS BEGIN"))
        { $in = $true }
    elseif ($_.Contains("SCHEDULEREC STATUS END"))
        { $in = $false; }
    elseif ($in)
        { Write-Host $_ } # Or Out-File ...
}

Open in new window

It outputs

Is there anyway to loop this output to only match yesterday's date as a last step?

Qlemo

You can work with a regex or wildcard mask here too:

$in = $false
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' *  --- SCHEDULEREC STATUS BEGIN*'
$(
  switch -wildcard -file C:\tools\logs\dsmsched.log
  {
    $pattern   { $in = $true }
    '*SCHEDULEREC STATUS END*'  { $in = $false }
    default { $_ }
  }
)

Open in new window

samiam41

ASKER

Displays everything in the log file regardless of filters.

J0rtIT

I'd use my function twice.
one between dates (today and yesterday)
And another between the "start and end", just don't have time to code it right now.

Qlemo

Of course I should check $in before writing out lines ...

$in = $false
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' *  --- SCHEDULEREC STATUS BEGIN*'
$(
  switch -wildcard -file C:\tools\logs\dsmsched.log
  {
    $pattern   { $in = $true }
    '*SCHEDULEREC STATUS END*'  { $in = $false }
    default { $_ | ? { $in } }
  }
)

Open in new window

samiam41

ASKER

:)

Now nothing shows

J0rtIT

#The path is: 
$path = ".\ex.log"
[string]$Txtlines = Get-Content $path
#Filter1 Dates:

$YestObj = (Get-Date).AddDays(-1)
$yeste = (($YestObj -f "MM/dd/yyyy").Split(' ')[0]).replace('/','\/')

#This will match yesterday's date (complete Lines).
$DateRegex = ".*($yeste).*"
[System.Text.RegularExpressions.MatchCollection] $TheMatches = [System.Text.RegularExpressions.Regex]::Matches($Txtlines,$DateRegex)

if($TheMatches.Count -gt 0){
    #Create Filtered string.
    $FilteredString= new-object System.Text.StringBuilder 
#this line will fill out the filteredstring (stringbuilder obj).
    $TheMatches | %{$filteredstring.AppendLine($_.Value) | Out-Null}

    $pattern = "SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
    [System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches($FilteredString.ToString(),$pattern)

    if($TheMatch.Count -gt 0){
        Write-host "Match found" -foreground green
        write-host $($TheMatch.value) -ForegroundColor Cyan
    }
    else{
        Write-host "No match were found" -foreground red
    }
}
else{
    Write-host "No date found that starts  with $(($YestObj -f "MM/dd/yyyy").Split(' ')[0])" -foreground red
}

Open in new window

samiam41

ASKER

Jose, ran your script against the newest log file which contains data from Jan 2nd-8th. It pulls all the data between those dates under those headers.

samiam41

ASKER

Jose, went back to this script with a couple of minor tweaks to output to a log file and it's running/working although the formatting is lost but I'm guessing that is to be expected.

#The path is: 
$path = "C:\tools\logs\dsmsched.log"
$log = "c:\tools\logs\combine.log"
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
[System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches((Get-Content $path),$pattern)
if($TheMatch.Count) {
    Write-host "Match found" > $log
    write-host $($TheMatch.groups[1].value > $log)
}
else{
     Write-host "No match were found" >$log
}

Open in new window

SOLUTION

J0rtIT

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER CERTIFIED SOLUTION

Qlemo

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

samiam41

ASKER

Qlemo, that worked! What was the typo??

Qlemo

$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' * --- SCHEDULEREC STATUS BEGIN*'
vs.
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' * --- SCHEDULEREC STATUS BEGIN*'
which has been one space too many pre '---' ...

samiam41

ASKER

Thank you both for working tirelessly on your suggested script. Ultimately the having a script that output the text that retained its formatting was the one I went with as it will make it easier to work with as the script under goes the next evolutions. Regardless, your time and patience was much appreciated and I look forward to working with you both in the near future.

J0rtIT

Np glad to help either way I always enjoy testing the Qlemo's and Obda's ps codes :) I've learned a few things from them.