Copy portion of text in a log file with PS

Hey Experts.  I've got the beginnings of a PS script that will copy text between two lines but I need one more variable to be involved.  I need the script to find the lines AND yesterday's date, then copy the text between those lines.  I can't figure out how to get it to do that second filter or where to specify where the output log file goes.  Here is what I have so far so please feel free to modify what's there.

$arr = @()
$path = "c:\tools\logs\backup.log"
$pattern = "(?<=.SCHEDULEREC STATUS BEGIN)\w+?(?=SCHEDULEREC STATUS END.*)"

Get-Content $path | Foreach {if ([Regex]::IsMatch($_, $pattern)) {
           $arr += [Regex]::Match($_, $pattern)
            }
        }
$arr | Foreach {$_.Value}

Open in new window


Thanks Experts!

//If you need additional information or want to suggest a different PS script, I'm game.  I'm new to PS and learning a lot of this so help is appreciated\\
LVL 9
samiam41Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

samiam41Author Commented:
Ok.  So I'm finding bits and pieces of code.  Help would still be appreciated as I try to make this happen.

[regex]::Match() only returns the first match. Use [regex]::Matches() to capture all matches:

$s = Get-Content 'C:\app.config'
[regex]::Matches($s, '{{([^/)]+)}}') |ForEach-Object { $_.Groups[1].Value }
0
Jose Gabriel Ortega CastroCEOCommented:
Hi Sam, The issue with your Regex is that it has the bad focus.
you need all the information between 2 strings.
That's the correct focus.

#The path is: 
$path = "c:\tools\logs\backup.log"
$Txtlines = Get-Content $path
$pattern = "SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
[System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches($Txtlines,$pattern)
if($TheMatch.Count -gt 0){
    Write-host "Match found" -foreground green
    write-host $($TheMatch.value) -ForegroundColor Cyan
}
else{
     Write-host "No match were found" -foreground red
}

Open in new window

Using "text.txt" is the ps1 file, but EE doesn't allow to upload it as ps1, so you must change the extension to ps1.
Using the dummy text file  dummy.txt
The result of the code is:
Screenshot_3.png
====
Another approach is to create a function like this:
function GetStringBetweenTwoStrings($firstString, $secondString, $importPath){

    #Get content from file
    $file = Get-Content $importPath

    #Regex pattern to compare two strings
    $pattern = "$firstString(.*?)$secondString"

    #Perform the opperation
    $result = [regex]::Match($file,$pattern).Groups[1].Value

    #Return result
    return $result

}

Open in new window

and use it like this:
GetStringBetweenTwoStrings -firstString "Lorem" -secondString "is" -importPath "C:\Temp\test.txt"

Open in new window


Source is in here:
https://stackoverflow.com/questions/36746272/powershell-extract-text-between-two-strings
0
samiam41Author Commented:
Thanks Jose!  How do I get that to match yesterday's date (the second filter)?  Here is an example of the log file (this would be all in one log file) I need the script to search through and locate 1) yesterday's date and 2) the two fields:


12/30/2017 20:25:04 Subfile objects reduced by:                   0%
12/30/2017 20:25:04 Elapsed processing time:               00:20:19
12/30/2017 20:25:04 --- SCHEDULEREC STATUS END
12/30/2017 20:25:04 --- SCHEDULEREC OBJECT END DAILYINC_8PM 12/30/2017 20:00:00
12/30/2017 20:25:04 Scheduled event 'DAILYINC_8PM' completed successfully.
12/30/2017 20:25:04 Sending results for scheduled event 'DAILYINC_8PM'.
12/30/2017 20:25:04 Results sent to server for scheduled event 'DAILYINC_8PM'.

12/30/2017 20:25:04 ANS1483I Schedule log pruning started.
12/30/2017 20:25:04 ANS1484I Schedule log pruning finished successfully.
12/30/2017 20:25:04 TSM Backup-Archive Client Version 6, Release 4, Level 0.11  
12/30/2017 20:25:04 Querying server for next scheduled event.
12/30/2017 20:25:04 Node Name: W2K12FS01
12/30/2017 20:25:04 Session established with server xx: Linux/x86_64
12/30/2017 20:25:04   Server Version 7, Release 1, Level 3.0
12/30/2017 20:25:04   Server date/time: 12/30/2017 20:25:04  Last access: 12/30/2017 20:09:25

12/30/2017 20:25:04 --- SCHEDULEREC QUERY BEGIN
12/30/2017 20:25:04 --- SCHEDULEREC QUERY END
12/30/2017 20:25:04 Next operation scheduled:
12/30/2017 20:25:04 ------------------------------------------------------------
12/30/2017 20:25:04 Schedule Name:         DAILYINC_8PM
12/30/2017 20:25:04 Action:                Incremental
12/30/2017 20:25:04 Objects:              
12/30/2017 20:25:04 Options:              
12/30/2017 20:25:04 Server Window Start:   20:00:00 on 12/31/2017
12/30/2017 20:25:04 ------------------------------------------------------------
12/30/2017 20:25:04 Schedule will be refreshed in 12 hours.
12/31/2017 08:25:04 TSM Backup-Archive Client Version 6, Release 4, Level 0.11  
12/31/2017 08:25:04 Querying server for next scheduled event.
12/31/2017 08:25:04 Node Name: W2K12FS01
12/31/2017 08:25:05 Session established with server xx: Linux/x86_64
12/31/2017 08:25:05   Server Version 7, Release 1, Level 3.0
12/31/2017 08:25:05   Server date/time: 12/31/2017 08:25:04  Last access: 12/30/2017 20:25:04

12/31/2017 08:25:05 --- SCHEDULEREC QUERY BEGIN
12/31/2017 08:25:05 --- SCHEDULEREC QUERY END
12/31/2017 08:25:05 Next operation scheduled:
12/31/2017 08:25:05 ------------------------------------------------------------
12/31/2017 08:25:05 Schedule Name:         DAILYINC_8PM
12/31/2017 08:25:05 Action:                Incremental
12/31/2017 08:25:05 Objects:              
12/31/2017 08:25:05 Options:              
12/31/2017 08:25:05 Server Window Start:   20:00:00 on 12/31/2017
12/31/2017 08:25:05 ------------------------------------------------------------
12/31/2017 08:25:05 Command will be executed in 11 hours and 36 minutes.
12/31/2017 20:01:05
Executing scheduled command now.
12/31/2017 20:01:06 Node Name: W2K12FS01
12/31/2017 20:01:06 Session established with server xx: Linux/x86_64
12/31/2017 20:01:06   Server Version 7, Release 1, Level 3.0
12/31/2017 20:01:06   Server date/time: 12/31/2017 20:01:06  Last access: 12/31/2017 08:25:05

12/31/2017 20:01:06 --- SCHEDULEREC OBJECT BEGIN DAILYINC_8PM 12/31/2017 20:00:00
12/31/2017 20:01:06 Incremental backup of volume '\\w2k12fs01\e$'
12/31/2017 20:03:28 ANS1898I ***** Processed     2,000 files *****
12/31/2017 20:03:29 ANS1898I ***** Processed     7,500 files *****
12/31/2017 20:03:30 ANS1898I ***** Processed    12,000 files *****

01/04/2018 23:43:40 --- SCHEDULEREC STATUS BEGIN
01/04/2018 23:43:40 Total number of objects inspected:    1,130,802
01/04/2018 23:43:40 Total number of objects backed up:        1,112
01/04/2018 23:43:40 Total number of objects updated:              0
01/04/2018 23:43:40 Total number of objects rebound:              0
01/04/2018 23:43:40 Total number of objects deleted:              0
01/04/2018 23:43:40 Total number of objects expired:            788
01/04/2018 23:43:40 Total number of objects failed:               0
01/04/2018 23:43:40 Total number of subfile objects:              0
01/04/2018 23:43:40 Total number of bytes inspected:         805.53 GB
01/04/2018 23:43:40 Total number of bytes transferred:        35.60 GB
01/04/2018 23:43:40 Data transfer time:                   11,967.81 sec
01/04/2018 23:43:40 Network data transfer rate:            3,119.66 KB/sec
01/04/2018 23:43:40 Aggregate data transfer rate:          2,809.67 KB/sec
01/04/2018 23:43:40 Objects compressed by:                        9%
01/04/2018 23:43:40 Total data reduction ratio:               95.58%
01/04/2018 23:43:40 Subfile objects reduced by:                   0%
01/04/2018 23:43:40 Elapsed processing time:               03:41:28
01/04/2018 23:43:40 --- SCHEDULEREC STATUS END

So this script would have run in the morning of Jan 5th and would have captured the data I put in bold above (for ease of identifying it).  How would this script accomplish this?  Some sort of piping?
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Bad example, as the snippet only contains one Begin/End pair ;-). BTW, it is better to post such into a CODE block instead of a quote.
If you use the following pattern, you'll filter for the past day:
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"

Open in new window

You'll have to use $TheMatch.Groups[1].Value to get the result. Sadly, that will not preserve line breaks - they are gone.
1
samiam41Author Commented:
Thanks Qlemo.  Yeah that probably wasn't the best example.  :) And I wondered if I should have put that into a code block or if it was more for just code.  I'll use that going forward.

For the code portion, based on your other recommendation, something like this?

$arr = @()
$path = "c:\tools\logs\backup.log"
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"

Get-Content $path | Foreach {if ([Regex]::IsMatch($_, $pattern)) {
           $arr += [Regex]::Match($_, $pattern)
            }
        }
$arr | Foreach {$_.Value}

Open in new window


I've got a limited understanding of PS so I'm not sure what you mean by this or where it would go:  $TheMatch.Groups[1].Value

Thanks for the continued help!
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Based on Jose's code:
#The path is: 
$path = "c:\tools\logs\backup.log"
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
[System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches((Get-Content $path),$pattern)
if($TheMatch.Count) {
    Write-host "Match found" -foreground green
    write-host $($TheMatch.groups[1].value) -ForegroundColor Cyan
}
else{
     Write-host "No match were found" -foreground red
}

Open in new window

0
samiam41Author Commented:
When I run it from PS ISE, it returns right to the prompt with nothing shown besides the coding.  How do I get the output?

script output
0
samiam41Author Commented:
When I run this script
$script = Get-Content C:\tools\logs\dsmsched.log

$in = $false

$script | %{
    if ($_.Contains("SCHEDULEREC STATUS BEGIN"))
        { $in = $true }
    elseif ($_.Contains("SCHEDULEREC STATUS END"))
        { $in = $false; }
    elseif ($in)
        { Write-Host $_ } # Or Out-File ...
}

Open in new window

It outputs
output
Is there anyway to loop this output to only match yesterday's date as a last step?
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
You can work with a regex or wildcard mask here too:
$in = $false
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' *  --- SCHEDULEREC STATUS BEGIN*'
$(
  switch -wildcard -file C:\tools\logs\dsmsched.log
  {
    $pattern   { $in = $true }
    '*SCHEDULEREC STATUS END*'  { $in = $false }
    default { $_ }
  }
)

Open in new window

0
samiam41Author Commented:
Displays everything in the log file regardless of filters.
0
Jose Gabriel Ortega CastroCEOCommented:
I'd use my function twice.
one between dates (today and yesterday)
And another between the "start and end", just don't have time to code it right now.
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Of course I should check $in before writing out lines ...
$in = $false
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' *  --- SCHEDULEREC STATUS BEGIN*'
$(
  switch -wildcard -file C:\tools\logs\dsmsched.log
  {
    $pattern   { $in = $true }
    '*SCHEDULEREC STATUS END*'  { $in = $false }
    default { $_ | ? { $in } }
  }
)

Open in new window

0
samiam41Author Commented:
:)

Now nothing shows

no_output.png
0
Jose Gabriel Ortega CastroCEOCommented:
#The path is: 
$path = ".\ex.log"
[string]$Txtlines = Get-Content $path
#Filter1 Dates:

$YestObj = (Get-Date).AddDays(-1)
$yeste = (($YestObj -f "MM/dd/yyyy").Split(' ')[0]).replace('/','\/')

#This will match yesterday's date (complete Lines).
$DateRegex = ".*($yeste).*"
[System.Text.RegularExpressions.MatchCollection] $TheMatches = [System.Text.RegularExpressions.Regex]::Matches($Txtlines,$DateRegex)

if($TheMatches.Count -gt 0){
    #Create Filtered string.
    $FilteredString= new-object System.Text.StringBuilder 
#this line will fill out the filteredstring (stringbuilder obj).
    $TheMatches | %{$filteredstring.AppendLine($_.Value) | Out-Null}

    $pattern = "SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
    [System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches($FilteredString.ToString(),$pattern)

    if($TheMatch.Count -gt 0){
        Write-host "Match found" -foreground green
        write-host $($TheMatch.value) -ForegroundColor Cyan
    }
    else{
        Write-host "No match were found" -foreground red
    }
}
else{
    Write-host "No date found that starts  with $(($YestObj -f "MM/dd/yyyy").Split(' ')[0])" -foreground red
}

Open in new window


t.png
0
samiam41Author Commented:
Jose, ran your script against the newest log file which contains data from Jan 2nd-8th.  It pulls all the data between those dates under those headers.

Jose_EE_Results.png
0
samiam41Author Commented:
Jose, went back to this script with a couple of minor tweaks to output to a log file and it's running/working although the formatting is lost but I'm guessing that is to be expected.

#The path is: 
$path = "C:\tools\logs\dsmsched.log"
$log = "c:\tools\logs\combine.log"
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + " .{8} --- SCHEDULEREC STATUS BEGIN(.*?)SCHEDULEREC STATUS END"
[System.Text.RegularExpressions.MatchCollection] $TheMatch = [System.Text.RegularExpressions.Regex]::Matches((Get-Content $path),$pattern)
if($TheMatch.Count) {
    Write-host "Match found" > $log
    write-host $($TheMatch.groups[1].value > $log)
}
else{
     Write-host "No match were found" >$log
}

Open in new window

0
Jose Gabriel Ortega CastroCEOCommented:
Yeah the formating is being lost on this line:

  $TheMatches | %{$filteredstring.AppendLine($_.Value) | Out-Null}

Open in new window

It's supposed to add the Linebreak, but when it passes the second regex is superlost lol.

if you want to preserve the format you can do a Regex Line by line.
something like
$thematches| %{
  #here the logic line by line.
}

Open in new window

1
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
I had a small typo in the pattern, sigh.
$in = $false
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' * --- SCHEDULEREC STATUS BEGIN*'
$(
  switch -wildcard -file C:\tools\logs\dsmsched.log
  {
    $pattern   { $in = $true }
    '*SCHEDULEREC STATUS END*'  { $in = $false }
    default { $_ | ? { $in } }
  }
)

Open in new window

1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
samiam41Author Commented:
Qlemo, that worked!  What was the typo??
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' *  --- SCHEDULEREC STATUS BEGIN*'
vs.
$pattern = (get-date (get-date).AddDays(-1) -f 'MM\/dd\/yyyy') + ' * --- SCHEDULEREC STATUS BEGIN*'
which has been one space too many pre '---' ...
1
samiam41Author Commented:
Thank you both for working tirelessly on your suggested script.  Ultimately the having a script that output the text that retained its formatting was the one I went with as it will make it easier to work with as the script under goes the next evolutions.  Regardless, your time and patience was much appreciated and I look forward to working with you both in the near future.
0
Jose Gabriel Ortega CastroCEOCommented:
Np glad to help either way I always enjoy testing the Qlemo's and Obda's ps codes :) I've learned a few things from them.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Powershell

From novice to tech pro — start learning today.