powershell script to delete string in text files

simon vitali
simon vitali used Ask the Experts™
on
Hello
My goal is to have a script delete a string(error message) from all text files in a folder. (*.*)
I just need the script to find that string and delete the whole line.
Is it possible to customize this script to find more than one string and deletes those lines as well?
Thanks in advance, i did some searching but my powershell knowledge is newb level and I will require some more practice.  I though a select-string could do this but i could not get it to delete the string I found
Thank You
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
RobertSystem Admin

Commented:
Try something like the following.
The script will do the following.
get all files in the root
run a loop through each file and select all lines that do not match your string
then saves the lines (not matching the string) back to the file back to the original name.  
be sure to test with a sample first.

#replace path with path you want
$root = 'C:\temp'
Get-ChildItem $Root -file | 
        Foreach-Object {
        $content = Get-Content $_.FullName
        #replace xxxx below with string you want to search 
        $content | ? {$_.trim() -ne "xxxx" } | Set-Content $_.FullName
       
   }

Open in new window

Author

Commented:
thanks im going to give this a try
two questions for you
1) will this same logic and functionality apply if there are multiple files in the c:\temp folder meaning that the end result will be the same file name but without the search string?
2) is the search string "xxxx" case sensitive?

thank you

Author

Commented:
some notes after running the script
it takes a really long time to run on a 57MB file, over an hour running and it did not finish so had to stop the script as it was bogging down my pc
stopped the script and the source file didnt change in size but this is probably because i terminated the script before completion

so this is not feasible as i would be using this script on even larger files(200MB+)
do you know any command line utility that could accomplish my goal? like findstr

the difficulty im having is finding something that works like a textpad editor in command line form...because this script will be intended for an end user
thank you for any advice.
CompTIA Network+

Prepare for the CompTIA Network+ exam by learning how to troubleshoot, configure, and manage both wired and wireless networks.

Top Expert 2014

Commented:
Instead of reading the entire file into a variable, pipe the file contents.

A variation on Robert's script for large files:
#replace path with path you want
$root = 'C:\temp'
Get-ChildItem $Root -file | 
    Foreach-Object {
	  $Filename = $_.FullName
      Get-Content $FileName -readcount 2000 |
      Foreach-Object {
        #replace xxxx below with string you want to search 
        Foreach ($line in $_){
	  if ($line.trim() -notmatch "xxxx") {
          out-file -FilePath "$Filename.new" -InputObject $line -append} 
         }
      }
      remove-item $Filename
      rename-item -Path "$Filename.new" -NewName $Filename	  
   }

Open in new window

RobertSystem Admin

Commented:
Most files only take a few seconds on my PC.
That said I was only using short strings in the where clause when I tested, so if your passing a long string to find it would likely increase the time needed to run.

Author

Commented:
so im using aikimark's script
my source file started of at 164kb searching for word 'violation'
after the script ran it generated the new file without the string 'violation' BUT the new file was somehow bigger
how is that possible?

Author

Commented:
Robert i tried your script like so in powershell ISE

#replace path with path you want
$root = 'C:\temp\psproject\test\'
Get-ChildItem $Root -file |
        Foreach-Object {
        $content = Get-Content $_.FullName
        #replace xxxx below with string you want to search
        $content | ? {$_.trim() -ne "Violation" } | Set-Content $_.FullName
       
   }

it runs but the file is unchanged, is the string case sensitive? or maybe the $root is where i need to specify 'c:\temp\psproject\test\*.*'

Thank You
Ben Personick (Previously QCubed)Lead SaaS Infrastructure Engineer

Commented:
update 2: PS Method is finished but untested.


update 1: CMD Method is finished but untested,I suspectIhavethe wrong switcheson the move.  Powershell to come later.

----

I tried to do this when I thought I had more time, gotta pick up the Kids eat go night night etc, the below are shells they don't do much of anything yet, gonna post becauee on mobile, you can just ignore them untilI have time to update


In CMD Script

@(
  Setlocal enable delayed expansion
  Set "_Path=C:\Some\Folder\Path"
  Set "_TmpPath=%TEMP%\logchange"
  Set "_FileGlob=*.txt"
  Set "_MatchString=Violation"
  Set "_MatchLineRegex=^.*!_MatchString!.*$"
  Set "_MatchedFile="
  Set "
  Echo off
)
IF EXIST "%_TmpPath%\" (
  RD /Q "%_TmpPath%\"
)
MD "%_TmpPath%\"

ECHO(Find all files with the matching string in them to create a list

FOR /F "Tokens=1 delims=:" %%A IN ('
  Findstr  /I /R
  /C:"%_MatchRegex%"
 "%_Path%\%_FileGlob%"
') DO (
    IF /I "%%A" NEQ "!_MatchedFile!" (
      Set "_MatchedFile=%%A"
      Set "_FileList=!_FileList! "%%A""
      Find  /I /V "%_MatchString%" "%_Path%\%%A">>"%_TmpPath%\%%A"
  )
)
MOVE /F /Y "%_TmpPath%\*.*" "%_Path%\" 

Open in new window



InPowershell
$MatchString="Violation"
$MatchLineRegex=^.*Violation.*$"
$SearchPath="C:\Your\Folder\path"
$FileList = Gci $Searchpath -File " | Select-string $MatchString -list
Foreach ($File in $FileList) {
  $TContent=$($(Get-content $File ) -replace $MatchLineRegex,$Null)
  $TContent | Set-Content $File
}

Open in new window

System Administrator / Postmaster
Commented:
$root = '~/temp'
# set exclusions: none 
$exclude = $null
$SearchString = 'Volumes'
# find all files which have $searchstring inside 
$object = Get-ChildItem -Path $root -Recurse -exclude $exclude | Select-String $SearchString  -list # list: first occurence
# delete all lines which contain $searchstring and overwrite source file
$object | % { (get-content  $_.path) | select-string $SearchString -notmatch | Set-Content $_.path}

Open in new window

RobertSystem Admin

Commented:
No the string should not be case sensitive. If your file is in a sub folder you would need to add the -recurse option after the get-childitem.

Author

Commented:
@Michelangelo
that works like a charm!
if I would want to add more than one search string would it be?

$SearchString = 'Volumes' 'stringb' 'stringc'

Thank You

Author

Commented:
thank you all for your input/help
i shall pay forward your kindness!
cheers and be well
MichelangeloSystem Administrator / Postmaster
Commented:
Answering to Simon's last question:
use a pipe  ( same as unix grep -iE)  to separate search strings inside $searchstring. Find amended script below where I also explicited -pattern parameter keyword to underline the commandlet is using regexps.
Also, have a look at select-string manpage.

$root = '~/temp'
# set exclusions: none 
$exclude = $null
$SearchString = 'Volumes|SecondSearchString'
# find all files which have $searchstring inside 
$object = Get-ChildItem -Path $root -Recurse -exclude $exclude | Select-String $SearchString  -list # list: first occurence
# delete all lines which contain $searchstring and overwrite source file
$object | % { (get-content  $_.path) | select-string $SearchString -notmatch | Set-Content $_.path}

Open in new window

Author

Commented:
so im back!
the conditions for this project has changed a little rendering the script that Michelangelo had provided no longer useful for my cause although it worked for the previous purpose

heres what changed:
the files in questions are XML format (but are not named .xml they are .901 files)
i need to remove a whole node from the *.xml files ,  including the tag

i.e.
 <errsql yadadayda > 
random sql error the statement has terminated
</errsql>

this whole line need to be removed form the file

here is a snippet of the xml file:

<chgsql ts="07:01:58.971" op="O" jb="ADD" tb="CLL_TAB" sc="BATCH">
<fld fd="F1000" af="PAL"/>
<fld fd="F1148" af="43000135163"/>
<idx fd="F1577" af="5167599565"/>
<idx fd="F1578" af="LINK"/>
<fld fd="F1001" af="1"/>
<fld fd="F902" af="00003814"/>
</chgsql>
<errsql ts="07:01:58.986" op="O" lv="50" ob="">
<sql>INSERT INTO CLL_TAB [F1000,F1148,F1577,F1578,F1001,F902] VALUES [:F1000,:F1148,:F1577,:F1578,:F1001,:F902] </sql>
<prm>PAL,43000135163,5167599565,LINK,1,00003814,</prm>
</errsql>
<chgsql ts="07:01:58.986" op="O" jb="ADD" tb="CLL_TAB" sc="BATCH">
<fld fd="F1000" af="PAL"/>
<fld fd="F1148" af="43000135164"/>
<idx fd="F1577" af="5166282554"/>
<idx fd="F1578" af="LINK"/>
<fld fd="F1001" af="1"/>
<fld fd="F902" af="00003814"/>
</chgsql>
<errsql ts="07:01:58.986" op="O" lv="50" ob="">
<sql>INSERT INTO CLL_TAB [F1000,F1148,F1577,F1578,F1001,F902] VALUES [:F1000,:F1148,:F1577,:F1578,:F1001,:F902] </sql>
<prm>PAL,43000135164,5166282554,LINK,1,00003814,</prm>
</errsql>


here is what i have as a command thats not working so far which is Michealangelo's script modified to search using xpath

$root = 'C:\storeman\Office\ZY999901\*.*'
# set exclusions: none 
$exclude = $null
$SearchString = select-xml -Path 'C:\storeman\Office\ZY999901\*.*' -XPath //errsql | Select-Object -ExpandProperty node
# find all files which have $searchstring inside 
$object = Get-ChildItem -Path $root -Recurse -exclude $exclude | Select-String -pattern $SearchString  -list # list: first occurence
# delete all lines which contain $searchstring and overwrite source file
$object | % { (get-content  $_.path) | select-string -pattern $SearchString -notmatch | Set-Content $_.path}

Open in new window

my xpath command works on its own just fine excpet it doesnt include the <errsql> </errsql> tags in results , i could live with this

when i run the command in powershell ISE it runs for a very long time and doesnt return and results or errors but it doesnt do antything
pls help, thank you
Top Expert 2014

Commented:
I'd use a -replace operator
$root = 'C:\storeman\Office\ZY999901'
gci -path $root -filter '*.901' -file -recurse |
  % { (get-content  $_.fullname) -replace '<errsql (?:.|\n)+?</errsql>', ''  | 
  Set-Content $_.path}

Open in new window

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial