We value your feedback.
Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!
Course Of The Month
Become a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Regex Problem Using Perl One-liner
Posted on 2013-05-16
I am trying to help a friend sort out a problem with some web sites that he administers. These are Joomla sites and he has found that the files for each of the sites have been hacked and iFrames have been injected. He wants to remove these dubious iFrames from all the sites and asked if I could provide some code to do this which he could run as a command at SSH.
A search of the web showed that one way of removing these was to use grep and sed but my knowledge of bash is limited and I’m much more knowledgable with Perl so I looked for a Perl solution. The offending code is of the form
The code now seemed to work all right in removing the offending code. However when I tried it on a file that had two lots of that bad code it didn’t remove both bits despite the ‘g’ option in the regex, and in fact it didn’t remove the first bit properly.
Can someone explain why this code didn’t work with the CRLF line endings, and also why the global version didn’t remove all the bad code. Can I make any change to the one liner so that it works properly?
A search of the web showed that one way of removing these was to use grep and sed but my knowledge of bash is limited and I’m much more knowledgable with Perl so I looked for a Perl solution. The offending code is of the form
<!-- . --><iframe width="1px" height="1px" src= “http://abcd/fghi/appropriate/promise-ourselves.php” style="display:block;" ></iframe>
<!-- . -->
I thought that there might be some other legitimate iFrames on the site so I set about producing some code that would remove the <!-- . --> tags and the iFrames within them and produced the following online Perl code:<!-- . -->
perl -pi.bak -e '$pattern="<!-- . -->"; s/$pattern.+?$pattern//gs'
The first problem I came across was that this didn’t find the offending code although I had included the ’s’ option to treat everything as a single line. Although the regex I was using worked perfectly well using the Perl code normally I found that the problem seemed to be due to the fact that the file I had been given to test had CRLF line endings. I used another one liner to change all line endings to LF and tried my code again.The code now seemed to work all right in removing the offending code. However when I tried it on a file that had two lots of that bad code it didn’t remove both bits despite the ‘g’ option in the regex, and in fact it didn’t remove the first bit properly.
Can someone explain why this code didn’t work with the CRLF line endings, and also why the global version didn’t remove all the bad code. Can I make any change to the one liner so that it works properly?
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
7
6-
6
20 Comments
Active 6 days ago
perl -p explicitly loops over each line in the input file. You need to slurp the file in and manipulate it as a file. Something like:
perl -i.bak -e '$/ = undef; $f = <>; $pattern = "<!-- . -->"; $f =~ s{$pattern.+?$pattern}{}gs; print $f' `find . -name "*" -type f`
Many thanks for this quick reply. I used MO=Deparse to find out what was happening and I should have realised that the <> was just bringing in one line at a time and just checking that.
My friend will be very happy when I give him the revised code - he has to remove the offending code from 29 different websites.
My friend will be very happy when I give him the revised code - he has to remove the offending code from 29 different websites.
I had seen this -0777 parameter used but didn't register just what it did. I see from a Google search that it slurps the whole file into the $_ variable so this is another way to produce the result I wanted, thanks.
Oh dear, I've been too hasty, neither solution is working as I want it to.
I produced two files, index.html and index2.html. The first one contained that piece of offending iFrame and then I duplicated it, to check that both iFrames would be removed, and this had CRLF endings. The second was identical but with LF endings. I'm attaching both files. I also duplicated the files in subfolders, just to check that the one-liner was working recursively correctly.
I first of all tried to run the code in the main folder by replacing the back ticked 'find' with * so as not to change the sub folders. The version from ozo did remove the iFrames from both files, with the expected warning about not being able to anything with the folders in that main folder. However all the line endings were removed, which I didn't want to happen. I then tried the version with the 'find' and it did work recursively all right but with the same problem with the line endings.
I next tried wilcoxon's version, again changing the back ticked 'find' with a *. This removed the iFrames from the index.html file but left the other unchanged, and there was no warning about the folders. As with ozo's version all the line endings had been removed. I then tried the version with the 'find' but it did nothing at all.
Help!
index.html
index2.html
I produced two files, index.html and index2.html. The first one contained that piece of offending iFrame and then I duplicated it, to check that both iFrames would be removed, and this had CRLF endings. The second was identical but with LF endings. I'm attaching both files. I also duplicated the files in subfolders, just to check that the one-liner was working recursively correctly.
I first of all tried to run the code in the main folder by replacing the back ticked 'find' with * so as not to change the sub folders. The version from ozo did remove the iFrames from both files, with the expected warning about not being able to anything with the folders in that main folder. However all the line endings were removed, which I didn't want to happen. I then tried the version with the 'find' and it did work recursively all right but with the same problem with the line endings.
I next tried wilcoxon's version, again changing the back ticked 'find' with a *. This removed the iFrames from the index.html file but left the other unchanged, and there was no warning about the folders. As with ozo's version all the line endings had been removed. I then tried the version with the 'find' but it did nothing at all.
Help!
index.html
index2.html
The only line endings that are removed are the ones between the <!-- . --> <!-- . -->
along with everything else removed between the <!-- . --> <!-- . -->
If, instead of removing everything, you want to replace it with \n or \r\n,
you can do
s/$pattern.+?$pattern/\r\n
along with everything else removed between the <!-- . --> <!-- . -->
If, instead of removing everything, you want to replace it with \n or \r\n,
you can do
s/$pattern.+?$pattern/\r\n
Active 6 days ago
Good point. I forgot about that part. Hopefully, this version will fix that.
Otherwise, it gets longer...
perl -i.bak -e '$/ = undef; while (@ARGV) { $f = <>; $pattern = "<!-- . -->"; $f =~ s{$pattern.+?$pattern}{}gs; print $f; shift @ARGV; }' `find . -name "*" -type f`
Otherwise, it gets longer...
perl -i.bak -e '$/ = undef; foreach my $n (@ARGV) { open(IN,$n) or die $!; $f = <IN>; close IN; $pattern = "<!-- . -->"; $f =~ s{$pattern.+?$pattern}{}gs; print $f' `find . -name "*" -type f`
Active 6 days ago
If the -0777 works for you, it will produce shorter code than my solution (but both should work). The only problem I can think of with it is if any of the files are any form of unicode (0777 is a valid unicode character so could malfunction).
Hi ozo, don't know what I was thinking about when I said the line endings were all removed, of course as you say the only ones in the examples were within the code I wanted to remove. You code works as it should, sorry about doubting that!
You mentioned that there was no looping in wilcoxon's original code. I did try just adding the 'p' but that didn't work.
Wilcoxon, thanks for the other two suggestions. I've tried both - the first one for some reason or other just adjusts the second file, index2.html, in all the folders but leaves the first ones untouched. Your second one failed to compile because of a missing bracket, and I couldn't work out where it should be placed.
You mentioned that there was no looping in wilcoxon's original code. I did try just adding the 'p' but that didn't work.
Wilcoxon, thanks for the other two suggestions. I've tried both - the first one for some reason or other just adjusts the second file, index2.html, in all the folders but leaves the first ones untouched. Your second one failed to compile because of a missing bracket, and I couldn't work out where it should be placed.
Active 6 days ago
The bracket in the second version should go at the end of the perl code section:
perl -i.bak -e '$/ = undef; foreach my $n (@ARGV) { open(IN,$n) or die $!; $f = <IN>; close IN; $pattern = "<!-- . -->"; $f =~ s{$pattern.+?$pattern}{}gs; print $f }' `find . -name "*" -type f`
-p adds
while( <> ){
...
}continue{ print or die "-p destination: $!\n"; }
around your program
so if you use it, the $f = <>; and print $f are uncalled-for
You should either include the loop in your own code, or use the <> and print from the -p loop
while( <> ){
...
}continue{ print or die "-p destination: $!\n"; }
around your program
so if you use it, the $f = <>; and print $f are uncalled-for
You should either include the loop in your own code, or use the <> and print from the -p loop
Active 6 days ago
Based on ozo's comment about what -p adds and some testing, this will work (and is slightly simpler than my previous answer):
-0777 will work fine in most instances but I'm not a big fan of non-obvious behaviors with hidden gotchas (eg -0777 will choke on (some) unicode files).
perl -i.bak -e '$/ = undef; while (<>) { $pattern = "<!-- . -->"; s{$pattern.+?$pattern}{}gs; print }' `find . -name "*" -type f`
-0777 will work fine in most instances but I'm not a big fan of non-obvious behaviors with hidden gotchas (eg -0777 will choke on (some) unicode files).
Active 6 days ago
Is that a recent change? I know you used to be able to specify octal 777 (and hex 0x1FF will have the same potential issue - it's a valid unicode character (though likely uncommon depending on which unicode encoding the file uses)).
It looks like it's been there since 2003
http://www.nntp.perl.org/group/perl.perl5.changes/2003/04/msg7155.html
If you want to slurp the whole file, you can use -0777, same as pre-Unicode
If you want to separate lines with ¿ (latin small letter o with stroke and acute), you can use -0x1FF
http://www.nntp.perl.org/group/perl.perl5.changes/2003/04/msg7155.html
If you want to slurp the whole file, you can use -0777, same as pre-Unicode
If you want to separate lines with ¿ (latin small letter o with stroke and acute), you can use -0x1FF
Many thanks to both of you for your help in finding a solution to my problem, and for the interesting information about how these one-liners work.
I was too hasty in taking wilcoxon's initial solution without checking that it provided the solution I required, and then ozo's one worked (despite my thinking initially that there was a problem with missing line endings!). Ozo's one was more concise and worked perfectly with the websites that had to be amended so I have split the points, giving zoo 100 more than wilcoxon for this reason.
I was too hasty in taking wilcoxon's initial solution without checking that it provided the solution I required, and then ozo's one worked (despite my thinking initially that there was a problem with missing line endings!). Ozo's one was more concise and worked perfectly with the websites that had to be amended so I have split the points, giving zoo 100 more than wilcoxon for this reason.
Featured Post
We value your feedback.
Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
On Microsoft Windows, if when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions.
Consider email address RFCs:
Look at HTML5 form input element (with type=email) regex pattern:
T…
Suggested Courses
Earn Certification
HTML5 Specialist - Certification
Free withPremium
Course of the Month4 days, 23 hours left to enroll
635 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.