Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Cat, Grep, Variable and a list file

Posted on 2006-11-03
9
3,181 Views
Last Modified: 2013-12-26
Ok here's the situation. I need to strip all lines from a long file which contain certain words. This list of words is ever-growing, and will probably stay that way. Currently I have a bash script that cats a file, then uses grep -v to strip that line, then it outputs it to another file after all the stripping is complete. At first it wasn't difficult, until I realised that I could only use so many instances of grep in a row before the script would break. So I was resolved to output it to another file and cat it again. I know there is an easier way to do this, I just don't know enough about shell scripting to figure it out.

Basicly what I want to have happen is I would like to maintain a separate file containing a list of words who's lines need to be stripped. I would then like the bash script to cat that file, grep for the variable and then output it to a new file with the words stripped.

Kinda like this.
cat filename.txt | grep -v $stripme > stripped.txt

I just don't know how to tell the script that $stripme is the contents of the stripme.txt file. The contents of the stripme.txt file can consist of 1 or more words, each on a new line. Example:
Item A
Bob
Snack Master Supreme
John


500 points for working code, and my thanks.
0
Comment
Question by:DarkHound
9 Comments
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866805

taking your sample above as "search.lst":
(assuming the file to process is "process.file")

#!/bin/sh

rm temp.file
cp process.file temp.file

cat search.lst | while read line; do
    grep -v "$line" temp.file > temp.file2
    mv temp.file2 temp.file
done


Cheers,
C.
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866843
however...

if your system has a decent implementation of "grep" the same can be done using the one-liner:

grep -vf stripme.txt filename.txt

this is probably the way to go :-)

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867296
I've tried both of these, neither seem to strip the lines.
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 11

Expert Comment

by:cjjclifford
ID: 17867353
Hi,

Here's a test session showing both working as expected (for me on OSX)

$ cat > stripme.txt << EOF
> hello world
> this is a test
> goodbye creul world
> EOF
$
$
$
$ cat > filename.txt << EOF
> line should not be stripped
> hello world
> yet another line that should not be stripped
> this is a test line that should be stripped
> goodbye creul world, see you tomorrow
> last line should not be stripped
> EOF
$
$
$ cp filename.txt filename2.txt
$
$
$ cat stripme.txt | while read line; do
> grep -v "$line" filename.txt > filename.txt.tmp
> mv filename.txt.tmp filename.txt
> done
$
$ cat filename.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$
$
$ grep -vf stripme.txt filename2.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$


The "grep -vf" relies on specific Grep implementation, the shell approach should work on any grep implementation, as its not using anything special...

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867607
Ok well here's what my code looks like. It seems the same except for the mv bit at the end. The system seems to be processing the file, but the resulting file has exactly the same number of lines as the orginal file.

#!/bin/bash
rm temp.file
cp cigna temp.file
 
cat list.lst | while read line; do
   grep -v "$line" temp.file > cigtst.txt
done
echo "Rows successfully deleted"
0
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 17869323
grep -f stripme.txt -v filename.txt > stripped.txt
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17873406
you need to output to a temporary file, and make that the working file each iteration through the loop, otherwise only the last entry in "stripme.txt" will get eliminated.

But, as I suggested, and Ozo has also suggested, try the one-liner grep command, it may work immediately, much cleaner, and faster too,...


Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17880756
It seems that grep -vf stripme.txt filename.txt did not work, but Ozo's variation (grep -f stripme.txt -v filename.txt > stripped.txt) did. I'm not sure why there is a difference, probably due to the older version of AIX on this box.
0
 

Expert Comment

by:adamrosshill
ID: 22225017
I would love to know why this actually works..  if you would be so kind to as explain it.
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Separate files macro - Errors on larger data set 18 73
How to know only "File created" with EventLog 3 35
fizzArray2 challenge 1 91
Detect file exist or not 3 179
Introduction: Dialogs (2) modeless dialog and a worker thread.  Handling data shared between threads.  Recursive functions. Continuing from the tenth article about sudoku.   Last article we worked with a modal dialog to help maintain informat…
If you use Adobe Reader X it is possible you can't open OLE PDF documents in the standard. The reason is the 'save box mode' in adobe reader X. Many people think the protected Mode of adobe reader x is only to stop the write access. But this fe…
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question