Need Multiple Patterns In Grep

Posted on 2014-11-11
Last Modified: 2014-11-11
I have the need to grep a log file on a continuous basis.  I have it broken down to 3 steps I can get two of them to work but I am unclear how to add the 3rd grep.

This works but it doesn't cull out the .jpg and such from the list.
egrep -w "Nov" /var/log/access.log | grep -P 'HTTP/1.1\" 404' /var/log/access.log

This is what I have tried that doesn't work.

egrep -w "Nov" /var/log/access.log | grep  -v -E "favicon.ico|robots.txt|mailalerts.js|btn_bg.gif|cgi-bin|.jpg"
 | grep -P 'HTTP/1.1\" 404' /data/log/access_thp.log-20141109

Obviously, I want a way to add in the following portion to a grep that works.
grep  -v -E "favicon.ico|robots.txt|mailalerts.js|btn_bg.gif|cgi-bin|.jpg"

Question by:sharingsunshine
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +1
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40435460
The part that doesn't work is containing a second filename: /data/log/access_thp.log-20141109

Do you want to grep in that file as well?
LVL 84

Expert Comment

ID: 40435466
Can you give an example of a line that was returned by your grep attempts that you did not want, or a line that was not returned that you did want?
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40435469
You'd normally pipe multiple grep's after each other like this:

egrep -w "Nov" /var/log/access.log | grep "something" | grep "something again" | grep ...
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.


Author Comment

ID: 40435803
When I pipe them together it stops the 404 grep.  Consequently, I get all the file types not just the 404's.  Here is what I tried combining them as you suggested:

egrep -w "Nov" /data/log/access_thp.log-20141109 | grep  -v -E "favicon.ico|robots.txt|mailalerts.js|btn_bg.gif|cgi-bin|.jpg"
 | grep -P 'HTTP/1.1\" 404

That file name was a misprint on my part it should be /var/log/access.log  instead.

here is an example what i want to remove from - - [11/Nov/2014:15:19:37 +0000] "GET /392-large_default/Black_Walnut_100_Capsules_p_658.jpg HTTP/1.1" 404 281
LVL 84

Expert Comment

ID: 40435846
If you want to remove - - [11/Nov/2014:15:19:37 +0000] "GET /392-large_default/Black_Walnut_100_Capsules_p_658.jpg HTTP/1.1" 404 281
then shouldn't it be
grep -v 'HTTP/1.1\" 404'

Author Comment

ID: 40435931
sorry I didn't reference back to the original question.  I want the 404's I don't want the specific file types.  You asked for an example of what I didn't want.  Which is this line because it is referencing the .jpg file type. - - [11/Nov/2014:15:19:37 +0000] "GET /392-large_default/Black_Walnut_100_Capsules_p_658.jpg HTTP/1.1" 404 281

Even though this is a 404 I don't want any of these file types
grep  -v -E "favicon.ico|robots.txt|mailalerts.js|btn_bg.gif|cgi-bin|.jpg"
LVL 62

Expert Comment

ID: 40436026
grep -e cat -e dog -e -i november -e -v 2014
LVL 38

Accepted Solution

Gerwin Jansen, EE MVE earned 500 total points
ID: 40436048
How about this:
grep "Nov" /var/log/access.log | grep -P 'HTTP/1.1\" 404' | grep  -v -E "favicon.ico|robots.txt|mailalerts.js|btn_bg.gif|cgi-bin|.jpg"

Open in new window

(in text: You want all 404's from November but not the favicon.ico|robots.txt|mailalerts.js|btn_bg.gif|cgi-bin|.jpg lines)

Author Closing Comment

ID: 40436232
Great job, thanks for the help and the line of code.

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question