Link to home
Start Free TrialLog in
Avatar of sharingsunshine
sharingsunshineFlag for United States of America

asked on

Help With Shell Script Centos 6

I have Included the script.  This is a new install but the access_log is real and has values in it.  This script works on another instance without any problems.  The script reads the access log and counts how many times an ip hits the instance during a given time period.  It then sorts based on the highest hits at the top of the report via ssh.

Here is the message when it is ran
[root@ip-172-31-31-103 html]# ./modified_gawk.sh "26 Dec 2016" 06:00:00 "28 Dec 2016" 22:00:00
Examining from Mon Dec 26 06:00:00 UTC 2016 (1482732000)
            to Wed Dec 28 22:00:00 UTC 2016 (1482962400)

Processing /var/log/httpd/access_log-20161127 file

Open in new window


This is an excerpt of the log file
77.75.79.72 - - [28/Dec/2016:22:00:41 +0000] "GET /robots.txt HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.79.72 - - [28/Dec/2016:22:00:41 +0000] "GET /category/vce/images/vce/20081117-fungusgnat.jpg HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.79.72 - - [28/Dec/2016:22:00:42 +0000] "GET /wp-content/uploads/wpcf7_captcha/1575611804.png HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.79.72 - - [28/Dec/2016:22:00:44 +0000] "GET /about/ HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
52.3.127.144 - - [28/Dec/2016:22:07:57 +0000] "GET /robots.txt HTTP/1.1" 500 4596 "-" "ltx71 - (http://ltx71.com/)"
211.0.154.124 - - [28/Dec/2016:22:08:57 +0000] "GET / HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"
202.126.123.53 - - [28/Dec/2016:22:10:24 +0000] "GET /page/19/?page_id=0 HTTP/1.1" 500 4596 "http://www.fluvannamg.org/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60"

Open in new window

modified_gawk.sh
Avatar of Sudeep Sharma
Sudeep Sharma
Flag of India image

Hi,

The file name is "Processing /var/log/httpd/access_log-20161127" file which in my opinion is until November 27 and data you are searching is between Dec 26 to Dec 28. Is that correct?

Could you post the head and tail of file /var/log/httpd/access_log-20161127 file?

head /var/log/httpd/access_log-20161127
tail /var/log/httpd/access_log-20161127

Open in new window


Thanks,
Sudeep
Avatar of sharingsunshine

ASKER

[root@ip-172-31-31-103 log]# head /var/log/httpd/access_log-20161127
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET / HTTP/1.1" 403 4891 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://ec2-54-197-220-84.compute-1.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /icons/poweredby.png HTTP/1.1" 200 3412 "http://ec2-54-197-220-84.compute-1.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:25:49 +0000] "GET /phpinfo.php HTTP/1.1" 200 86740 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:36:57 +0000] "GET / HTTP/1.1" 403 4891 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:38:26 +0000] "GET /phpmyadmin HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
84.1.59.178 - - [25/Nov/2016:19:49:31 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
1.2.3.4 - - [25/Nov/2016:22:34:17 +0000] "GET /phpmyadmin HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"



[root@ip-172-31-31-103 log]# tail /var/log/httpd/access_log-20161127
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/js/whitelist.php?lang=en&db=&token=71300608a5431b592e617e9b1f492c47 HTTP/1.1" 200 2269 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/js/messages.php?lang=en&db=&token=71300608a5431b592e617e9b1f492c47 HTTP/1.1" 200 16784 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/phpmyadmin.css.php?server=1&token=71300608a5431b592e617e9b1f492c47&nocache=5902979087ltr HTTP/1.1" 200 89029 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomme/img/logo_left.png HTTP/1.1" 200 2327 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomme/img/ajax_clock_small.gif HTTP/1.1" 200 1810 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomme/img/left_nav_bg.png HTTP/1.1" 200 297 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/version_check.php?server=1&token=71300608a5431b592e617e9b1f492c47&_nocache=1480115513607452343 HTTP/1.1" 200 39 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
42.119.113.169 - - [26/Nov/2016:03:40:15 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
78.168.135.149 - - [26/Nov/2016:18:50:14 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
123.192.152.207 - - [26/Nov/2016:23:29:04 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Looks like I had my dates wrong.  However, I changed them and still get no results.

[root@ip-172-31-31-103 html]#  ./modified_gawk.sh "25 Nov 2016" 19:20:00 "26 Nov 2016" 23:29:00
Examining from Fri Nov 25 19:20:00 UTC 2016 (1480101600)
            to Sat Nov 26 23:29:00 UTC 2016 (1480202940)

Processing /var/log/httpd/access_log-20161127 file
ASKER CERTIFIED SOLUTION
Avatar of MURUGESAN N
MURUGESAN N
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Description for each of them
1. forward slash is a special character hence written \/
2. 2[0-9] => number 2 following by any one digit number. Any number from 20 to 29
3. [0-9][0-9] => any 2 digit number
4. 2[0-3] => number 2 followed by one digit number from 0 to 3. number from 20 or 22 or 23
5. [0-1][0-9] => 1 digit number (either 0 or 1) followed by any 1 digit number. Number from 00 to 19
6. 2[1-2] => 2 followed by 1 or 2. Either 21 or 22
7. 2[0-8] => 2 followed by one digit number from 0 to 8. From number 20 to 28
Let me know if more required.
1. Initially find the lines having 25/Nov/2016:19:2 followed by any one digit number =>
25\/Nov\/2016:19:2[0-9]:[0-9][0-9]

Open in new window

2. Next find the lines having 25/Nov/2016:20 to 23: followed by any 2 digit number =>
25\/Nov\/2016:2[0-3]:[0-9][0-9]:[0-9][0-9]

Open in new window

Like the same add further regular expression for searching related string search using
1. either /bin/grep -E => -E to mention that the string is regular expression
2. or /bin/egrep => /bin/egrep being used to use regular expression
Also always use full path instead of only the command
/bin/ls for ls
/bin/egrep for egrep
/usr/bin/tail for tail
/usr/bin/head for head
....
Three dot in regular expression means any three charcters :)