sharingsunshine
asked on
Help With Shell Script Centos 6
I have Included the script. This is a new install but the access_log is real and has values in it. This script works on another instance without any problems. The script reads the access log and counts how many times an ip hits the instance during a given time period. It then sorts based on the highest hits at the top of the report via ssh.
Here is the message when it is ran
This is an excerpt of the log file
Here is the message when it is ran
[root@ip-172-31-31-103 html]# ./modified_gawk.sh "26 Dec 2016" 06:00:00 "28 Dec 2016" 22:00:00
Examining from Mon Dec 26 06:00:00 UTC 2016 (1482732000)
to Wed Dec 28 22:00:00 UTC 2016 (1482962400)
Processing /var/log/httpd/access_log-20161127 file
This is an excerpt of the log file
77.75.79.72 - - [28/Dec/2016:22:00:41 +0000] "GET /robots.txt HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.79.72 - - [28/Dec/2016:22:00:41 +0000] "GET /category/vce/images/vce/20081117-fungusgnat.jpg HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.79.72 - - [28/Dec/2016:22:00:42 +0000] "GET /wp-content/uploads/wpcf7_captcha/1575611804.png HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.79.72 - - [28/Dec/2016:22:00:44 +0000] "GET /about/ HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
52.3.127.144 - - [28/Dec/2016:22:07:57 +0000] "GET /robots.txt HTTP/1.1" 500 4596 "-" "ltx71 - (http://ltx71.com/)"
211.0.154.124 - - [28/Dec/2016:22:08:57 +0000] "GET / HTTP/1.1" 500 4596 "-" "Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"
202.126.123.53 - - [28/Dec/2016:22:10:24 +0000] "GET /page/19/?page_id=0 HTTP/1.1" 500 4596 "http://www.fluvannamg.org/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60"
modified_gawk.sh
ASKER
[root@ip-172-31-31-103 log]# head /var/log/httpd/access_log- 20161127
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET / HTTP/1.1" 403 4891 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://ec2-54-197-220-84.compute-1.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /icons/poweredby.png HTTP/1.1" 200 3412 "http://ec2-54-197-220-84.compute-1.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:25:49 +0000] "GET /phpinfo.php HTTP/1.1" 200 86740 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:36:57 +0000] "GET / HTTP/1.1" 403 4891 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:38:26 +0000] "GET /phpmyadmin HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
84.1.59.178 - - [25/Nov/2016:19:49:31 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
1.2.3.4 - - [25/Nov/2016:22:34:17 +0000] "GET /phpmyadmin HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
[root@ip-172-31-31-103 log]# tail /var/log/httpd/access_log- 20161127
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/js/whitelist.p hp?lang=en &db=&token =71300608a 5431b592e6 17e9b1f492 c47 HTTP/1.1" 200 2269 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/js/messages.ph p?lang=en& db=&token= 71300608a5 431b592e61 7e9b1f492c 47 HTTP/1.1" 200 16784 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/phpmyadmin.css .php?serve r=1&token= 71300608a5 431b592e61 7e9b1f492c 47&nocache =590297908 7ltr HTTP/1.1" 200 89029 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomm e/img/logo _left.png HTTP/1.1" 200 2327 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomm e/img/ajax _clock_sma ll.gif HTTP/1.1" 200 1810 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomm e/img/left _nav_bg.pn g HTTP/1.1" 200 297 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/version_check. php?server =1&token=7 1300608a54 31b592e617 e9b1f492c4 7&_nocache =148011551 3607452343 HTTP/1.1" 200 39 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
42.119.113.169 - - [26/Nov/2016:03:40:15 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
78.168.135.149 - - [26/Nov/2016:18:50:14 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
123.192.152.207 - - [26/Nov/2016:23:29:04 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Looks like I had my dates wrong. However, I changed them and still get no results.
[root@ip-172-31-31-103 html]# ./modified_gawk.sh "25 Nov 2016" 19:20:00 "26 Nov 2016" 23:29:00
Examining from Fri Nov 25 19:20:00 UTC 2016 (1480101600)
to Sat Nov 26 23:29:00 UTC 2016 (1480202940)
Processing /var/log/httpd/access_log- 20161127 file
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET / HTTP/1.1" 403 4891 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://ec2-54-197-220-84.compute-1.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /icons/poweredby.png HTTP/1.1" 200 3412 "http://ec2-54-197-220-84.compute-1.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:10:26 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:25:49 +0000] "GET /phpinfo.php HTTP/1.1" 200 86740 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:36:57 +0000] "GET / HTTP/1.1" 403 4891 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
1.2.3.4 - - [25/Nov/2016:19:38:26 +0000] "GET /phpmyadmin HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
84.1.59.178 - - [25/Nov/2016:19:49:31 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
1.2.3.4 - - [25/Nov/2016:22:34:17 +0000] "GET /phpmyadmin HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:50.0) Gecko/20100101 Firefox/50.0"
[root@ip-172-31-31-103 log]# tail /var/log/httpd/access_log-
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/js/whitelist.p
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/js/messages.ph
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/phpmyadmin.css
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomm
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomm
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/themes/pmahomm
1.2.3.4 - - [25/Nov/2016:23:11:53 +0000] "GET /phpmyadmin/version_check.
42.119.113.169 - - [26/Nov/2016:03:40:15 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
78.168.135.149 - - [26/Nov/2016:18:50:14 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
123.192.152.207 - - [26/Nov/2016:23:29:04 +0000] "GET / HTTP/1.0" 403 4891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Looks like I had my dates wrong. However, I changed them and still get no results.
[root@ip-172-31-31-103 html]# ./modified_gawk.sh "25 Nov 2016" 19:20:00 "26 Nov 2016" 23:29:00
Examining from Fri Nov 25 19:20:00 UTC 2016 (1480101600)
to Sat Nov 26 23:29:00 UTC 2016 (1480202940)
Processing /var/log/httpd/access_log-
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Description for each of them
1. forward slash is a special character hence written \/
2. 2[0-9] => number 2 following by any one digit number. Any number from 20 to 29
3. [0-9][0-9] => any 2 digit number
4. 2[0-3] => number 2 followed by one digit number from 0 to 3. number from 20 or 22 or 23
5. [0-1][0-9] => 1 digit number (either 0 or 1) followed by any 1 digit number. Number from 00 to 19
6. 2[1-2] => 2 followed by 1 or 2. Either 21 or 22
7. 2[0-8] => 2 followed by one digit number from 0 to 8. From number 20 to 28
Let me know if more required.
1. forward slash is a special character hence written \/
2. 2[0-9] => number 2 following by any one digit number. Any number from 20 to 29
3. [0-9][0-9] => any 2 digit number
4. 2[0-3] => number 2 followed by one digit number from 0 to 3. number from 20 or 22 or 23
5. [0-1][0-9] => 1 digit number (either 0 or 1) followed by any 1 digit number. Number from 00 to 19
6. 2[1-2] => 2 followed by 1 or 2. Either 21 or 22
7. 2[0-8] => 2 followed by one digit number from 0 to 8. From number 20 to 28
Let me know if more required.
1. Initially find the lines having 25/Nov/2016:19:2 followed by any one digit number =>
1. either /bin/grep -E => -E to mention that the string is regular expression
2. or /bin/egrep => /bin/egrep being used to use regular expression
25\/Nov\/2016:19:2[0-9]:[0-9][0-9]
2. Next find the lines having 25/Nov/2016:20 to 23: followed by any 2 digit number =>
25\/Nov\/2016:2[0-3]:[0-9][0-9]:[0-9][0-9]
Like the same add further regular expression for searching related string search using1. either /bin/grep -E => -E to mention that the string is regular expression
2. or /bin/egrep => /bin/egrep being used to use regular expression
Also always use full path instead of only the command
/bin/ls for ls
/bin/egrep for egrep
/usr/bin/tail for tail
/usr/bin/head for head
....
Three dot in regular expression means any three charcters :)
/bin/ls for ls
/bin/egrep for egrep
/usr/bin/tail for tail
/usr/bin/head for head
....
Three dot in regular expression means any three charcters :)
The file name is "Processing /var/log/httpd/access_log-20161127" file which in my opinion is until November 27 and data you are searching is between Dec 26 to Dec 28. Is that correct?
Could you post the head and tail of file /var/log/httpd/access_log-
Open in new window
Thanks,
Sudeep