?
Solved

increasing Corrupted logs in Awstats:: running on Unix and parsing the logs of Apache running on Unix.

Posted on 2009-04-29
33
Medium Priority
?
2,536 Views
Last Modified: 2013-12-07
I have configured Awstats 6.9 on one of my Unix machine and parsing the apache logs for one of my website configured on Apache running on Unix,now days i am getting increment in corrupted logs for Awstats Parsing.
LogFormat we are using in Awstats conf file is="%host %other %logname %time1 %methodurl %code %bytesd %uaquot"]
but it is treating as corrupted for the logs given below...
Corrupted record line 4061381 (record format does not match LogFormat parameter): 127.0.0.1 - - [18/Apr/2009:23:57:57 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "72057730050135041,0"
Corrupted record line 4061391 (record format does not match LogFormat parameter): 127.0.0.1 - - [18/Apr/2009:23:58:11 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "97357515777,0"
How to resolve this problem......?
0
Comment
Question by:Brijeshk9
  • 20
  • 4
  • 4
  • +1
29 Comments
 
LVL 51

Expert Comment

by:ahoffmann
ID: 24267515
the logformat you posted does not match the loglines
is there a typo?
0
 

Author Comment

by:Brijeshk9
ID: 24288781
Yes I know, hence I am asking about the solution on it. I want to know whether any change is required with application running on that Box (Web server-from where I m getting logs) because I have placed the above given format as defines in web server for creating logs and we are getting most of the lines as per given format, but the lines mentioned above showing hits from local host and treating as corrupted by Awastes...! Also confirm about the format coming in these corrupted logs.
I just want to solve this corrupted logs issue and one more thing all corrupted logs are showing hit from local host: 127.0.0.1
      
0
 

Author Comment

by:Brijeshk9
ID: 24292276
And the logs which are getting parsed successfully  by Awstats are:(not coming as corrupted entry)
XX.XX.XX.XX - - [02/May/2009:00:00:01 -0600] "GET /portal/page/portal/IRPS_Main/IRPS_Home/Home_content HTTP/1.1" 200 11501 "128572556609,0"
XX.XX.XX.XX - - [02/May/2009:00:00:01 -0600] "POST /portal/pls/portal/!PORTAL.wwpob_page.show/ADMINTOOLPAGESPGRP/AdminToolsMain HTTP/1.1" 200 5377 "115687654719,2"
[Note:chnaged the Ip to XX here]
but all the logs coming in below given format are accepted as corrupted as corrupted logs.
Corrupted record line 3555735 (record format does not match LogFormat parameter): 127.0.0.1 - - [02/May/2009:23:59:32 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "90006294529,0"
Corrupted record line 3555738 (record format does not match LogFormat parameter): 127.0.0.1 - - [02/May/2009:23:59:39 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "72057679749262337,0"
Corrupted record line 3555745 (record format does not match LogFormat parameter): 127.0.0.1 - - [02/May/2009:23:59:52 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "90006315009,0"
And IP:127.0.0.1 , Keyword: GET and Page value:  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 :: is same in all corrupted logs:
So I need solution for all the corrupted logs, so that either I can avoid all the local hits/can consider all the local hits  and I also need to understand why I m getting so many log from localhost:127.0.0.1
Finally want to remove all the corrupted logs entry from Awstats Parsing.
0
NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24343358
Hi,

It seems to me that the filename _oracle_http_server_webcache_static_.html causes the problem. Will you just use the awstats with the switch -showcorrupted.

Then open your log file and remove the leading "_" underscore before Oracle and rerun awstats if corrupted entry was stooped for the line. If not next try the trailing "_" and rerun. If not try ro rename the entry to read oracle.html and retry.

If the drooped status disappeared for the line in any of the attempts please go to this address and add a new bugtrack entry.

Cheers,
K.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24343372
In fact if the line was stopped being dropped out in any of the attemmpts it means that there's a problem in parsing the filename in either with a filename starting with a leadin or thailing underscore or this is a name length issue.

In either case it means that this is a bug and you need to create a bug report. I've forgot to attach the link to create the bug-port link in the last posting.

Here's the link:
http://sourceforge.net/tracker/?group_id=13764&atid=113764&func=browse

Visit there and click "Add new" just under the Summary at the top left of the page.

Cheers,
K
 
0
 

Author Comment

by:Brijeshk9
ID: 24343805
And all the hits are coming from local host..!
0
 

Author Comment

by:Brijeshk9
ID: 24343854
Dont you think that these types of hits from local host should not come in log file in such a big manner, is it callback to the same Server...?
Anyway I will try the option suggested by you, but please suggest me on these huge hits from local host....?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24345101
I don't think there's something wrong about these hits. Server logs whatever request it gets. Furthermore This is an Oracle AS behaviour. If you want to avoid tis calls you should disable them from the Oracle AS like that:

Application Server Control Console:

Web Cache Home page > Administration tab > Properties > Web Cache > Auto-Restart

OracleAS Web Cache Manager:

Properties > Auto-Restart

This is enabled by default. You just disable it and no more requests will be logged.

Though this answer is way out of the scope of the initial question :)


0
 
LVL 23

Accepted Solution

by:
Mysidia earned 1500 total points
ID: 24346174
Take a look at this line:
XX.XX.XX.XX - - [02/May/2009:00:00:01 -0600] "GET /portal/page/portal/IRPS_Main/IRPS_Home/Home_content HTTP/1.1" 200 11501 "128572556609,0"

There is ONE space between the GET and the /  inside the  %methodurl.

Now take a look at THIS line:
[02/May/2009:23:59:39 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1"
 200 99 "72057679749262337,0"


The way you posted it, there are  TWO spaces after the GET,  before the/


Try editing awstats.pl   (First make a backup, because this is completely untested)

Find the text that says
                        elsif ( $f =~ /%methodurl$/ ) {


And below it,  try changing
the line just after
                                push @fieldlib, 'url';







FROM
 
                                $PerlParsingFormat .=
"\\\"([^$LogSeparatorWithoutStar]+) ([^$LogSeparatorWithoutStar]+)(?: [^\\\"]+|)
\\\"";
 
TO
 
                                $PerlParsingFormat .=
"\\\"([^$LogSeparatorWithoutStar]+) +([^$LogSeparatorWithoutStar]+)(?: [^\\\"]+|)
\\\"";
 
 
 
i.e. add a  +  after the space between the  GET/POST word and the URL part,  to allow for multiple spaces  between them.

Open in new window

0
 

Author Comment

by:Brijeshk9
ID: 24351975
Thanks a lot Mysidia. it worked and solved my corrupted log issues, but there is no increment in daily stats report like still it is showing the  data as showing previously.
Like now showing 0 corrupted /dropped records  and before this it was showing 300000 corrupted record but report is same in both case..?
Also suggest about awredir.pl like I need to put some extra sections(to track exit clicks) on Awstats(look for example: 3 given at URL: http://awstats.sourceforge.net/docs/awstats_extra.html ) so I need to make some entry on my web server or anything required with awredir.pl on Awstats Box&?
KeremE:  Please confirm how to make changes in conf file, as I do not have any GUI for this &?
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 24352065
That would seem to confirm it as a bug in awstats.

Possibly the corrupted records were still counting towards some stats?

Apache passes along  extra spaces in the "GET"  line before the /  directly to the logs, so if the browser requesting a web page specifies 3  spaces in front of the "/"  in the HTTP request, there will be extra space.

You might also like to check your awstats.conf  file for a line like

SkipHosts="127.0.0.1 "


By default 127.0.0.1  is excluded from statistics, in many cases.
0
 

Author Comment

by:Brijeshk9
ID: 24360834
Yes i have tried it,but not working....!
0
 

Author Comment

by:Brijeshk9
ID: 24421857
Any other suggestion...!
0
 

Author Comment

by:Brijeshk9
ID: 24447738
Well I need to put some more extra sections on  Awstats to improve its performance and to get more data from it. will anyone help me to explore on it
Because its taking very long time  due to lack of documentation/support available on Net for Awstats.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24448689
What about disabling Oracle to check its server as I suggested?? Would you really like to see so many local automatic requests to be counted as wits in awstats???
0
 

Author Comment

by:Brijeshk9
ID: 24456833
No I dont want to count any local hit  in my Awstats report(also dont want any local hit entry in log file)  I am waiting for my Unix Admans resonance on it. Can you please suggest me the required changes in conf (with name) file..? as I do not have any GUI access.
Also suggest if you have any better idea about use/configuration of awredir.pl in Awstats...?
For more detail you can check the Extra sections or page 38 in attached file. or can get it from URL:
http://awstats.sourceforge.net/docs/awstats.pdf      

awstats.pdf
0
 

Author Comment

by:Brijeshk9
ID: 24489491
any idea on using awredir.pl in Awstats for applying extra section..?
0
 

Author Comment

by:Brijeshk9
ID: 24509028
Please suggest..!!
      
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 24509116
What exactly are you trying to accomplish with awredir.pl ?
Seems a bit out of scope of the question asked, which was about log entries getting reported as corrupt...


The awstats docs are pretty detailed on how to use extra sections; you first define the extra section, you can adapt one of the examples, but you need to know details of what you want to track specifically.
And then you change the corresponding links on your site to use awredir.pl....


Honestly,  when I want to start tracking exit clicks,  I start thinking about using  Google Analytics,  or Urchin..
0
 

Author Comment

by:Brijeshk9
ID: 24509190
I agree with you, but my client is not allowing using Google analytics and any other tool except Awstats..!
If you can suggest anything in Awstats. It will be really helpful for me.
Thanks for you Support..!
0
 

Author Comment

by:Brijeshk9
ID: 24529228
Mysidia , sorry again I m getting some corrupted logs, for more detail you can check the below given corrupted log:
Corrupted record line 3114611 (record format does not match LogFormat parameter): 192.168.1.11- - [28/May/2009:06:51:33 -0600] "GET /portal/page/portal/IMP_Main/IMP_AssetsLibrary/logout_button.jpg 172.17.0.1 - - [28/May/2009:06:51:36 -0600] "GET / HTTP/1.0" 200 370 "87948484950,0"
Please suggest for the better/permanent solution on it also confirm if anymore configuration is required on Awstats box .
Thanks.!
Corrupted record line 3114611 (record format does not match LogFormat parameter): 192.168.1.11- - [28/May/2009:06:51:33 -0600] "GET /portal/page/portal/IMP_Main/IMP_AssetsLibrary/logout_button.jpg 172.17.0.1 - - [28/May/2009:06:51:36 -0600] "GET / HTTP/1.0" 200 370 "87948484950,0"

Open in new window

0
 

Author Comment

by:Brijeshk9
ID: 24561938
ok ,now that corrupted log I have commented ,but not sure why two line came in a single log line..!
any  more idea..!
0
 

Author Comment

by:Brijeshk9
ID: 24598568
any suggestion in Awstats..!!
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 24598765
If your seeing two log entries on the same line, that's suggestive of corruption.
If in the actual file, then this would be a bug in Apache,  a fairly common thing.

You might check your Apache config to ensure
BufferedLogs

is disabled.    There is a huge performance advantage of having buffered logs, however,   periodic anomolies are normal with that feature.

And check that you are running the current recommended release of Apache httpd, that would be version  2.2.11.

Failing that.. there is a possibility of placing an additional  \n  at the end of the logformat in Apache


If you  see such lines only in Awstats,  that's not expected behavior either, and indicates possible impact of some bug in awstats.

0
 

Author Comment

by:Brijeshk9
ID: 24626152
ok, can you pls. suggest about the Keywords "TRACE","CONNCET",GNUTELLA,SEARCH because all the lines having these keyword are treated as dropped records  by Awstats Process.
How Can we resolve this..?
For example pls. check the attached output, i got from awstats process.(showing 3 dropped records)
I am also trying to filter out the local ip not to include in Awstats report but no success..?
[webadmin@xyz01 cgi-bin]$ ./awstats.pl -config=www.mytrust.com -update -showdropped
Create/Update database for config "/etc/awstats/awstats.www.mytrust.com.conf" by AWStats version 6.9 (build 1.925)
From data in log file "/usr/local/awstats/tools/logresolvemerge.pl /aw_transfer/data/www.mytrust.com/*/aw_transfer/data/www.mytrust.com/*|"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Dropped record (method/protocol 'TRACE' not qualified when LogType=W): 207.188.34.66 - - [10/Jun/2009:10:20:58 -0600] "TRACE / HTTP/1.0" 403 313 "-"
Dropped record (method/protocol 'TRACE' not qualified when LogType=W): 207.188.34.66 - - [10/Jun/2009:12:07:31 -0600] "TRACE / HTTP/1.0" 403 313 "-"
Dropped record (method/protocol 'CONNECT' not qualified when LogType=W): 65.117.241.100 - - [12/Jun/2009:12:51:35 -0600] "CONNECT http://lti-mail01.ltinetworks.com:25 HTTP/1.0" 400 260 "72057713400389940,0"
Jumped lines in file: 0
Parsed lines in file: 3538562
 Found 3 dropped records,
 Found 0 corrupted records,
 Found 3538559 old records,
 Found 0 new qualified records.

Open in new window

0
 

Author Comment

by:Brijeshk9
ID: 24638680
any idea..!
0
 

Author Comment

by:Brijeshk9
ID: 24648640
Any suggestion please..!
0
 

Author Comment

by:Brijeshk9
ID: 24717458
Kindly let me know, if I can avoid dropped records after putting ips /host in skipped host, because all the skipped ips are coming under dropped records..?
0
 

Author Closing Comment

by:Brijeshk9
ID: 31575896
Thanks for all your help and efforts on It.!
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Although a lot of people devote their energy toward marketing for specific industries, there are some basic principles that can be applied to any sector imaginable. We’ll look at four steps to take and examine how those steps were put into action fo…
Australian government abolished Visa 457 earlier this April and this article describes how this decision might affect Australian IT scene and IT experts.
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
Suggested Courses
Course of the Month16 days, 23 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question