increasing Corrupted logs in Awstats:: running on Unix and parsing the logs of Apache running on Unix.

the logformat you posted does not match the loglines
is there a typo?

Yes I know, hence I am asking about the solution on it. I want to know whether any change is required with application running on that Box (Web server-from where I m getting logs) because I have placed the above given format as defines in web server for creating logs and we are getting most of the lines as per given format, but the lines mentioned above showing hits from local host and treating as corrupted by Awastes...! Also confirm about the format coming in these corrupted logs.
I just want to solve this corrupted logs issue and one more thing all corrupted logs are showing hit from local host: 127.0.0.1
      

And the logs which are getting parsed successfully  by Awstats are:(not coming as corrupted entry)
XX.XX.XX.XX - - [02/May/2009:00:00:01 -0600] "GET /portal/page/portal/IRPS_Main/IRPS_Home/Home_content HTTP/1.1" 200 11501 "128572556609,0"
XX.XX.XX.XX - - [02/May/2009:00:00:01 -0600] "POST /portal/pls/portal/!PORTAL.wwpob_page.show/ADMINTOOLPAGESPGRP/AdminToolsMain HTTP/1.1" 200 5377 "115687654719,2"
[Note:chnaged the Ip to XX here]
but all the logs coming in below given format are accepted as corrupted as corrupted logs.
Corrupted record line 3555735 (record format does not match LogFormat parameter): 127.0.0.1 - - [02/May/2009:23:59:32 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "90006294529,0"
Corrupted record line 3555738 (record format does not match LogFormat parameter): 127.0.0.1 - - [02/May/2009:23:59:39 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "72057679749262337,0"
Corrupted record line 3555745 (record format does not match LogFormat parameter): 127.0.0.1 - - [02/May/2009:23:59:52 -0600] "GET  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 "90006315009,0"
And IP:127.0.0.1 , Keyword: GET and Page value:  /_oracle_http_server_webcache_static_.html HTTP/1.1" 200 99 :: is same in all corrupted logs:
So I need solution for all the corrupted logs, so that either I can avoid all the local hits/can consider all the local hits  and I also need to understand why I m getting so many log from localhost:127.0.0.1
Finally want to remove all the corrupted logs entry from Awstats Parsing.

Hi,

It seems to me that the filename _oracle_http_server_webcache_static_.html causes the problem. Will you just use the awstats with the switch -showcorrupted.

Then open your log file and remove the leading "_" underscore before Oracle and rerun awstats if corrupted entry was stooped for the line. If not next try the trailing "_" and rerun. If not try ro rename the entry to read oracle.html and retry.

If the drooped status disappeared for the line in any of the attempts please go to this address and add a new bugtrack entry.

Cheers,
K.

In fact if the line was stopped being dropped out in any of the attemmpts it means that there's a problem in parsing the filename in either with a filename starting with a leadin or thailing underscore or this is a name length issue.

In either case it means that this is a bug and you need to create a bug report. I've forgot to attach the link to create the bug-port link in the last posting.

Here's the link:
http://sourceforge.net/tracker/?group_id=13764&atid=113764&func=browse

Visit there and click "Add new" just under the Summary at the top left of the page.

Cheers,
K
 

And all the hits are coming from local host..!

Dont you think that these types of hits from local host should not come in log file in such a big manner, is it callback to the same Server...?
Anyway I will try the option suggested by you, but please suggest me on these huge hits from local host....?

I don't think there's something wrong about these hits. Server logs whatever request it gets. Furthermore This is an Oracle AS behaviour. If you want to avoid tis calls you should disable them from the Oracle AS like that:

Application Server Control Console:

Web Cache Home page > Administration tab > Properties > Web Cache > Auto-Restart

OracleAS Web Cache Manager:

Properties > Auto-Restart

This is enabled by default. You just disable it and no more requests will be logged.

Though this answer is way out of the scope of the initial question :)

Thanks a lot Mysidia. it worked and solved my corrupted log issues, but there is no increment in daily stats report like still it is showing the  data as showing previously.
Like now showing 0 corrupted /dropped records  and before this it was showing 300000 corrupted record but report is same in both case..?
Also suggest about awredir.pl like I need to put some extra sections(to track exit clicks) on Awstats(look for example: 3 given at URL: http://awstats.sourceforge.net/docs/awstats_extra.html ) so I need to make some entry on my web server or anything required with awredir.pl on Awstats Box&?
KeremE:  Please confirm how to make changes in conf file, as I do not have any GUI for this &?

That would seem to confirm it as a bug in awstats.

Possibly the corrupted records were still counting towards some stats?

Apache passes along  extra spaces in the "GET"  line before the /  directly to the logs, so if the browser requesting a web page specifies 3  spaces in front of the "/"  in the HTTP request, there will be extra space.

You might also like to check your awstats.conf  file for a line like

SkipHosts="127.0.0.1 "


By default 127.0.0.1  is excluded from statistics, in many cases.

Yes i have tried it,but not working....!

Any other suggestion...!

Well I need to put some more extra sections on  Awstats to improve its performance and to get more data from it. will anyone help me to explore on it
Because its taking very long time  due to lack of documentation/support available on Net for Awstats.

What about disabling Oracle to check its server as I suggested?? Would you really like to see so many local automatic requests to be counted as wits in awstats???

No I dont want to count any local hit  in my Awstats report(also dont want any local hit entry in log file)  I am waiting for my Unix Admans resonance on it. Can you please suggest me the required changes in conf (with name) file..? as I do not have any GUI access.
Also suggest if you have any better idea about use/configuration of awredir.pl in Awstats...?
For more detail you can check the Extra sections or page 38 in attached file. or can get it from URL:
http://awstats.sourceforge.net/docs/awstats.pdf      

awstats.pdf

any idea on using awredir.pl in Awstats for applying extra section..?

Please suggest..!!
      

What exactly are you trying to accomplish with awredir.pl ?
Seems a bit out of scope of the question asked, which was about log entries getting reported as corrupt...


The awstats docs are pretty detailed on how to use extra sections; you first define the extra section, you can adapt one of the examples, but you need to know details of what you want to track specifically.
And then you change the corresponding links on your site to use awredir.pl....


Honestly,  when I want to start tracking exit clicks,  I start thinking about using  Google Analytics,  or Urchin..

I agree with you, but my client is not allowing using Google analytics and any other tool except Awstats..!
If you can suggest anything in Awstats. It will be really helpful for me.
Thanks for you Support..!

Mysidia , sorry again I m getting some corrupted logs, for more detail you can check the below given corrupted log:
Corrupted record line 3114611 (record format does not match LogFormat parameter): 192.168.1.11- - [28/May/2009:06:51:33 -0600] "GET /portal/page/portal/IMP_Main/IMP_AssetsLibrary/logout_button.jpg 172.17.0.1 - - [28/May/2009:06:51:36 -0600] "GET / HTTP/1.0" 200 370 "87948484950,0"
Please suggest for the better/permanent solution on it also confirm if anymore configuration is required on Awstats box .
Thanks.!

Corrupted record line 3114611 (record format does not match LogFormat parameter): 192.168.1.11- - [28/May/2009:06:51:33 -0600] "GET /portal/page/portal/IMP_Main/IMP_AssetsLibrary/logout_button.jpg 172.17.0.1 - - [28/May/2009:06:51:36 -0600] "GET / HTTP/1.0" 200 370 "87948484950,0"

Select allOpen in new window

ok ,now that corrupted log I have commented ,but not sure why two line came in a single log line..!
any  more idea..!

any suggestion in Awstats..!!

If your seeing two log entries on the same line, that's suggestive of corruption.
If in the actual file, then this would be a bug in Apache,  a fairly common thing.

You might check your Apache config to ensure
BufferedLogs

is disabled.    There is a huge performance advantage of having buffered logs, however,   periodic anomolies are normal with that feature.

And check that you are running the current recommended release of Apache httpd, that would be version  2.2.11.

Failing that.. there is a possibility of placing an additional  \n  at the end of the logformat in Apache


If you  see such lines only in Awstats,  that's not expected behavior either, and indicates possible impact of some bug in awstats.

ok, can you pls. suggest about the Keywords "TRACE","CONNCET",GNUTELLA,SEARCH because all the lines having these keyword are treated as dropped records  by Awstats Process.
How Can we resolve this..?
For example pls. check the attached output, i got from awstats process.(showing 3 dropped records)
I am also trying to filter out the local ip not to include in Awstats report but no success..?

[webadmin@xyz01 cgi-bin]$ ./awstats.pl -config=www.mytrust.com -update -showdropped
Create/Update database for config "/etc/awstats/awstats.www.mytrust.com.conf" by AWStats version 6.9 (build 1.925)
From data in log file "/usr/local/awstats/tools/logresolvemerge.pl /aw_transfer/data/www.mytrust.com/*/aw_transfer/data/www.mytrust.com/*|"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Dropped record (method/protocol 'TRACE' not qualified when LogType=W): 207.188.34.66 - - [10/Jun/2009:10:20:58 -0600] "TRACE / HTTP/1.0" 403 313 "-"
Dropped record (method/protocol 'TRACE' not qualified when LogType=W): 207.188.34.66 - - [10/Jun/2009:12:07:31 -0600] "TRACE / HTTP/1.0" 403 313 "-"
Dropped record (method/protocol 'CONNECT' not qualified when LogType=W): 65.117.241.100 - - [12/Jun/2009:12:51:35 -0600] "CONNECT http://lti-mail01.ltinetworks.com:25 HTTP/1.0" 400 260 "72057713400389940,0"
Jumped lines in file: 0
Parsed lines in file: 3538562
 Found 3 dropped records,
 Found 0 corrupted records,
 Found 3538559 old records,
 Found 0 new qualified records.

Select allOpen in new window

any idea..!

Any suggestion please..!

Kindly let me know, if I can avoid dropped records after putting ips /host in skipped host, because all the skipped ips are coming under dropped records..?

Thanks for all your help and efforts on It.!