horizzang
asked on
how to analyze server log?
I have long list of log file.
For example,
123.1.123.333 - - [27/May/2002:21:23:08 -0400] "GET /~alpha/dbman/html.pl HTTP/1.0" 200 44424
First, what are 200 and 44424?
Second, with this list of ip address, how can I anlalyze this file without using any analysis tool on the web?
Do I have to parse the log file and find any pattern?
Third, if I do, what program language do I have to use? I know java and php.
Please give me idea..thank you
For example,
123.1.123.333 - - [27/May/2002:21:23:08 -0400] "GET /~alpha/dbman/html.pl HTTP/1.0" 200 44424
First, what are 200 and 44424?
Second, with this list of ip address, how can I anlalyze this file without using any analysis tool on the web?
Do I have to parse the log file and find any pattern?
Third, if I do, what program language do I have to use? I know java and php.
Please give me idea..thank you
ASKER
Thank you, Tacobell777
I already knew many analyzing software.
I have to use programming langague but I have no idea how to start.
I already knew many analyzing software.
I have to use programming langague but I have no idea how to start.
Allright!
I would start with writing some function that imports the data into a database, when it is in a database it is easier to manage, quicker to work with and really the only way to report on.
It depends on what rdbms you work with, if it's MS SQL then you are sound as you can write a DTS package that performs the import on a regular basis. Not much programming required there yet.
If it's Access you work with, I'm sure it too has some feature that can import the log files for you into the db.
Some people insert a log entry into the database when a user accesses a website, but this is not the way to go, it puts extra strain on your application/sites. A import with a 5 or 10 minute interval is THE way to go.
The part where the programming comes in is when you need to write the reports and display them. But to go into more detail there it would take a day to write HOW TO's. This is where your programming creativity needs to kick in ;-))
I wrote some code to run reports on logs, your welcome to look at it and get some ideas from it, but it's old and I used MS SQL and ColdFusion.
I would start with writing some function that imports the data into a database, when it is in a database it is easier to manage, quicker to work with and really the only way to report on.
It depends on what rdbms you work with, if it's MS SQL then you are sound as you can write a DTS package that performs the import on a regular basis. Not much programming required there yet.
If it's Access you work with, I'm sure it too has some feature that can import the log files for you into the db.
Some people insert a log entry into the database when a user accesses a website, but this is not the way to go, it puts extra strain on your application/sites. A import with a 5 or 10 minute interval is THE way to go.
The part where the programming comes in is when you need to write the reports and display them. But to go into more detail there it would take a day to write HOW TO's. This is where your programming creativity needs to kick in ;-))
I wrote some code to run reports on logs, your welcome to look at it and get some ideas from it, but it's old and I used MS SQL and ColdFusion.
Might I just add that by looking at your webserver's log format you can tell exactly what each entry in your log is, not knowing your log format I assumed the last entry was the bytes transfered.
ASKER
Thank you Tacobell777,
Replicating what you said,
I have to start with importing the data into a database. I don't know how but I will figour out. I used MySQL before and probably it has feature to import.
Then, what programming lauange is good for log analysis? I know PHP and java(both beginning level).
Where Can I look at your codes to get some idea?
Thank you so much again for your help.
Replicating what you said,
I have to start with importing the data into a database. I don't know how but I will figour out. I used MySQL before and probably it has feature to import.
Then, what programming lauange is good for log analysis? I know PHP and java(both beginning level).
Where Can I look at your codes to get some idea?
Thank you so much again for your help.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
For the cost of the SQL Server License you could better go and get a dedicated statistics package. Download some evaluation versions and you will see that they are very sophisticated, for you to be able to identify sessions within the log files will be horribly complex - why reinvent the wheel. WHen you consider your hourly cost, (provided you are not doing this for a hobby) you will be much better just buying something.
I Quote "I already knew many analyzing software. I have to use programming langague but I have no idea how to start."
The point I was trying to make was that some people seem to think that it is always better and cheaper to do things yourself, when in truth - especially here - a bespoke applicaiton will not make financial sense nor produce better results. The line you quote says "I have no idea how to start" - that itself describes where we are on the learning curve, the fact that horizzang admits to only being a beginner at programming would only go to enforce the fact that this could take a very long project for him/her.
Even if we talk about a base version of live stats, which elsewhere we have agreed is well worth the money, that is $700 which when you take into account basic wage, tax, floor space and every other cost for an employee that is at best two weeks wages - can you honestly say that it is not worth atleast encouraging horizzang speaking to their manager and suggesting that this project could be somewhat of a false economy
It may be that the project manager has tasked horizzang with this as some sort of training exercise to learn the languages then fine, however personally if I was tasked with this, then I would look for the easy out and in this case a win-win by buying in the product
Stuart
Even if we talk about a base version of live stats, which elsewhere we have agreed is well worth the money, that is $700 which when you take into account basic wage, tax, floor space and every other cost for an employee that is at best two weeks wages - can you honestly say that it is not worth atleast encouraging horizzang speaking to their manager and suggesting that this project could be somewhat of a false economy
It may be that the project manager has tasked horizzang with this as some sort of training exercise to learn the languages then fine, however personally if I was tasked with this, then I would look for the easy out and in this case a win-win by buying in the product
Stuart
Agreed.
see http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
If I remember correctly, the last entry is the bytes transfered.
You can download log analyzing software from here http://www.mrunix.net/webalizer/download.html
You don't need to learn any programming language, when you install the software and parse the logs it will show you graphs which will make sense to you.