how to analyze server log?

I have long list of log file.
For example,
123.1.123.333 - - [27/May/2002:21:23:08 -0400] "GET /~alpha/dbman/html.pl HTTP/1.0" 200 44424

First, what are 200 and 44424?
Second, with this list of ip address, how can I anlalyze this file without using any analysis tool on the web?
Do I have to parse the log file and find any pattern?
Third, if I do, what program language do I have to use? I know java and php.
Please give me idea..thank you

horizzangAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Tacobell777Commented:
200 is the http status code, which means OK
see http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

If I remember correctly, the last entry is the bytes transfered.

You can download log analyzing software from here http://www.mrunix.net/webalizer/download.html

You don't need to learn any programming language, when you install the software and parse the logs it will show you graphs which will make sense to you.
0
horizzangAuthor Commented:
Thank you, Tacobell777
I already knew many analyzing software.
I have to use programming langague but I have no idea how to start.
0
Tacobell777Commented:
Allright!

I would start with writing some function that imports the data into a database, when it is in a database it is easier to manage, quicker to work with and really the only way to report on.

It depends on what rdbms you work with, if it's MS SQL then you are sound as you can write a DTS package that performs the import on a regular basis. Not much programming required there yet.

If it's Access you work with, I'm sure it too has some feature that can import the log files for you into the db.

Some people insert a log entry into the database when a user accesses a website, but this is not the way to go, it puts extra strain on your application/sites. A import with a 5 or 10 minute interval is THE way to go.

The part where the programming comes in is when you need to write the reports and display them. But to go into more detail there it would take a day to write HOW TO's. This is where your programming creativity needs to kick in ;-))

I wrote some code to run reports on logs, your welcome to look at it and get some ideas from it, but it's old and I used MS SQL and ColdFusion.

0
Cloud Class® Course: Microsoft Windows 7 Basic

This introductory course to Windows 7 environment will teach you about working with the Windows operating system. You will learn about basic functions including start menu; the desktop; managing files, folders, and libraries.

Tacobell777Commented:
Might I just add that by looking at your webserver's log format you can tell exactly what each entry in your log is, not knowing your log format I assumed the last entry was the bytes transfered.
0
horizzangAuthor Commented:
Thank you Tacobell777,

Replicating what you said,

I have to start with importing the data into a database. I don't know how but I will figour out. I used MySQL before and probably it has feature to import.

Then, what programming lauange is good for log analysis? I know PHP and java(both beginning level).

Where Can I look at your codes to get some idea?

Thank you so much again for your help.
0
Tacobell777Commented:
It really does not matter what programming language you would use, as long as you get the concept of log analysis.

My code would not do you any good if Java and PHP is what you know.

One thing you got to keep in mind with reporting, you will be working with hundreds of thousands of records after a year, so when you import you need to figure out some way to create summary, i.e. update statistics while importing.

Example
If you import the log file into one flat table and after one year you want to see how many bytes were transfered for example, you would need to report on all those records, the best thing to do is keep a summary, i.e. you import the log and while importing with each row you update your summary table, for example this table is about bytes transfered, then you would do the following on each row import UPDATE tblBytesTransfer SET byteTransfer = byteTransfer + valueOfOutCurrentRow

Hope that makes sense, I can't give much more hints or pointers otherwise I'd be writing a book here.

Following might give you an idea of what I mean by looking at the table layout

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[lkpClientToServerUserAgent]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[lkpClientToServerUserAgent]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentDay]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentDay]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentMonth]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentMonth]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentWeek]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentWeek]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentYear]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentYear]
GO

CREATE TABLE [dbo].[lkpClientToServerUserAgent] (
      [pkIDUserAgent] [int] IDENTITY (1, 1) NOT NULL ,
      [userAgentName] [varchar] (200) COLLATE Latin1_General_CI_AS NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentDay] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentMonth] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentWeek] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentYear] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO



0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
fz2hqsCommented:
For the cost of the SQL Server License you could better go and get a dedicated statistics package. Download some evaluation versions and you will see that they are very sophisticated, for you to be able to identify sessions within the log files will be horribly complex - why reinvent the wheel. WHen you consider your hourly cost, (provided you are not doing this for a hobby) you will be much better just buying something.
0
Tacobell777Commented:
I Quote "I already knew many analyzing software. I have to use programming langague but I have no idea how to start."
0
fz2hqsCommented:
The point I was trying to make was that some people seem to think that it is always better and cheaper to do things yourself, when in truth - especially here - a bespoke applicaiton will not make financial sense nor produce better results. The line you quote says "I have no idea how to start" - that itself describes where we are on the learning curve, the fact that horizzang admits to only being a beginner at programming would only go to enforce the fact that this could take a very long project for him/her.

Even if we talk about a base version of live stats, which elsewhere we have agreed is well worth the money, that is $700 which when you take into account basic wage, tax, floor space and every other cost for an employee that is at best two weeks wages - can you honestly say that it is not worth atleast encouraging horizzang speaking to their manager and suggesting that this project could be somewhat of a false economy

It may be that the project manager has tasked horizzang with this as some sort of training exercise to learn the languages then fine, however personally if I was tasked with this, then I would look for the easy out and in this case a win-win by  buying in the product

Stuart
0
Tacobell777Commented:
Agreed.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Servers

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.