Solved

how to analyze server log?

Posted on 2003-11-07
10
257 Views
Last Modified: 2010-04-20
I have long list of log file.
For example,
123.1.123.333 - - [27/May/2002:21:23:08 -0400] "GET /~alpha/dbman/html.pl HTTP/1.0" 200 44424

First, what are 200 and 44424?
Second, with this list of ip address, how can I anlalyze this file without using any analysis tool on the web?
Do I have to parse the log file and find any pattern?
Third, if I do, what program language do I have to use? I know java and php.
Please give me idea..thank you

0
Comment
Question by:horizzang
  • 6
  • 2
  • 2
10 Comments
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9703919
200 is the http status code, which means OK
see http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

If I remember correctly, the last entry is the bytes transfered.

You can download log analyzing software from here http://www.mrunix.net/webalizer/download.html

You don't need to learn any programming language, when you install the software and parse the logs it will show you graphs which will make sense to you.
0
 

Author Comment

by:horizzang
ID: 9704152
Thank you, Tacobell777
I already knew many analyzing software.
I have to use programming langague but I have no idea how to start.
0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9704296
Allright!

I would start with writing some function that imports the data into a database, when it is in a database it is easier to manage, quicker to work with and really the only way to report on.

It depends on what rdbms you work with, if it's MS SQL then you are sound as you can write a DTS package that performs the import on a regular basis. Not much programming required there yet.

If it's Access you work with, I'm sure it too has some feature that can import the log files for you into the db.

Some people insert a log entry into the database when a user accesses a website, but this is not the way to go, it puts extra strain on your application/sites. A import with a 5 or 10 minute interval is THE way to go.

The part where the programming comes in is when you need to write the reports and display them. But to go into more detail there it would take a day to write HOW TO's. This is where your programming creativity needs to kick in ;-))

I wrote some code to run reports on logs, your welcome to look at it and get some ideas from it, but it's old and I used MS SQL and ColdFusion.

0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9704513
Might I just add that by looking at your webserver's log format you can tell exactly what each entry in your log is, not knowing your log format I assumed the last entry was the bytes transfered.
0
 

Author Comment

by:horizzang
ID: 9704519
Thank you Tacobell777,

Replicating what you said,

I have to start with importing the data into a database. I don't know how but I will figour out. I used MySQL before and probably it has feature to import.

Then, what programming lauange is good for log analysis? I know PHP and java(both beginning level).

Where Can I look at your codes to get some idea?

Thank you so much again for your help.
0
Network it in WD Red

There's an industry-leading WD Red drive for every compatible NAS system to help fulfill your data storage needs. With drives up to 8TB, WD Red offers a wide array of solutions for customers looking to build the biggest, best-performing NAS storage solution.  

 
LVL 17

Accepted Solution

by:
Tacobell777 earned 50 total points
ID: 9704761
It really does not matter what programming language you would use, as long as you get the concept of log analysis.

My code would not do you any good if Java and PHP is what you know.

One thing you got to keep in mind with reporting, you will be working with hundreds of thousands of records after a year, so when you import you need to figure out some way to create summary, i.e. update statistics while importing.

Example
If you import the log file into one flat table and after one year you want to see how many bytes were transfered for example, you would need to report on all those records, the best thing to do is keep a summary, i.e. you import the log and while importing with each row you update your summary table, for example this table is about bytes transfered, then you would do the following on each row import UPDATE tblBytesTransfer SET byteTransfer = byteTransfer + valueOfOutCurrentRow

Hope that makes sense, I can't give much more hints or pointers otherwise I'd be writing a book here.

Following might give you an idea of what I mean by looking at the table layout

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[lkpClientToServerUserAgent]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[lkpClientToServerUserAgent]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentDay]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentDay]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentMonth]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentMonth]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentWeek]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentWeek]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentYear]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentYear]
GO

CREATE TABLE [dbo].[lkpClientToServerUserAgent] (
      [pkIDUserAgent] [int] IDENTITY (1, 1) NOT NULL ,
      [userAgentName] [varchar] (200) COLLATE Latin1_General_CI_AS NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentDay] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentMonth] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentWeek] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentYear] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO



0
 
LVL 9

Expert Comment

by:fz2hqs
ID: 9713270
For the cost of the SQL Server License you could better go and get a dedicated statistics package. Download some evaluation versions and you will see that they are very sophisticated, for you to be able to identify sessions within the log files will be horribly complex - why reinvent the wheel. WHen you consider your hourly cost, (provided you are not doing this for a hobby) you will be much better just buying something.
0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9713560
I Quote "I already knew many analyzing software. I have to use programming langague but I have no idea how to start."
0
 
LVL 9

Expert Comment

by:fz2hqs
ID: 9713852
The point I was trying to make was that some people seem to think that it is always better and cheaper to do things yourself, when in truth - especially here - a bespoke applicaiton will not make financial sense nor produce better results. The line you quote says "I have no idea how to start" - that itself describes where we are on the learning curve, the fact that horizzang admits to only being a beginner at programming would only go to enforce the fact that this could take a very long project for him/her.

Even if we talk about a base version of live stats, which elsewhere we have agreed is well worth the money, that is $700 which when you take into account basic wage, tax, floor space and every other cost for an employee that is at best two weeks wages - can you honestly say that it is not worth atleast encouraging horizzang speaking to their manager and suggesting that this project could be somewhat of a false economy

It may be that the project manager has tasked horizzang with this as some sort of training exercise to learn the languages then fine, however personally if I was tasked with this, then I would look for the easy out and in this case a win-win by  buying in the product

Stuart
0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9714202
Agreed.
0

Featured Post

Free Gift Card with Acronis Backup Purchase!

Backup any data in any location: local and remote systems, physical and virtual servers, private and public clouds, Macs and PCs, tablets and mobile devices, & more! For limited time only, buy any Acronis backup products and get a FREE Amazon/Best Buy gift card worth up to $200!

Join & Write a Comment

In our day to day coding, how many times have we come across a necessity to check whether a URL is a broken link or not? For those of you that answered countless and are using ColdFusion like myself, then this article is for you.  It will show yo…
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
This video discusses moving either the default database or any database to a new volume.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now