?
Solved

how to analyze server log?

Posted on 2003-11-07
10
Medium Priority
?
264 Views
Last Modified: 2010-04-20
I have long list of log file.
For example,
123.1.123.333 - - [27/May/2002:21:23:08 -0400] "GET /~alpha/dbman/html.pl HTTP/1.0" 200 44424

First, what are 200 and 44424?
Second, with this list of ip address, how can I anlalyze this file without using any analysis tool on the web?
Do I have to parse the log file and find any pattern?
Third, if I do, what program language do I have to use? I know java and php.
Please give me idea..thank you

0
Comment
Question by:horizzang
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 2
  • 2
10 Comments
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9703919
200 is the http status code, which means OK
see http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

If I remember correctly, the last entry is the bytes transfered.

You can download log analyzing software from here http://www.mrunix.net/webalizer/download.html

You don't need to learn any programming language, when you install the software and parse the logs it will show you graphs which will make sense to you.
0
 

Author Comment

by:horizzang
ID: 9704152
Thank you, Tacobell777
I already knew many analyzing software.
I have to use programming langague but I have no idea how to start.
0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9704296
Allright!

I would start with writing some function that imports the data into a database, when it is in a database it is easier to manage, quicker to work with and really the only way to report on.

It depends on what rdbms you work with, if it's MS SQL then you are sound as you can write a DTS package that performs the import on a regular basis. Not much programming required there yet.

If it's Access you work with, I'm sure it too has some feature that can import the log files for you into the db.

Some people insert a log entry into the database when a user accesses a website, but this is not the way to go, it puts extra strain on your application/sites. A import with a 5 or 10 minute interval is THE way to go.

The part where the programming comes in is when you need to write the reports and display them. But to go into more detail there it would take a day to write HOW TO's. This is where your programming creativity needs to kick in ;-))

I wrote some code to run reports on logs, your welcome to look at it and get some ideas from it, but it's old and I used MS SQL and ColdFusion.

0
Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

 
LVL 17

Expert Comment

by:Tacobell777
ID: 9704513
Might I just add that by looking at your webserver's log format you can tell exactly what each entry in your log is, not knowing your log format I assumed the last entry was the bytes transfered.
0
 

Author Comment

by:horizzang
ID: 9704519
Thank you Tacobell777,

Replicating what you said,

I have to start with importing the data into a database. I don't know how but I will figour out. I used MySQL before and probably it has feature to import.

Then, what programming lauange is good for log analysis? I know PHP and java(both beginning level).

Where Can I look at your codes to get some idea?

Thank you so much again for your help.
0
 
LVL 17

Accepted Solution

by:
Tacobell777 earned 150 total points
ID: 9704761
It really does not matter what programming language you would use, as long as you get the concept of log analysis.

My code would not do you any good if Java and PHP is what you know.

One thing you got to keep in mind with reporting, you will be working with hundreds of thousands of records after a year, so when you import you need to figure out some way to create summary, i.e. update statistics while importing.

Example
If you import the log file into one flat table and after one year you want to see how many bytes were transfered for example, you would need to report on all those records, the best thing to do is keep a summary, i.e. you import the log and while importing with each row you update your summary table, for example this table is about bytes transfered, then you would do the following on each row import UPDATE tblBytesTransfer SET byteTransfer = byteTransfer + valueOfOutCurrentRow

Hope that makes sense, I can't give much more hints or pointers otherwise I'd be writing a book here.

Following might give you an idea of what I mean by looking at the table layout

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[lkpClientToServerUserAgent]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[lkpClientToServerUserAgent]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentDay]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentDay]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentMonth]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentMonth]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentWeek]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentWeek]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentYear]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentYear]
GO

CREATE TABLE [dbo].[lkpClientToServerUserAgent] (
      [pkIDUserAgent] [int] IDENTITY (1, 1) NOT NULL ,
      [userAgentName] [varchar] (200) COLLATE Latin1_General_CI_AS NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentDay] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentMonth] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentWeek] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentYear] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO



0
 
LVL 9

Expert Comment

by:fz2hqs
ID: 9713270
For the cost of the SQL Server License you could better go and get a dedicated statistics package. Download some evaluation versions and you will see that they are very sophisticated, for you to be able to identify sessions within the log files will be horribly complex - why reinvent the wheel. WHen you consider your hourly cost, (provided you are not doing this for a hobby) you will be much better just buying something.
0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9713560
I Quote "I already knew many analyzing software. I have to use programming langague but I have no idea how to start."
0
 
LVL 9

Expert Comment

by:fz2hqs
ID: 9713852
The point I was trying to make was that some people seem to think that it is always better and cheaper to do things yourself, when in truth - especially here - a bespoke applicaiton will not make financial sense nor produce better results. The line you quote says "I have no idea how to start" - that itself describes where we are on the learning curve, the fact that horizzang admits to only being a beginner at programming would only go to enforce the fact that this could take a very long project for him/her.

Even if we talk about a base version of live stats, which elsewhere we have agreed is well worth the money, that is $700 which when you take into account basic wage, tax, floor space and every other cost for an employee that is at best two weeks wages - can you honestly say that it is not worth atleast encouraging horizzang speaking to their manager and suggesting that this project could be somewhat of a false economy

It may be that the project manager has tasked horizzang with this as some sort of training exercise to learn the languages then fine, however personally if I was tasked with this, then I would look for the easy out and in this case a win-win by  buying in the product

Stuart
0
 
LVL 17

Expert Comment

by:Tacobell777
ID: 9714202
Agreed.
0

Featured Post

Optimum High-Definition Video Viewing and Control

The ATEN VM0404HA 4x4 4K HDMI Matrix Switch supports 4K resolutions of UHD (3840 x 2160) and DCI (4096 x 2160) with refresh rates of 30 Hz (4:4:4) and 60 Hz (4:2:0). It is ideal for applications where the routing of 4K digital signals is required.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In our day to day coding, how many times have we come across a necessity to check whether a URL is a broken link or not? For those of you that answered countless and are using ColdFusion like myself, then this article is for you.  It will show yo…
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
In this video you will find out how to export Office 365 mailboxes using the built in eDiscovery tool. Bear in mind that although this method might be useful in some cases, using PST files as Office 365 backup is troublesome in a long run (more on t…
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question