Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 267
  • Last Modified:

how to analyze server log?

I have long list of log file.
For example,
123.1.123.333 - - [27/May/2002:21:23:08 -0400] "GET /~alpha/dbman/html.pl HTTP/1.0" 200 44424

First, what are 200 and 44424?
Second, with this list of ip address, how can I anlalyze this file without using any analysis tool on the web?
Do I have to parse the log file and find any pattern?
Third, if I do, what program language do I have to use? I know java and php.
Please give me idea..thank you

0
horizzang
Asked:
horizzang
  • 6
  • 2
  • 2
1 Solution
 
Tacobell777Commented:
200 is the http status code, which means OK
see http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

If I remember correctly, the last entry is the bytes transfered.

You can download log analyzing software from here http://www.mrunix.net/webalizer/download.html

You don't need to learn any programming language, when you install the software and parse the logs it will show you graphs which will make sense to you.
0
 
horizzangAuthor Commented:
Thank you, Tacobell777
I already knew many analyzing software.
I have to use programming langague but I have no idea how to start.
0
 
Tacobell777Commented:
Allright!

I would start with writing some function that imports the data into a database, when it is in a database it is easier to manage, quicker to work with and really the only way to report on.

It depends on what rdbms you work with, if it's MS SQL then you are sound as you can write a DTS package that performs the import on a regular basis. Not much programming required there yet.

If it's Access you work with, I'm sure it too has some feature that can import the log files for you into the db.

Some people insert a log entry into the database when a user accesses a website, but this is not the way to go, it puts extra strain on your application/sites. A import with a 5 or 10 minute interval is THE way to go.

The part where the programming comes in is when you need to write the reports and display them. But to go into more detail there it would take a day to write HOW TO's. This is where your programming creativity needs to kick in ;-))

I wrote some code to run reports on logs, your welcome to look at it and get some ideas from it, but it's old and I used MS SQL and ColdFusion.

0
Become an Android App Developer

Ready to kick start your career in 2018? Learn how to build an Android app in January’s Course of the Month and open the door to new opportunities.

 
Tacobell777Commented:
Might I just add that by looking at your webserver's log format you can tell exactly what each entry in your log is, not knowing your log format I assumed the last entry was the bytes transfered.
0
 
horizzangAuthor Commented:
Thank you Tacobell777,

Replicating what you said,

I have to start with importing the data into a database. I don't know how but I will figour out. I used MySQL before and probably it has feature to import.

Then, what programming lauange is good for log analysis? I know PHP and java(both beginning level).

Where Can I look at your codes to get some idea?

Thank you so much again for your help.
0
 
Tacobell777Commented:
It really does not matter what programming language you would use, as long as you get the concept of log analysis.

My code would not do you any good if Java and PHP is what you know.

One thing you got to keep in mind with reporting, you will be working with hundreds of thousands of records after a year, so when you import you need to figure out some way to create summary, i.e. update statistics while importing.

Example
If you import the log file into one flat table and after one year you want to see how many bytes were transfered for example, you would need to report on all those records, the best thing to do is keep a summary, i.e. you import the log and while importing with each row you update your summary table, for example this table is about bytes transfered, then you would do the following on each row import UPDATE tblBytesTransfer SET byteTransfer = byteTransfer + valueOfOutCurrentRow

Hope that makes sense, I can't give much more hints or pointers otherwise I'd be writing a book here.

Following might give you an idea of what I mean by looking at the table layout

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[lkpClientToServerUserAgent]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[lkpClientToServerUserAgent]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentDay]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentDay]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentMonth]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentMonth]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentWeek]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentWeek]
GO

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[tblClientToServerUserAgentYear]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tblClientToServerUserAgentYear]
GO

CREATE TABLE [dbo].[lkpClientToServerUserAgent] (
      [pkIDUserAgent] [int] IDENTITY (1, 1) NOT NULL ,
      [userAgentName] [varchar] (200) COLLATE Latin1_General_CI_AS NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentDay] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentMonth] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentWeek] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO

CREATE TABLE [dbo].[tblClientToServerUserAgentYear] (
      [fkIDUserAgent] [int] NOT NULL ,
      [dateStamp] [smalldatetime] NOT NULL ,
      [hitCount] [int] NOT NULL
) ON [PRIMARY]
GO



0
 
fz2hqsCommented:
For the cost of the SQL Server License you could better go and get a dedicated statistics package. Download some evaluation versions and you will see that they are very sophisticated, for you to be able to identify sessions within the log files will be horribly complex - why reinvent the wheel. WHen you consider your hourly cost, (provided you are not doing this for a hobby) you will be much better just buying something.
0
 
Tacobell777Commented:
I Quote "I already knew many analyzing software. I have to use programming langague but I have no idea how to start."
0
 
fz2hqsCommented:
The point I was trying to make was that some people seem to think that it is always better and cheaper to do things yourself, when in truth - especially here - a bespoke applicaiton will not make financial sense nor produce better results. The line you quote says "I have no idea how to start" - that itself describes where we are on the learning curve, the fact that horizzang admits to only being a beginner at programming would only go to enforce the fact that this could take a very long project for him/her.

Even if we talk about a base version of live stats, which elsewhere we have agreed is well worth the money, that is $700 which when you take into account basic wage, tax, floor space and every other cost for an employee that is at best two weeks wages - can you honestly say that it is not worth atleast encouraging horizzang speaking to their manager and suggesting that this project could be somewhat of a false economy

It may be that the project manager has tasked horizzang with this as some sort of training exercise to learn the languages then fine, however personally if I was tasked with this, then I would look for the easy out and in this case a win-win by  buying in the product

Stuart
0
 
Tacobell777Commented:
Agreed.
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 6
  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now