asked on

Java App for analyzing webserver log

Hello Experts,

i need to create a Java application to process a simple log file of a web server. The application will read the file (webserver.log) and will try to extract information according to the user’s choices.

i would like for the user of the application to be able to select the type of information he would like to see, for example
by entering number 1, to see number of visits per month.

1.      Number of visits per month
2.      Number of visits per hour
3.      Number of unique visitors based on the IP
4.      Number of visits per page

i would also like for the application to calculate and present the requested information into another log file formatted as:
[date] [time] [action 1-4]

the below is a sample of the webserver log file that i would like to the java application to read from

190.12.1.100 2011-01-01 18:20 test.html
87.111.18.11 2011-01-01 18:30 index.html
128.33.100.1 2011-01-01 19:05 test.html
190.12.1.100 2011-01-05 07:46 test.html
190.12.1.100 2011-01-05 11:00 info.html
128.33.100.1 2011-01-18 12:30 test.html
128.33.100.1 2011-02-01 09:02 test.html
87.111.18.11 2011-02-01 15:07 info.html
190.12.1.100 2011-02-01 17:11 data.html
128.33.100.1 2011-02-22 18:20 test.html
87.111.18.11 2011-02-22 22:00 data.html
190.12.1.100 2011-02-22 23:31 info.html
128.33.100.1 2011-03-01 18:20 test.html
190.12.1.100 2011-03-01 07:00 index.html
190.12.1.100 2011-03-02 08:09 test.html
87.111.18.11 2011-03-02 08:10 index.html
190.12.1.100 2011-03-02 12:12 test.html
190.12.1.100 2011-03-03 13:18 data.html
128.33.100.1 2011-03-03 15:25 test.html
128.33.100.1 2011-03-04 18:30 info.html

any and all help is greatly appreciated!!

for_yan

Well, you probably want to present user
with a choice of date interval - start date - end date,
to ask the user if they are interested in hourly or daily data
and then , create a class,

public class Visit {
String IP address;
java.util.Date date;
String webpage;
}

and then read through your file
ingnoring those records which are outside
of the time interval specified by the user
and accumulate ArrayList of Visit's
Then you may go through that ArrayList
for each visit calculate how many days passed from the start
and use this number as key for HashMap
and arraylist of Visitis within that day or hour
wuould be the values.
When such Hashmap is accumulatesd you
then go through it and calculate whatever you need
from the array of Visits.

You can probably accumulate this hashmap
already on reading of the log.

Something like that should work.

pk24573

ASKER

hello for_yan,
and thanks for you're reply.

the procedure method you describe looks good,
but my problem is in writing the code.

i need help in writing the program.

thanks

for_yan

Let me think about it

pk24573

ASKER

i really appreciate it, thank you!

ASKER CERTIFIED SOLUTION

for_yan

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

rrz

Here is a JSP that does what you wanted it do. To display it the way you asked will take some more work.
I changed hour into a variable called dayHour which is the day of the year combined with the hour of the day. So for example Jan 01 at the 18th hour is displayed as 1-18 and March 02 at the 8th hour is displayed as 61-8

<%@ page import="java.io.*,java.util.*,java.text.SimpleDateFormat" %>
<%
   GregorianCalendar calendar = new GregorianCalendar();
   SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-ddHH:mm");
   HashSet<String> ipSet = new HashSet<String>();
   HashMap<String,Integer> pageMap = new HashMap<String,Integer>();
   HashMap<Integer,Integer> monthMap = new HashMap<Integer,Integer>();
   HashMap<String,Integer> dayHourMap = new HashMap<String,Integer>();
   // textLog.txt is located in web app's root folder
   BufferedReader reader = new BufferedReader(new FileReader(application.getRealPath("/testLog.txt")));
   String line = null;
   int currentMonth = -1;
   int currentDay = -1;
   String currentDayHour = "dummy value";
   while((line = reader.readLine()) != null){
        String[] tokens = line.split(" ");
        if(tokens.length == 4){
              ipSet.add(tokens[0]);
              if(pageMap.containsKey(tokens[3])){
                    pageMap.put(tokens[3], pageMap.get(tokens[3]) + 1);
              } else pageMap.put(tokens[3], 1);
              calendar.setTime(format.parse(tokens[1] + tokens[2]));
              int month = calendar.get(Calendar.MONTH);
              int day = calendar.get(Calendar.DAY_OF_YEAR);
              int hour = calendar.get(Calendar.HOUR_OF_DAY);
              if(month == currentMonth){
                     monthMap.put(month, monthMap.get(month) + 1);
              } else{
                     currentMonth = month;
                     monthMap.put(month, 1);
                    }
              String dayHour = Integer.toString(day) + "-" + Integer.toString(hour);
              if(dayHour.equals(currentDayHour)){
                     dayHourMap.put(dayHour, dayHourMap.get(dayHour) + 1);
              } else{
                     currentDayHour = dayHour;
                     dayHourMap.put(dayHour, 1);
                    }
        }
   }
%>
1. Visits per month: <%=monthMap%><br/>
2. Visits per hour:  <%=dayHourMap%><br/>
3. Number of visitors: <%=ipSet.size()%><br/>
4. Visits per page: <%=pageMap%>

Open in new window

for_yan

This has a basic GUI (see below), and allows monthly, dauily and hourly
output as well as counts number of
visits for pages within the requested interval

import javax.swing.*;
import java.awt.*;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.BufferedReader;
import java.io.FileReader;
import java.text.ParsePosition;
import java.text.SimpleDateFormat;
import java.util.*;


public class ReadingLog extends JFrame implements ActionListener {
    private JTextField txtStart;
    private JLabel lblStart;
    private JTextField txtEnd;
    private JLabel lblEnd;
    private JComboBox lstDaily;
    private JButton Cancel;
    private JButton Submit;
    private JPanel upperPanel;
    private JPanel lowerPanel;
    private JLabel lblReportType;
    private JPanel middlePanel;
          SimpleDateFormat ff = new SimpleDateFormat("MM/dd/yy");
        SimpleDateFormat ff1 = new SimpleDateFormat("yyyy-MM-dd HH:mm");
     SimpleDateFormat ff2 = new SimpleDateFormat("yyyy-MM");
    boolean monthly;

    public ReadingLog(){
        super("Report ACtivities from Log");
        upperPanel = new JPanel();
        lowerPanel = new JPanel();
           middlePanel = new JPanel();
        lblStart = new JLabel("Start date (in mm/dd/yy format): ");
        lblEnd = new JLabel("End date (in mm/dd/yy format): ");
        txtStart = new JTextField(10);
        txtEnd = new JTextField(10);
        Vector v = new Vector();
        v.addElement("monthly");
        v.addElement("daily");
        v.addElement("hourly");
        lblReportType = new JLabel("Option: ");
        lstDaily = new JComboBox(v);
        Cancel = new JButton("Cancel");
        Submit = new JButton("Submit");
        Submit.addActionListener(this);
        Cancel.addActionListener(this);
        lowerPanel.setLayout(new FlowLayout());
         upperPanel.setLayout(new FlowLayout());
         middlePanel.setLayout(new FlowLayout());
        lowerPanel.add(Submit);
               lowerPanel.add(Cancel);
        upperPanel.add(lblStart);
               upperPanel.add(txtStart);
               upperPanel.add(lblEnd);
               upperPanel.add(txtEnd);
            middlePanel.add(lblReportType);
               middlePanel.add(lstDaily);




        Container c = this.getContentPane();
        c.setLayout(new GridLayout(3,1));

                c.add(upperPanel);
                c.add(middlePanel);
        c.add(lowerPanel);
        this.setSize(700,200);
        this.move(300,300);
        this.setVisible(true);
        
  /*
        String startDate = "01/01/11";
         String endDate = "04/01/11";
        String daily = "hourly";


        long msec = 60L*60L*1000L;

        if(daily.equals("daily"))msec *= 24L;

        readLog(startDate, endDate, msec);
        */

    }
public void actionPerformed(ActionEvent ae){
    if(ae.getSource().equals(Cancel)){
        System.exit(0);
    }
     if(ae.getSource().equals(Submit)) {
         String startDate = txtStart.getText().trim();
         String endDate = txtEnd.getText().trim();
         String daily = (String)lstDaily.getSelectedItem();

         boolean goodDates = true;

         try {
             ff.parse(startDate, new ParsePosition(0));
              ff.parse(endDate, new ParsePosition(0));


         } catch(Exception ex1){
             goodDates = false;
         }

         if(!goodDates){
             JOptionPane.showMessageDialog(this, "Please, make sure dates are formatted as shown. Try again");
             return;
         }
               long msec = 60L*60L*1000L;

         if(daily.equals("monthly"))monthly = true;
        if(daily.equals("daily"))msec *= 24L;
         readLog(startDate, endDate, msec);

     }

}

    public void readLog(String startDate, String endDate, long msec) {

      //  SimpleDateFormat ff = new SimpleDateFormat("MM/dd/yy");
      //  SimpleDateFormat ff1 = new SimpleDateFormat("yyyy-MM-dd HH:mm");

        java.util.Date d1 = ff.parse(startDate, new ParsePosition(0));
               java.util.Date d2 = ff.parse(endDate, new ParsePosition(0));


              HashMap m = new HashMap();

             ArrayList year_months = new ArrayList();
          HashMap m1 = new HashMap();
           ArrayList pages1 = new ArrayList();



        try{
        BufferedReader in = new BufferedReader(new FileReader("log.txt"));
               String line = null;
            while((line = in.readLine()) != null){
                if(line.trim().length()== 0)continue;
                StringTokenizer t = new StringTokenizer(line);
                String ip = t.nextToken();
                java.util.Date dd = ff1.parse(t.nextToken() + " " + t.nextToken(), new ParsePosition(0));
                    String page = t.nextToken();
                if(dd.before(d1) || dd.after(d2))continue;
              int  period = (int) ((dd.getTime() - d1.getTime())/msec);
                String year_month = ff2.format(dd);
                if(!year_months.contains(year_month))year_months.add(year_month);

                               if(!pages1.contains(page))pages1.add(page);
                             if(m1.get(page) != null){
                                 Integer ii = (Integer) m1.get(page);
                                 Integer iip1 = new Integer(ii.intValue()+1);
                                 m1.put(page, iip1);
                             }   else
                             {
                                 Integer ii = new Integer(1);
                                 m1.put(page, ii);

                             }

                Visit v = new Visit(ip,dd,page,period);
                Integer ii = new Integer(period);
                        if(!monthly){
                if(m.get(ii) != null){

                    ArrayList al = (ArrayList) m.get(ii);
                    al.add(v);
                    m.put(ii, al);
                       }


                  else
                {
                    ArrayList al = new ArrayList();
                    al.add(v);
                    m.put(ii,al);
              

                }
                        }
                  else
                        {
                     if(m.get(year_month) != null){

                    ArrayList al = (ArrayList) m.get(year_month);
                    al.add(v);
                    m.put(year_month, al);
                       }


                  else
                {
                    ArrayList al = new ArrayList();
                    al.add(v);
                    m.put(year_month,al);


                }
                        }


            }

          in.close();
        }catch(Exception ex){
            System.out.println("Error " + ex.toString());
            ex.printStackTrace();

        }

          int maxPeriod = (int)((d2.getTime()-d1.getTime())/msec) + 1;

        ArrayList ips;
        ArrayList pages;
        int count;
        Collections.sort(year_months);

        if(!monthly){
        System.out.println("Start            End                 #Visits         #IPs           #Pages");

        for(int jj=0; jj<maxPeriod; jj++){
            Integer ii = new Integer(jj);
            if(m.get(ii) == null)continue;
            ArrayList al = (ArrayList) m.get(ii);

             ips = new ArrayList();
              pages = new ArrayList();
            for(int kk=0; kk<al.size(); kk++){
                Visit v = (Visit) al.get(kk);
                 String ip = v.getIp();
                 String page = v.getPage();
                if(!ips.contains(ip))ips.add(ip);
                if(!pages.contains(page))pages.add(page);
            }
          java.util.Date  dleft = new java.util.Date(d1.getTime() + jj*msec);
          java.util.Date dright = new java.util.Date(d1.getTime() + jj*msec + msec);
           String dleftString = ff1.format(dleft);
           String drightString = ff1.format(dright);
            System.out.println(dleftString + "\t" + drightString + "\t" + al.size() + "\t\t" + ips.size() + "\t\t" + pages.size());

        }


        } else
        {

         System.out.println("Year-Month                #Visits         #IPs           #Pages");
            for(int jj=0; jj<year_months.size(); jj++){
                   String year_month = (String) year_months.get(jj);
                   if(m.get(year_month) == null)continue;
                   ArrayList al = (ArrayList) m.get(year_month);
               // System.out.println("al.size: " + al.size());

                    ips = new ArrayList();
                     pages = new ArrayList();
                   for(int kk=0; kk<al.size(); kk++){
                       Visit v = (Visit) al.get(kk);
                        String ip = v.getIp();
                        String page = v.getPage();
                       if(!ips.contains(ip))ips.add(ip);
                       if(!pages.contains(page))pages.add(page);
                   }
               //  java.util.Date  dleft = new java.util.Date(d1.getTime() + jj*msec);
               //  java.util.Date dright = new java.util.Date(d1.getTime() + jj*msec + msec);
               //   String dleftString = ff1.format(dleft);
                //  String drightString = ff1.format(dright);
                   System.out.println(year_month + "\t\t\t" + al.size() + "\t\t" + ips.size() + "\t\t" + pages.size());



        }

        }

       Collections.sort(pages1);

        System.out.println("  ");
        System.out.println("Page\t\tNumber of visits between " + startDate +" and " + endDate);
        for(int jj=0; jj<pages1.size(); jj++){
            String page = (String)pages1.get(jj);
            int freq = ((Integer)m1.get(page)).intValue();

            System.out.println(page + "    \t\t" + freq);


        }




 }
    public static void main(String [] args){
       new ReadingLog();

        
    }
    

}

class Visit{
    String ip_address;
    java.util.Date dd;
    String page;
    int period;

Visit(String ip_address, java.util.Date dd, String page, int period){
    this.ip_address= ip_address;
    this.dd = dd;
    this.page = page;
    this.period = period;
    


}

    String getIp(){
        return ip_address;
    }
    String getPage() {
        return page;
    }





}

Open in new window

Output hourly:

Start            End                 #Visits         #IPs           #Pages
2011-01-01 18:00	2011-01-01 19:00	2		2		2
2011-01-01 19:00	2011-01-01 20:00	1		1		1
2011-01-05 07:00	2011-01-05 08:00	1		1		1
2011-01-05 11:00	2011-01-05 12:00	1		1		1
2011-01-18 12:00	2011-01-18 13:00	1		1		1
2011-02-01 09:00	2011-02-01 10:00	1		1		1
2011-02-01 15:00	2011-02-01 16:00	1		1		1
2011-02-01 17:00	2011-02-01 18:00	1		1		1
2011-02-22 18:00	2011-02-22 19:00	1		1		1
2011-02-22 22:00	2011-02-22 23:00	1		1		1
2011-02-22 23:00	2011-02-23 00:00	1		1		1
2011-03-01 07:00	2011-03-01 08:00	1		1		1
2011-03-01 18:00	2011-03-01 19:00	1		1		1
2011-03-02 08:00	2011-03-02 09:00	2		2		2
2011-03-02 12:00	2011-03-02 13:00	1		1		1
2011-03-03 13:00	2011-03-03 14:00	1		1		1
2011-03-03 15:00	2011-03-03 16:00	1		1		1
2011-03-04 18:00	2011-03-04 19:00	1		1		1
  
Page		Number of visits between 1/1/11 and 4/1/11
data.html    		3
index.html    		3
info.html    		4
test.html    		10

Open in new window

Output monthly:

Year-Month                #Visits         #IPs           #Pages
2011-01			6		3		3
2011-02			6		3		3
2011-03			8		3		4
  
Page		Number of visits between 1/1/11 and 4/1/11
data.html    		3
index.html    		3
info.html    		4
test.html    		10

Open in new window

Analyzing-log.PNG

rrz

I did a little work on mine too. Still needs more work on display. I assumed that any particular log file would report a period of less than one year. This is the current output.

1. Visits per month: {Feb=6, Mar=8, Jan=6}
2. Visits per hour: {Mar 3:13=1, Jan 1:19=1, Jan 1:18=2, Mar 2:12=1, Feb 22:23=1, Mar 3:15=1, Feb 22:22=1, Jan 5:7=1, Jan 5:11=1, Feb 1:17=1, Mar 4:18=1, Feb 1:15=1, Mar 2:8=2, Feb 1:9=1, Jan 18:12=1, Mar 1:18=1, Feb 22:18=1, Mar 1:7=1}
3. Number of visitors: 3
4. Visits per page: {test.html=10, index.html=3, data.html=3, info.html=4}

<%@ page import="java.io.*,java.util.*,java.text.SimpleDateFormat" %>
<%
   GregorianCalendar calendar = new GregorianCalendar();
   SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-ddHH:mm");
   HashSet<String> ipSet = new HashSet<String>();
   HashMap<String,Integer> pageMap = new HashMap<String,Integer>();
   HashMap<String,Integer> monthMap = new HashMap<String,Integer>();
   HashMap<String,Integer> dayHourMap = new HashMap<String,Integer>();
   // textLog.txt is located in web app's root folder
   BufferedReader reader = new BufferedReader(new FileReader(application.getRealPath("/testLog.txt")));
   String line = null;
   String currentMonth = "dummy value";
   int currentDay = -1;
   String currentDayHour = "dummy value";
   while((line = reader.readLine()) != null){
        String[] tokens = line.split(" ");
        if(tokens.length == 4){
              ipSet.add(tokens[0]);
              if(pageMap.containsKey(tokens[3])){
                    pageMap.put(tokens[3], pageMap.get(tokens[3]) + 1);
              } else pageMap.put(tokens[3], 1);
              calendar.setTime(format.parse(tokens[1] + tokens[2]));
              String month = calendar.getDisplayName(Calendar.MONTH,Calendar.SHORT,Locale.getDefault());
              String day = Integer.toString(calendar.get(Calendar.DAY_OF_MONTH));
              String hour = Integer.toString(calendar.get(Calendar.HOUR_OF_DAY));
              if(month.equals(currentMonth)){
                     monthMap.put(month, monthMap.get(month) + 1);
              } else{
                     currentMonth = month;
                     monthMap.put(month, 1);
                    }
              String dayHour = month + " " + day + ":" + hour;
              if(dayHour.equals(currentDayHour)){
                     dayHourMap.put(dayHour, dayHourMap.get(dayHour) + 1);
              } else{
                     currentDayHour = dayHour;
                     dayHourMap.put(dayHour, 1);
                    }
        }
   }
%>
1. Visits per month: <%=monthMap%><br/>
2. Visits per hour:  <%=dayHourMap%><br/>
3. Number of visitors: <%=ipSet.size()%><br/>
4. Visits per page: <%=pageMap%>

Open in new window

pk24573

ASKER

Thanks Mates!!
gonna check it out so i can assign points accordingly!
Thanks Again!