Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Analyze Web Traffic generated by workstations

Posted on 2011-10-19
Medium Priority
Last Modified: 2012-05-12
Hello all

i have to analyze the web sites that employees are acessing and sort them in order to block the ones that are not work related. The current network configuration is that i am using a RedHat 8 Linux Gateway with iptables, no proxy server. I have seen snort, iptraf and argus programs, also i have them installed and captured traffic but i'm having a hard time to make use of this traffic they are capturing. Any help would be great to a quick solution in having this report done.

I mention that this monitoring would be used for a week or two in order to capture the data needed to make my report.

Question by:Cosmin Curticapean
  • 4
  • 3
  • 2

Expert Comment

ID: 36991910

You need to implement a Web Proxy such as squid to act as middle-man in order to track the websites being visited by employees in your organization. Since you are already using implementing SQUID would not be a difficult task. You would find a lot of information on internet on setting up transparent proxy, but here is the catch "you can't use transparent proxy for SSL traffic", so any websites viewed by using HTTPS protocol will be missed.

So, assuming you have a Active Directory in place, with all machines connected to Domain; you can perform following steps:

1. Configure a website IIS Server as wpad.<your domain name> and host your organization specific Proxy.pac and wpad.dat
2. Configure group policy to enable Automatic Configuration, and with information on automatic configuration script
3. Disable transparent NAT / access to internet.

This would force all traffic through your proxy and you would be able to prepare a report on Internet Access by IP Address.

But, next question would be are you using DHCP Server for IP addresses? If yes your report would not correctly map to respective users. There is different method to achieve that... But then it would make sense to talk about it only if it pertains to you.

Hope this helps!

LVL 19

Expert Comment

ID: 36994430
If you can't/won't push your users through a proxy, there is another quick and dirty way to at least get the IP addresses of remote sites that your users are visiting but you'll then have to do some kind of post processing to resolve those to names.  Still, at least its a no-brainer:

Just add the following somewhere in your iptables config (probably /etc/sysconfig/iptables):

-A OUTPUT -s -p tcp -m tcp --dport 80 -j LOG --log-level 4 --log-prefix "http "
-A OUTPUT -s -p tcp -m tcp --dport 443 -j LOG --log-level 4 --log-prefix "https "

If you use the log-level (4) I provided above, then you'll need an entry in /etc/syslog.conf to catch these:

# Log outbound http/https requests
kern.warning                                                                 /var/log/outbound_web.log

Restart both daemons (/etc/init.d/iptables restart; /etc/init.d/syslog restart)

Not ideal, but its a whole lot easier than trying to inspect gigs of packet capture logs.  If you want actual URLs, you're probably going to have to use tcpdump in non-interactive fashion, and then use ethereal to parse the logs for you.

Best of luck!


Author Comment

by:Cosmin Curticapean
ID: 37005295
In reply to pritamdutt:

squid is interesting as an option and if it works with ssl it's even better. I'm not sure about this, i need to document it. In my case i need to implement a proxy server (not transparent one), unfortunatelly the DC installed does not cover all PC's and my Linux box covers the firewall, gateway services.
I also have DHCP server, but it does not matter since i want a list of visited websites not the users who accesses them. Anyway the DHCP is configured to assign the same ip address to the mac address so it's even easier.

In reply to xterm:

this sounds like a more easy and quick solution at first look but the line you wrote to add to iptables does not work. Nothing gets added to the outbound_web log file. After playing a lot with iptables, i finally made this line write lines to the log file:

 iptables -t nat -A PREROUTING -p tcp --dport 80 -m state --state NEW -j LOG --log-level 4

unfortunatelly the messages log file gets the same output, so the data get's doubled. This after all would not be such an inconvenience, since this analisys takes a week maby.
My log file has this kind of lines:

Oct 21 11:46:07 HOSTNAME kernel: IN=eth1 OUT= MAC=x0:1x:x7:1x:x6:xa:x0:x4:x1:x6:8x:6x:0x:x0 SRC=192.16x.x.xxx DST=2xx.1xx.x6x.2 LEN=60 TOS=0x00 PREC=0x00

and i'm not sure that i can use the DST IP to trace the web page address.

Evaluating UTMs? Here's what you need to know!

Evaluating a UTM appliance and vendor can prove to be an overwhelming exercise.  How can you make sure that you're getting the security that your organization needs without breaking the bank? Check out our UTM Buyer's Guide for more information on what you should be looking for!


Accepted Solution

pritamdutt earned 450 total points
ID: 37005398
Hi Cosmin,

So, I understand you don't have all the machines on Domain, no worries! I am going to give a seamless way to implement proxy and ensure all traffic passes through proxy even SSL one....

Internet Explorer by default has Automatically Detect Settings enabled.

You are only required to do the following:

1. Configure your DHCP Server by creating a WPAD entry.
      Steps for Windows DHCP Server are given below:
      1. Log on to the Management Server by using an account that is a member of the Domain Admins group.
      2. Click Start, point to Administrative Tools, and then click DHCP.
      3. Expand the name of the Management Server, right-click IPv4, and then click Set Predefined Options.
      4. In the Predefined Options and Values dialog box, click Add.
      5. In the Option Type dialog box, do the following:
            1. In Name, type WPAD.
            2. In Code, type 252.
            3. In Data type, select String, and then click OK.
            4. In String, type http://<ServerName>:<PortNumber>/wpad.dat, where ServerName is the fully qualified domain name of your Web Server, and PortNumber is the port on which WPAD information is published, and then click OK.
      Note: Make sure that you use lowercase letters to type wpad.dat.
      6. In the console tree, expand the DHCP scope for which you want to configure WPAD, right-click Scope Options, and then click Configure Options.
      7. Click Advanced, and then in Vendor Class, click Standard Options.
      8. In Available Options, select 252 WPAD, and then click OK.

2. Configure your DHCP Server by creating a DNS Suffix Entry .
      Steps for Windows DHCP Server are given below:
      1. Log on to the Management Server by using an account that is a member of the Domain Admins group.
      2. Click Start, point to Administrative Tools, and then click DHCP.
      3. Expand the name of the Management Server, expand IPv4, right-click on Server Options, and then click on Configure Options...
      4. In the Server Options dialog box, scroll down for 015 DNS Domain Name, click the check box and a string value defining your local domains name such as Cosmin.local. The click OK.

3. Enable Automatic detection on DNS Server: To enable automatic detection of browser settings on DNS, you need to configure either the A record or CNAME "alias" record in the DNS database file
To configure a DNS database file for automatic detection of browser settings, in the DNS database file, enter a host record named wpad that points to the IP address of the Web server that contains the .pac, .jvs, .js, or .ins automatic configuration file. After the record is added and the database file is propagated to the server, the DNS name wpad.Cosmin.local should resolve to the same computer name as your server that contains the automatic configuration file.

4. Configure your WebBrowser to add wpad.dat, wpad.pac files. Essentially both are same files, its just that Firefox looks for .pac where as IE looks for .dat.

the contents of WPAD.DAT/WPAD.PAC could be as follows:

function FindProxyForURL(url, host) {

// If URL has no dots in host name, send traffic direct.
	if (isPlainHostName(host))
		return "DIRECT";

// All other traffic uses below proxies, in fail-over order.
	return "PROXY ProxyIP:ProxyPort; DIRECT";

Open in new window


How will these changes impact?

Now this how the process will go, once the changes are propagated on the network.
When user starts his/her browser,  the browser will send a query to download autoproxy config file http://wpad.Cosmin.local/wpad.dat and apply proxy configuration as specified in there.

You can verify that browsers are looking for this file by checking your webserver's log.

Hope this helps!


Expert Comment

ID: 37005405
You can read more about Proxy AutoConfig @ http://en.wikipedia.org/wiki/Proxy_auto-config

LVL 19

Assisted Solution

xterm earned 150 total points
ID: 37006195

To de-duplicate the iptables logging in /var/log/messages simply add to your line in syslog.conf for /var/log/messages a null for kern.warning, for example:

# standard syslog
*.info;mail.none;authpriv.none;cron.none;kern.!=warning            /var/log/messages

But you're right, this solution is only easy, not ideal.  A squid proxy would give you actual Apache-style logging which you could run webalizer or other easy web log analysis tools on.

Author Comment

by:Cosmin Curticapean
ID: 37029373
Sorry for the delay, the season's flu got me for a few days. My progress is in this direction:
-installed new version of squid (update actually)
-installed sarg, an application that uses the log files of squid to analyze and display a report per user (ip in my case), downloads and top sites. It helps me alot in my quest!

Now follows the transparent proxy (tutorial of pritamdutt) since now squid is configured by default to work with port 3128. Updates this afternoon!

Regards and thanks,
Cosmin C.

Author Comment

by:Cosmin Curticapean
ID: 37036678
In the end my working solution is this, having in mind that my DHCP server si on linux not on the domain server (microsoft).
1. created a file wpad.dat with content on one web server that can be accessed from anywhere (Public IP):
function FindProxyForURL(url, host) { return "PROXY; DIRECT"; }

Open in new window

2. created a policy in the domain controller for the automatic proxy detection and they appear in IE as follows:
IE configs from local policies
Did some tests on a workstation in local lan i can see that Internet traffic is going through my proxy server. If i have the workstation disconnected from the lan and access Internet the proxy server is bypassed.

Finally i will have to change the policies for firefox since it does not support GPO natively.

Cosmin C.

Author Closing Comment

by:Cosmin Curticapean
ID: 37036683
Thank you and keep up the good work in supporting others!

Featured Post


Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
​Being a Managed Services Provider (MSP) has presented you  with challenges in the past— and by meeting those challenges you’ve reaped the rewards of success.  In 2014, challenges and rewards remain; but as the Internet and business environment evol…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses
Course of the Month10 days, 14 hours left to enroll

571 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question