<

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x

Looking at the HTML source code doesn't show all the downloaded files, a proxy does

Published on
10,193 Points
3,693 Views
Last Modified:
Approved
When pages do not download correctly, and you don't know why, the first thing you do is to look at the HTML source code of that page, but not all the downloaded files appear always clearly. If your source includes a javascript that computes the name of the downloaded file, sometimes, you will find it very hard and even impossible to know which file was supposed to be downloaded. Some websites aim to make it cryptic for preventing you to download a part of their stuff from outside of a browser.

Happily, there is a trick to know every single file you download: if you have have access to the log files of a http proxy server, it will possible for you to know exactly which files have been downloaded.

Normally, you won't have access to such a proxy server, but it is very easy to configure your own logging proxy server.
In this article, I will configure the squid proxy server under GNU/Linux but Windows users can adapt the stuff here for their case, since squid is open source software and runs also on MS platforms.

Installing squid

Installing squid under Ubuntu is as easy as opening a terminal (Applications > Accessories > Terminal) and issuing
sudo aptitude install squid

Open in new window

You will have to provide your password.

Configuring squid

Now, the trick is to configure squid for running in user mode, not as root. Create a directory in your home directory. My choice was to create the directory squid inside of a my home directory. Edit a file called squid.conf in that directory by opening a terminal and typing:
mkdir ~/squid
cd ~/squid
gedit squid.conf

Open in new window


In that file, you have to provide the content below:
acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/32
http_access allow localhost 
http_access deny all
http_port 3128
cache_log /home/_myname_/squid/cache.log
cache_store_log none
pid_filename /home/_myname_/squid/pid
cache deny all
cache_dir null /tmp
logformat custom %mt %<st %ru
access_log /home/_myname_/squid/access.log custom

Open in new window

Attention: where you see _myname_ in this configuration file, you have to provide the name of your home directory.
This will make squid listen to port 3128 of the local host and write the file name of all the files that pass through the proxy server into the log file /home/_myname_/squid/access.log. We will only log the MIME type, the size and the URL of the file.

Starting and stopping squid

For starting the proxy server, you have to issue squid -f squid.conf -N -d 1 in the directory ~/squid. For stopping, you just interrupt the proxy by hitting control-C in the same terminal where you launched squid. You can also stop it by issuing squid -k shutdown -f ~/squid/squid.conf in any other terminal.
When something goes wrong, the logfile cache.log will give you more information. In some cases, you will need to stop squid by killing it explicitly with its PID. If you don't understand how to do this, just issue:
sudo killall squid

Open in new window


Enable your browser to pass through the proxy server

For logging the activity of your browser, you now need to tell your browser to use your local proxy, by setting 127.0.0.1 as IP address of the proxy and 3128 as the port.

Start surfing

Once the proxy is running and your browser point to it, you will see a lot of lines appearing into the access.log file, with the MIME type of the file, its size and its complete URL. Now you can scan that file to find the information you need. Enjoy.
0
Comment
0 Comments

Featured Post

Amazon Web Services

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Join & Write a Comment

This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
The viewer will learn how to count occurrences of each item in an array.

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month