Discovering proxies?

Hi thanks, but that's not helpful.

There are individuals/groups who are, at the minimum, taxing our servers by initiating an unusually large number of requests.  They happen to be requests for pages which contain data which our company owns, and it would definetly be illegal to redistribute.  We have tracked down at least one company doing that.
The more intelligent individuals/groups that do this choose to use publically available open HTTP proxies to tunnel their requests.  This way, when if/we block them they can switch to another proxy, therefore another IP.

If I can determine quickly and easily that at a particular IP, on a common port, an HTTP proxy that is open to the world, meaning anyone can access and use it without any kind of password, AND I see a significant number of suspect requests from that IP, it is a strong mark towards blocking that IP.

I have asked for a tool to aid me in making that decision, and I need no other information at this time.

And, by the way, yes we do: "You agree not to reproduce, duplicate, copy, sell, resell or...".  But as should have been clear from the original question, when someone is tunneling through a publically accessible proxy, there is little to do to, legally, for recourse.  All we can do is put effort into keeping them from getting the data to begin with.

Why don't you block access to valueable data?

Add a free registration to the site and make sensible parts unaccessible for the anonymous user. Switch to SSL (https://) this should increase security too.

If you really want to play with a portscanner use nmap.

This is one of the best:

http://www.insecure.org/nmap/

Nmap ("Network Mapper") is a free open source utility for network exploration or security auditing. It was designed to rapidly scan large networks, although it works fine against single hosts. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. Nmap runs on most types of computers and both console and graphical versions are available. Nmap is free software, available with full source code under the terms of the GNU GPL.

---
But keep in mind, you can knock off a customer, nmap can be quite brutal to slow targets.

Tolomir



 

I am aware of all these things.  I am fully cognizant of the various issues and solutions.  Regarding your specific suggestion, Tolomir, what if Google required you to register for access to get search results?  Or used HTTPS for all search results?  Idiotic, no?  It's perfectly reasonable for there to be data that is intended to be distributed to the public, and should be easily available.  But, when a single individual/group attempts to access all, or a significant portion, of the data, the owner of said data may not want that individual/group to do so.  That individual/group is likely taking the data to sell, thus hurting the owner's business.  At the very least, that individual/group is placing an unusual and undesirable stress on the web servers providing that content, which is a cost few wish to bear.

I want a piece of software as described in my original post.  Any other comment is irrelevant.  Please do not post alternative ideas or solutions.  I want a simple way to determine "This IP is/is not running an open HTTP proxy."  Anything else does not answer the question.  Please do not waste your effort if you don't have an answer to the question I asked.

If you do not specifically say that somebody is NOT allowed to access and resell the data, they are free to do it and they are not breaking any law.  It is not illegal for me to copy and distribute non-copyrighted data.  I am not sure, but I also do not believe that it is illegal for me to sell non-copyrighted data.
 
Google does NOT own the data and therefore has no legal control over the results of their search.  The sites that show up in their results do own the data and they are in control of what they will and will not allow.

Please just remember that by attempting to probe another site and see what they may or may not have on their server could be just as, if not more, illegal than what you are accusing them of (assuming that there is no posted copyright on the site they are scraping data from).
 
If you feel that somebody is breaking the law, contact local law enforcement.  Let them figure everything out.

Hmm...seems more like a admin issue rather a technical problem....in any case when the information is open for public i am afriad you dont have a complete control on it...if you want to block a specific user accessing your data you can block him..but there are tons of ways to get the data when its not copyrighted or restricted...
to know weather is there any proxy running on a particular ip address, this is also not very easy to do that..you need to use nmap to scan on the ports 80 or 8080 or any common ports to find the proxy then you need to block it...but if you see right now there will be about few hundred thousands of open proxys running all over the world, n tons of new boxes bein added every day..can you think of blocking all of them ?

Just to give you an idea:

http://www.atomintersoft.com/products/alive-proxy/proxy-list/

Free proxy list

(powered by AiS Alive Proxy)

451130 proxy servers in database
268060 Anonymous
183070 Transparent

An nmap scan takes about 1 minute (short scan).

so you can scan 1440 per day, lets say you use 10 computers: 14400 per day

Wow, you are done in about 1 month.

Bad enough, this list changes daily, soo you really should work faster...

Btw. this was the first enty in google, when I searched for anonymous proxies.

http://www.google.com/search?sourceid=mozclient&ie=utf-8&oe=utf-8&q=anonymous+proxies

There are a bit more proxies around.


Tolomir

You might be able to try a different approch:

Typically proxies run on port 80 too.
Whenever your firewall gets the port 80 request (SYN), you can
do reverse HTTP connection i.e. send TCP connect to source IP. If it
succeeds, it can be assumed that, the request came from proxy and
your firewall can log a message to the administartor or possibly
block the request. This scheme does not work, if client machine requires
HTTP server.

You may check http headers such as x-refferer, forwarded-by,... and
other headers added by proxies. Unfortunatly, every proxy add its own
header and may even not add headers at all. Such a filter would leave
you with some false negatives (you'll not catch hackers).

An other alternative would be to check there is only one agent type per
ip address. As a single user may use netscape and ie on the same
computer, you'll get a high false positive rate. Additionaly your
customer's proxy or personal firewall may hide agent type or any other
http header, thus, high false negative rate.

You could also check for the source port. NT (and many others) are using
a low sourceport, between 1024 and 2-3000 (considering the workstation
is shutdown every day). There is a chance proxy servers are not rebooted
so often, so blocking proxy access from ports higher than 4000 or 5000
would block proxies... and unix workstations. Has for the previous
solution, high false and negative rates.

You could also scan the client's ip for an open 1080,8080,8000 or 80
port during the first proxy access and discard this address for a while
if it answers something. You'll get high negative rates as (personal)
firewalls would block such connections. You should check you contracts
allows you to scan your customers

As far as I can imagine, there are no efficient way of blocking proxy
chaining. I guess you want to avoid companies with 10 or 20 pcs using a
low-end internet connection instead of purchasing a business access.
Most (personal guess) of these companies have low IT knowledge and the
first solution should fit your need.

Taken from

http://seclists.org/lists/firewall-wizards/2002/May/0011.html

Tolomir

What you may want to look is to see if there are any IPS/IDS type systems that when they detect an "abnormally high amount of reqeust" or "requests that are coming in faster than normal" from the same host, they will start injecting FIN to both sides.

I have heard that these do exist.  Of course you need to difine "faster that normal" and "abnormally high".

I am simply an IT personell, I have no power to make decisions about these things.  I have no authority to implement a system that automatically takes any sort of action.

I have asked, repeatedly, for one particular thing.  A thing that makes it easier for me to manually investigate the situation.  You've all taken wildly diverging ideas, and none are helpful, so I have requested that this thread be deleted.  Unless you plan on answering my actual question, please, just stop.

Ok, short version:

You asked for: I would like a piece of software (preferably open source, freeware if not when possible) that, given an IP address, will test for any open web proxy running on that system.

I told you to check with nmap. Additionally I pointed out, that scanning each computer for an open proxy, connecting to your site would take too long.

Then I gave you the hint to check for http headers such as x-referer, forwarded-by,... and other headers added by proxies.

I simply don't understand why you say, we don't answer your question. If you want more details about some suggestion, you can ask so, but I will not invest my time in a solution you will potentially reject at once.

There are question, that are simple to answer and question that need some research, other question/problems have no clean, proper, fast solution, and this here is definitivly one of the last kind..

just my 2 Cents,
Tolomir

arantius-

Have you considered setting up snort from http://www.snort.org/ and adding a web based front end such as http://sourceforge.net/projects/secureideas/ to id nefarious traffic for you? So you can spend less time on analysis and more on response?

If you cannot put it onto your webserver you could set up a spanned port off of your switch and have a dedicated monitoring box.

HTH

-t

You may want to try SuperScan, which when scanning port will actually attempt to connect and show you the response, but again, it can’t detect if the port is a HTTP proxy or even a open HTTP proxy.

You have been given solutions, nmap.

You have been given options to search for IPS/IDS systems.

There is NO magic software that will do what you want.  Part of the reason is that scanning for http proxies can take forever and will change from hour to hour.  If it was that simple and easy, don't you think that somebody would have come up with something that you could easily go to Google and find?

You have also been given opinions that the way you are going about this is (in our opinions) the wrong way and possibility illegal.

Forcing somebody to signup to gain access to copyrighted information is NOT bad.  Did you not have to signup for EE?

I don't have to signup for Google, but Google is not providing me with copyrighted information, they are only providing me with a link to somebody else site that may or may not be copyrighted.  Google legally can NOT prevent me from taking data from a Web site they found and doing anything I want with it, only the owner of the web site can.

Even if you found out that is was, as others have posted, there are some out there that are setup for public access to allow me to hide.  What are you going to do block assess from all of them?

If you do not want to force users to signup and you are unwilling to use IDS/IPS, there is no a whole lot you can do.

Somebody keeps breaking into your house.  You are unwilling to put locks on your door, you are unwilling install cameras, and you are unwilling to call the police.  You want to go to everybody’s house and see if they are harboring, or could be harboring, somebody that did, or may, break into your house.