Monitoring URLs visited

The answer is simple: it's impossible.

If it were possible, then the purpose of HTTPS would be void. You don't want to send an encrypted package knowing that anybody can read it. Attempts to break HTTPS so far have failed and are estimated (last time I looked) to take a fast PC a couple of millennia. No joke, really.

The question itself is on the brink of what EE allows. We are not allowed to help you with decrypting, decyphering or reverse engineering secured connections, protocols or SSL certificates.

-- Abel --

Not exactly true - it depends on where the encryption for the HTTPS originates from at the box or from the proxy server ... if you request an https site from a proxy server it'll be able to log the request and tally the page ...

> if you request an https site from a proxy server it'll be able to log the request and tally the page

true, but then the end-to-end protection required for SSL to work correctly (sender knows receiver) becomes void. It's like sending a secured letter to a whole office: until the office it is safe, then everybody can see it. Normally, the secured letter is supposed to be delivered to one person only.

Most modern proxies can intercept SSL traffic.

abel: If you need to monitor it - you cannot secure it internally. You just have to choose but it's possible without any problem.
BTW there's also possibility of 'repackaging' https on proxy.

Got to love these discussions - everything's readable as far as the computer is concerned -- sometimes it's a matter of permissions, sometimes it's a matter of where the security is placed, etc ... but it's all bits and bytes in the end and can be read by someone with the proper tools to decode the info properly - nothing is safe on computers no matter what anyone tells you, it's a matter of the morals of the people that do have access at the proper level to the data streams as to whether or not your stuff is "secure" or not.

Interesting, as this is against the idea of SSL. HTTPS traffic can be intercepted with a man in the middle attack, which can only succeed if the certificate issued has the special domain "*". The RFC 2818 forbids issuing such certificates. That means you need your own CA. Not that hard, but your browser will (correctly) not recognize them.

In other words: "yes you may listen in" is possible if the client is so nice to help you and allow your "open ended" certificate. This has little to do with security, other then making it a bit harder to intercept.

Interesting link btw. Haven't read it through, but I don't believe that "just placing a proxy" somewhere would allow listening in. That would mean that any ISP, without consent of any client (or issuer) could control your HTTPS traffic. Don't think that's a good thing for security, but if it is true, they'll have to rewrite a whole bunch of books ;-)

-- Abel --

Ah, I see it goes to reading it from the client's computer. Sorry, if that's the case, the story is different. Of course, on the client's computer you can access the certificate so you can decipher the traffic that uses a certain certificate.

If you don't have this access, the best thing you can do is (afaik) inspect the TCP/IP packages to trace the end to end points. That is: the IP, not the full URL.

The goal was not to "listen in" on the traffic for SSL (yes it's possible given the right tools and time and proper physical access to the right spots, etc.) but rather to just know what websites were being visited via HTTPS and a proxy server will serve that purpose ... as the client still has to request DNS from somewhere to get to the site it just doesn;t need to tell everyone what it's saying  -- you just need to know where not what ...

If you need listen to your trafic - you can. It's your choice.
Your first answer was "it's impossible". This is not true at all. Maybe it's insecure maybe bad maybe not very common. But it's perfectly possible and system owner can make such decision if believes it's better for his organization.

Hello TBK-Consulting, the idea is to program this application myself.

Hi abel, of course this is possible and I'm NOT asking for anything ilegal or against EE rules, the idea is to keep track of the websites visited by employees (just the URLS, not the actual contents). Applications like http://www.ematrixsoft.com/website-spy-monitor-software.htm can do it and I think is not that complex.

I'd suggest looking at the squid server code for linux then and getting the ideas for coding the application yourself from that - yay! open source

> Hi abel, of course this is possible and I'm NOT asking for anything ilegal or against EE rules

Ok. In my country it is, unless the user/employee knows about it and has given written consent. But I won't bug you with this illegal/legal discussion (earlier, something similar, I got called back by EE for helping someone to do http sniffing).

> Your first answer was "it's impossible". This is not true at all.

Point taken. I answered from the view of "from anywhere in the network". But the text "where I need to keep track of all visited WebPages" in the original q. does not say from where, so you are absolutely right that in a controlled environment, with the right tools and from the right spot, it is possible. Which again (imho) defies the idea of using SSL. But hey, that's just me: I thought security was for securing your data from others viewing it. lol


Interesting discussion though, learned something today: I always thought that only criminals and governments were interested in reading encrypted data, apparently that group is much larger.

-- Abel --

Corporations have always taken this discussion from the point of view that all data on it's own (internal) networks belongs to them, and NOT to the individual user - this includes email, web traffic, etc ... this is why you hear about people being fired for surfing porn, or for spending too much time on non-work related websites, or too much personal email, etc ... the data simply belongs to them on their networks ... this is in a corporate network, not an ISP-related thing ... ISPs have different standards to uphold, and yes they can do it as well, but only if asked by the feds to do so and with proper court documentation, etc ... but as for corporations the courts have upheld their right to own the data time and time again ...

> but as for corporations the courts have upheld their right to own the data time and time again

though I said I wouldn't take this discussion further, you raise an interesting point. I happen to know lawyers that have dealt with such cases and depending on country, region, local law, internal company rules and regulations and whether or not you have been given something to sign for when you joined the company that says "anything you do in this company can and will be monitored, including your internet traffic, whether secure or not" you may or may not get your right in court.

What I'm trying to say: in general (and in my country: most often) monitoring of internet usage is allowed, monitoring data is not, but indeed: "all you do in the company belongs to the company".

This is a point of conflicting interests: your right of privacy and the right of the company to protect itself against abuse of means and time.

(In The Netherlands, within reason, employees are allowed to use computers of the company for private matters (say: up to about 5-10 min/day) and the company is only allowed to monitor the anonymous traffic and only take action when high traffic occurs or illegal domains are visited and then issue a warning. This privacy at work is protected by Dutch law.)

Like I said: a very interesting point and I'm sure a vibrant discussion in many countries (it sure is here, not all companies like that act).

no not all companies act like that at all ... in the US there are a few that are very aggressive about it and some that are not so aggressive as well ... it is a major privacy issue to some people here and to others it is more about the company's right to their profitability and by monitoring employee access if it offends some but keeps profits up - then so be it .... But you also have issues between husbands and wives that checkup on each other as well, there's so much to the arguments and so much to be said for each side of it as well - personally I just mind my P's and Q's and don't go to far out of the box - but it's probably a good thing I own my own company tho or maybe I should fire myself - I do waste a lot of time on sites like icanhascheeseburger.com !!!

haha, same here. I'd have myself fired long ago if I were me!

So, back on topic. The OP has an excellent option if using the squid box you suggested and expanding from there, or the option of buying and installing a monitoring proxy like ssl inspector as stefanx mentioned. Not sure if other proxies might be worth looking into. Someone suggested that all modern proxies can do that, but I doubt that strongly.

Hi

I have found a library that can do what I need:
http://www.komodia.com/index.php?page=interceptor.html

I think this will work for me.

Thanks everyone.

Is it ok with you if I ask to delete this question?

Deletion is not a good option, because this thread received excellent coverage. If you don't feel that any of the experts have helped or tried to help you in any way and you found the answer yourself (which you did) you can select your own comment as an answer. If nobody objects within four days, the question will automatically be PAQ'ed (archived).

http://www.experts-exchange.com/help.jsp?hi=407

-- Abel --