Solved

Capture HTTP/HTTPS and IP address orgin

Posted on 2014-01-14
13
339 Views
Last Modified: 2014-01-21
I am looking for a simple command line linux utility to have it run on automatically and capture HTTP/HTTPS and the computer IP it originated from.
0
Comment
Question by:georgopanos
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 5
  • 2
  • +1
13 Comments
 
LVL 35

Expert Comment

by:Seth Simmons
ID: 39781097
if you are talking about seeing where traffic is coming from to access your web site, it should be in the logs

if you are running apache, it will appear in the access log
by default it will include the source IP address, date, time, page requested, browser used and result code
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39781103
Do you want to capture the HTTP and HTTPS payload as well?  If so, then use tcpdump, wireshark, ngrep or similar to do a packet capture.   Note that it's a little more involved to capture HTTPS in plain text.
0
 

Author Comment

by:georgopanos
ID: 39781116
I think wireshark is overkill for what I want.

I basically want a simple log file that will list

HTTP/HTTPS - orgin IP address or (computer name)

I was looking for a simple command line utility I can setup on my Rasberry Pi to autorun on boot with a Network tap and capture the above INFORMATION from a network. Just to monitor what is being browsed.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 48

Expert Comment

by:Tintin
ID: 39781229
Why not just do

tail -f /var/log/apache2/access.log
0
 

Author Comment

by:georgopanos
ID: 39781856
I am not running a website, I want to see the websites the computers on my network are going to.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39783962
Ah, makes sense now.

Assuming the traffic goes through your pi, or you have a promiscuous network, then you can do:

tcpdump port 80 or port 443

if that's a little too verbose, you could do

tcpdump port 80 or port 443 | sed "s/\.http.*//"
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39788112
in single line

#tcpdump port 80
0
 

Author Comment

by:georgopanos
ID: 39795131
tintin: this works for my needs but one other question,

I see for example when it runs the computer will hit an identical address 10 times in a row
for example:

11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com

how can I run the tcpdump and do an IF THEN comparison
to cut the above down to look like this 3 lines instead of 15 just to be able to read the log
easier.

11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com

Thanks
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39795946
Depends if you want to read it in realtime or not.

If you capture to a log, you can simply do:

uniq -d logfile
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39795981
otherwise

#sort logfile | uniq -d
0
 

Author Comment

by:georgopanos
ID: 39796882
I would like to sort it in real time if possible so as to display it to a screen
0
 
LVL 48

Accepted Solution

by:
Tintin earned 500 total points
ID: 39798459
Try this

tcpdump -q dst port http or https|awk '{print $1,$5}'|sed "s/\.http.*//"|sed "s/\.[0-9]* / /"|uniq -d


Note that you may need to wait a little while to see any output as it will be buffered.
0
 

Author Closing Comment

by:georgopanos
ID: 39798504
Great! Thank you!!!!
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Fine Tune your automatic Updates for Ubuntu / Debian
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question