Solved

Capture HTTP/HTTPS and IP address orgin

Posted on 2014-01-14
13
334 Views
Last Modified: 2014-01-21
I am looking for a simple command line linux utility to have it run on automatically and capture HTTP/HTTPS and the computer IP it originated from.
0
Comment
Question by:georgopanos
  • 5
  • 5
  • 2
  • +1
13 Comments
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39781097
if you are talking about seeing where traffic is coming from to access your web site, it should be in the logs

if you are running apache, it will appear in the access log
by default it will include the source IP address, date, time, page requested, browser used and result code
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39781103
Do you want to capture the HTTP and HTTPS payload as well?  If so, then use tcpdump, wireshark, ngrep or similar to do a packet capture.   Note that it's a little more involved to capture HTTPS in plain text.
0
 

Author Comment

by:georgopanos
ID: 39781116
I think wireshark is overkill for what I want.

I basically want a simple log file that will list

HTTP/HTTPS - orgin IP address or (computer name)

I was looking for a simple command line utility I can setup on my Rasberry Pi to autorun on boot with a Network tap and capture the above INFORMATION from a network. Just to monitor what is being browsed.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39781229
Why not just do

tail -f /var/log/apache2/access.log
0
 

Author Comment

by:georgopanos
ID: 39781856
I am not running a website, I want to see the websites the computers on my network are going to.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39783962
Ah, makes sense now.

Assuming the traffic goes through your pi, or you have a promiscuous network, then you can do:

tcpdump port 80 or port 443

if that's a little too verbose, you could do

tcpdump port 80 or port 443 | sed "s/\.http.*//"
0
Save on storage to protect fatherhood memories

You're the dad who has everything. This Father's Day, make sure your family memories are protected. My Passport Ultra has automatic backup and password protection to keep your cherished photos and videos safe. With up to 3TB, you have plenty of room to hold the adventures ahead.

 
LVL 13

Expert Comment

by:Sandy
ID: 39788112
in single line

#tcpdump port 80
0
 

Author Comment

by:georgopanos
ID: 39795131
tintin: this works for my needs but one other question,

I see for example when it runs the computer will hit an identical address 10 times in a row
for example:

11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com

how can I run the tcpdump and do an IF THEN comparison
to cut the above down to look like this 3 lines instead of 15 just to be able to read the log
easier.

11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com

Thanks
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39795946
Depends if you want to read it in realtime or not.

If you capture to a log, you can simply do:

uniq -d logfile
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39795981
otherwise

#sort logfile | uniq -d
0
 

Author Comment

by:georgopanos
ID: 39796882
I would like to sort it in real time if possible so as to display it to a screen
0
 
LVL 48

Accepted Solution

by:
Tintin earned 500 total points
ID: 39798459
Try this

tcpdump -q dst port http or https|awk '{print $1,$5}'|sed "s/\.http.*//"|sed "s/\.[0-9]* / /"|uniq -d


Note that you may need to wait a little while to see any output as it will be buffered.
0
 

Author Closing Comment

by:georgopanos
ID: 39798504
Great! Thank you!!!!
0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

861 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now