Solved

Capture HTTP/HTTPS and IP address orgin

Posted on 2014-01-14
13
335 Views
Last Modified: 2014-01-21
I am looking for a simple command line linux utility to have it run on automatically and capture HTTP/HTTPS and the computer IP it originated from.
0
Comment
Question by:georgopanos
  • 5
  • 5
  • 2
  • +1
13 Comments
 
LVL 34

Expert Comment

by:Seth Simmons
ID: 39781097
if you are talking about seeing where traffic is coming from to access your web site, it should be in the logs

if you are running apache, it will appear in the access log
by default it will include the source IP address, date, time, page requested, browser used and result code
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39781103
Do you want to capture the HTTP and HTTPS payload as well?  If so, then use tcpdump, wireshark, ngrep or similar to do a packet capture.   Note that it's a little more involved to capture HTTPS in plain text.
0
 

Author Comment

by:georgopanos
ID: 39781116
I think wireshark is overkill for what I want.

I basically want a simple log file that will list

HTTP/HTTPS - orgin IP address or (computer name)

I was looking for a simple command line utility I can setup on my Rasberry Pi to autorun on boot with a Network tap and capture the above INFORMATION from a network. Just to monitor what is being browsed.
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 
LVL 48

Expert Comment

by:Tintin
ID: 39781229
Why not just do

tail -f /var/log/apache2/access.log
0
 

Author Comment

by:georgopanos
ID: 39781856
I am not running a website, I want to see the websites the computers on my network are going to.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39783962
Ah, makes sense now.

Assuming the traffic goes through your pi, or you have a promiscuous network, then you can do:

tcpdump port 80 or port 443

if that's a little too verbose, you could do

tcpdump port 80 or port 443 | sed "s/\.http.*//"
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39788112
in single line

#tcpdump port 80
0
 

Author Comment

by:georgopanos
ID: 39795131
tintin: this works for my needs but one other question,

I see for example when it runs the computer will hit an identical address 10 times in a row
for example:

11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com

how can I run the tcpdump and do an IF THEN comparison
to cut the above down to look like this 3 lines instead of 15 just to be able to read the log
easier.

11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com

Thanks
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39795946
Depends if you want to read it in realtime or not.

If you capture to a log, you can simply do:

uniq -d logfile
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39795981
otherwise

#sort logfile | uniq -d
0
 

Author Comment

by:georgopanos
ID: 39796882
I would like to sort it in real time if possible so as to display it to a screen
0
 
LVL 48

Accepted Solution

by:
Tintin earned 500 total points
ID: 39798459
Try this

tcpdump -q dst port http or https|awk '{print $1,$5}'|sed "s/\.http.*//"|sed "s/\.[0-9]* / /"|uniq -d


Note that you may need to wait a little while to see any output as it will be buffered.
0
 

Author Closing Comment

by:georgopanos
ID: 39798504
Great! Thank you!!!!
0

Featured Post

Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
AWS- KeepAlived notify script not working 23 86
AD LDAP LDS 3 68
Why  my code (program) build with old compiler? 11 49
linux SFTP 8 46
Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question