Solved

Capture HTTP/HTTPS and IP address orgin

Posted on 2014-01-14
13
338 Views
Last Modified: 2014-01-21
I am looking for a simple command line linux utility to have it run on automatically and capture HTTP/HTTPS and the computer IP it originated from.
0
Comment
Question by:georgopanos
  • 5
  • 5
  • 2
  • +1
13 Comments
 
LVL 35

Expert Comment

by:Seth Simmons
ID: 39781097
if you are talking about seeing where traffic is coming from to access your web site, it should be in the logs

if you are running apache, it will appear in the access log
by default it will include the source IP address, date, time, page requested, browser used and result code
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39781103
Do you want to capture the HTTP and HTTPS payload as well?  If so, then use tcpdump, wireshark, ngrep or similar to do a packet capture.   Note that it's a little more involved to capture HTTPS in plain text.
0
 

Author Comment

by:georgopanos
ID: 39781116
I think wireshark is overkill for what I want.

I basically want a simple log file that will list

HTTP/HTTPS - orgin IP address or (computer name)

I was looking for a simple command line utility I can setup on my Rasberry Pi to autorun on boot with a Network tap and capture the above INFORMATION from a network. Just to monitor what is being browsed.
0
NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

 
LVL 48

Expert Comment

by:Tintin
ID: 39781229
Why not just do

tail -f /var/log/apache2/access.log
0
 

Author Comment

by:georgopanos
ID: 39781856
I am not running a website, I want to see the websites the computers on my network are going to.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39783962
Ah, makes sense now.

Assuming the traffic goes through your pi, or you have a promiscuous network, then you can do:

tcpdump port 80 or port 443

if that's a little too verbose, you could do

tcpdump port 80 or port 443 | sed "s/\.http.*//"
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39788112
in single line

#tcpdump port 80
0
 

Author Comment

by:georgopanos
ID: 39795131
tintin: this works for my needs but one other question,

I see for example when it runs the computer will hit an identical address 10 times in a row
for example:

11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com
11:53 10.1.10.1 > compute.amazonaws.com

how can I run the tcpdump and do an IF THEN comparison
to cut the above down to look like this 3 lines instead of 15 just to be able to read the log
easier.

11:51 10.1.10.1 > compute.amazonaws.com
11:52 10.1.10.1> www.google.com
11:53 10.1.10.1 > compute.amazonaws.com

Thanks
0
 
LVL 48

Expert Comment

by:Tintin
ID: 39795946
Depends if you want to read it in realtime or not.

If you capture to a log, you can simply do:

uniq -d logfile
0
 
LVL 13

Expert Comment

by:Sandy
ID: 39795981
otherwise

#sort logfile | uniq -d
0
 

Author Comment

by:georgopanos
ID: 39796882
I would like to sort it in real time if possible so as to display it to a screen
0
 
LVL 48

Accepted Solution

by:
Tintin earned 500 total points
ID: 39798459
Try this

tcpdump -q dst port http or https|awk '{print $1,$5}'|sed "s/\.http.*//"|sed "s/\.[0-9]* / /"|uniq -d


Note that you may need to wait a little while to see any output as it will be buffered.
0
 

Author Closing Comment

by:georgopanos
ID: 39798504
Great! Thank you!!!!
0

Featured Post

Back Up Your Microsoft Windows Server®

Back up all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
SonarQube on Linux vs Windows 3 82
pvcreate issue 5 41
Can't connect to FTP 18 111
Ubuntu don’t allow SU command in terminal 7 70
Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
BIND is the most widely used Name Server. A Name Server is the one that translates a site name to it's IP address. There is a new bug in BIND (https://kb.isc.org/article/AA-01272), affecting all versions of BIND 9 from BIND 9.1.0 (inclusive) thro…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question