Link to home
Start Free TrialLog in
Avatar of sysautomation
sysautomation

asked on

High Bandwidth Usage

I am running a CentOS 6.7 64 bit server hosted in a data center. For last some weeks it is consuming over 500GB/day bandwidth. It is too much and it has increased suddenly. I don't have any idea what is consuming that much bandwidth.

I run iftop but it shows only a few KBs usage.  But I feel the usage is done at some time when I am not running iftop.

I am looking for a tool which can summarize bandwidth usage by process for last 24 hours or so. Is there anything available like that?

I will be much thankful if someone can please help?
ASKER CERTIFIED SOLUTION
Avatar of gheist
gheist
Flag of Belgium image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Bill Bach
The first step is to find out WHEN during the day that the bandwidth needs increase.  You can use many different bandwidth tools for this, and you can find a small list at https://www.dynacont.net/documentation/linux/network_monitoring/.

Once you have the time of day, you then need to find a definitive answer of WHAT is using that bandwidth.  I would install Wireshark (www.wireshark.org) and set it up to capture network data during the critical time period(s).  (Obviously, you'll need to have lots of disk space for this.)  With this data captured, you'll see EXACTLY what is going in and out of the box, and you can track it back to the original application and the machine on the remote side, too.
Avatar of sysautomation
sysautomation

ASKER

# pstree -u
init─┬─NetworkManager
     ├─VBoxSVC(srvprodh-scm)─┬─VirtualBox───23*[{VirtualBox}]
     │                       └─13*[{VBoxSVC}]
     ├─VBoxXPCOMIPCD(srvprodh-scm)
     ├─VirtualBox(srvprodh-scm)───5*[{VirtualBox}]
     ├─Xvnc(srvprodh-scm)
     ├─abrtd
     ├─acpid
     ├─atd
     ├─auditd───{auditd}
     ├─bonobo-activati(srvprodh-scm)───{bonobo-activat}
     ├─ck-xinit-sessio(srvprodh-scm)─┬─gnome-session─┬─gnome-panel───{gnome-panel}
     │                               │               ├─gnome-power-man
     │                               │               ├─gnome-volume-co
     │                               │               ├─gpk-update-icon───{gpk-update-ico}
     │                               │               ├─metacity
     │                               │               ├─nautilus
     │                               │               ├─polkit-gnome-au───{polkit-gnome-a}
     │                               │               ├─ssh-agent
     │                               │               └─{gnome-session}
     │                               └─vncconfig
     ├─clock-applet(srvprodh-scm)
     ├─console-kit-dae───63*[{console-kit-da}]
     ├─crond
     ├─dbus-daemon(dbus)
     ├─dbus-daemon(srvprodh-scm)
     ├─dbus-launch(srvprodh-scm)
     ├─devkit-power-da
     ├─dovecot─┬─anvil(dovecot)
     │         └─log
     ├─fail2ban-server───6*[{fail2ban-serve}]
     ├─gconfd-2(srvprodh-scm)
     ├─gdm-user-switch(srvprodh-scm)
     ├─gnome-keyring-d(srvprodh-scm)───2*[{gnome-keyring-}]
     ├─gnome-screensav(srvprodh-scm)
     ├─gnome-settings-(srvprodh-scm)───{gnome-settings}
     ├─gvfs-gdu-volume(srvprodh-scm)
     ├─gvfsd(srvprodh-scm)
     ├─gvfsd-dnssd(srvprodh-scm)
     ├─gvfsd-metadata(srvprodh-scm)
     ├─gvfsd-network(srvprodh-scm)
     ├─gvfsd-trash(srvprodh-scm)
     ├─hald(haldaemon)─┬─hald-runner(root)─┬─hald-addon-acpi(haldaemon)
     │                 │                   └─hald-addon-inpu
     │                 └─{hald}
     ├─httpd───12*[httpd(apache)]
     ├─irqbalance
     ├─java───119*[{java}]
     ├─mailmanctl(mailman)───8*[python]
     ├─master─┬─pickup(postfix)
     │        └─qmgr(postfix)
     ├─6*[mingetty]
     ├─2*[miniserv.pl]
     ├─modem-manager
     ├─mysqld_safe───mysqld(mysql)───9*[{mysqld}]
     ├─named(named)───26*[{named}]
     ├─notification-ar(srvprodh-scm)
     ├─61*[oracle(oracle)]
     ├─polkitd
     ├─portreserve
     ├─proftpd(nobody)
     ├─pulseaudio(srvprodh-scm)─┬─gconf-helper
     │                          └─{pulseaudio}
     ├─rsyslogd───3*[{rsyslogd}]
     ├─rtkit-daemon(rtkit)───2*[{rtkit-daemon}]
     ├─saslauthd───4*[saslauthd]
     ├─sshd───sshd───bash───pstree
     ├─2*[sudo───nethogs]
     ├─tnslsnr(oracle)───2*[{tnslsnr}]
     ├─trashapplet(srvprodh-scm)
     ├─udevd───2*[udevd]
     ├─udisks-daemon───udisks-daemon
     ├─wnck-applet(srvprodh-scm)
     ├─wpa_supplicant
     └─zabbix_agentd(zabbix)───5*[zabbix_agentd]

# netstat -lputn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:20000               0.0.0.0:*                   LISTEN      5022/perl           
tcp        0      0 0.0.0.0:993                 0.0.0.0:*                   LISTEN      3706/dovecot        
tcp        0      0 0.0.0.0:10050               0.0.0.0:*                   LISTEN      3879/zabbix_agentd  
tcp        0      0 0.0.0.0:995                 0.0.0.0:*                   LISTEN      3706/dovecot        
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      3678/mysqld         
tcp        0      0 0.0.0.0:587                 0.0.0.0:*                   LISTEN      3805/master         
tcp        0      0 0.0.0.0:9292                0.0.0.0:*                   LISTEN      3534/sshd           
tcp        0      0 0.0.0.0:5901                0.0.0.0:*                   LISTEN      18317/Xvnc          
tcp        0      0 0.0.0.0:110                 0.0.0.0:*                   LISTEN      3706/dovecot        
tcp        0      0 0.0.0.0:143                 0.0.0.0:*                   LISTEN      3706/dovecot        
tcp        0      0 0.0.0.0:10000               0.0.0.0:*                   LISTEN      5031/perl           
tcp        0      0 0.0.0.0:6001                0.0.0.0:*                   LISTEN      18317/Xvnc          
tcp        0      0 192.228.106.35:53           0.0.0.0:*                   LISTEN      15288/named         
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      15288/named         
tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      15288/named         
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      3805/master         
tcp        0      0 :::993                      :::*                        LISTEN      3706/dovecot        
tcp        0      0 :::10050                    :::*                        LISTEN      3879/zabbix_agentd  
tcp        0      0 :::9091                     :::*                        LISTEN      4830/java           
tcp        0      0 :::995                      :::*                        LISTEN      3706/dovecot        
tcp        0      0 :::9191                     :::*                        LISTEN      4830/java           
tcp        0      0 :::587                      :::*                        LISTEN      3805/master         
tcp        0      0 :::9292                     :::*                        LISTEN      3534/sshd           
tcp        0      0 :::8686                     :::*                        LISTEN      4830/java           
tcp        0      0 :::110                      :::*                        LISTEN      3706/dovecot        
tcp        0      0 :::143                      :::*                        LISTEN      3706/dovecot        
tcp        0      0 :::4848                     :::*                        LISTEN      4830/java           
tcp        0      0 :::80                       :::*                        LISTEN      3862/httpd          
tcp        0      0 :::6001                     :::*                        LISTEN      18317/Xvnc          
tcp        0      0 :::1521                     :::*                        LISTEN      3919/tnslsnr        
tcp        0      0 :::45203                    :::*                        LISTEN      4143/ora_d000_orcl  
tcp        0      0 :::3700                     :::*                        LISTEN      4830/java           
tcp        0      0 :::53                       :::*                        LISTEN      15288/named         
tcp        0      0 :::21                       :::*                        LISTEN      3824/proftpd        
tcp        0      0 ::1:953                     :::*                        LISTEN      15288/named         
tcp        0      0 :::25                       :::*                        LISTEN      3805/master         
tcp        0      0 :::443                      :::*                        LISTEN      3862/httpd          
tcp        0      0 :::7676                     :::*                        LISTEN      4830/java           
udp        0      0 0.0.0.0:783                 0.0.0.0:*                               2772/portreserve    
udp        0      0 0.0.0.0:10000               0.0.0.0:*                               5031/perl           
udp        0      0 0.0.0.0:57118               0.0.0.0:*                               18615/VirtualBox    
udp        0      0 0.0.0.0:20000               0.0.0.0:*                               5022/perl           
udp        0      0 192.228.106.35:53           0.0.0.0:*                               15288/named         
udp        0      0 127.0.0.1:53                0.0.0.0:*                               15288/named         
udp        0      0 :::9496                     :::*                                    4131/ora_lgwr_orcl  
udp        0      0 :::53                       :::*                                    15288/named         
udp        0      0 :::18751                    :::*                                    4139/ora_mmon_orcl  
udp        0      0 ::1:17635                   :::*                                    4143/ora_d000_orcl  
udp        0      0 ::1:14184                   :::*                                    4097/ora_pmon_orcl  
udp        0      0 ::1:41210                   :::*                                    4145/ora_s000_orcl

Open in new window

Could be DNS amplification. Can you examine named logs that by default land in /var/log/messages?
I turned off BIND  but the bandwidth usage looks the same.
Are you running Oracle database available to anybody with internet?
Please install some network firewall and follow up its logs. From your netstat is visible that you dont control what is running in the system.