sysautomation
asked on
High Bandwidth Usage
I am running a CentOS 6.7 64 bit server hosted in a data center. For last some weeks it is consuming over 500GB/day bandwidth. It is too much and it has increased suddenly. I don't have any idea what is consuming that much bandwidth.
I run iftop but it shows only a few KBs usage. But I feel the usage is done at some time when I am not running iftop.
I am looking for a tool which can summarize bandwidth usage by process for last 24 hours or so. Is there anything available like that?
I will be much thankful if someone can please help?
I run iftop but it shows only a few KBs usage. But I feel the usage is done at some time when I am not running iftop.
I am looking for a tool which can summarize bandwidth usage by process for last 24 hours or so. Is there anything available like that?
I will be much thankful if someone can please help?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
# pstree -u
init─┬─NetworkManager
├─VBoxSVC(srvprodh-scm)─┬─VirtualBox───23*[{VirtualBox}]
│ └─13*[{VBoxSVC}]
├─VBoxXPCOMIPCD(srvprodh-scm)
├─VirtualBox(srvprodh-scm)───5*[{VirtualBox}]
├─Xvnc(srvprodh-scm)
├─abrtd
├─acpid
├─atd
├─auditd───{auditd}
├─bonobo-activati(srvprodh-scm)───{bonobo-activat}
├─ck-xinit-sessio(srvprodh-scm)─┬─gnome-session─┬─gnome-panel───{gnome-panel}
│ │ ├─gnome-power-man
│ │ ├─gnome-volume-co
│ │ ├─gpk-update-icon───{gpk-update-ico}
│ │ ├─metacity
│ │ ├─nautilus
│ │ ├─polkit-gnome-au───{polkit-gnome-a}
│ │ ├─ssh-agent
│ │ └─{gnome-session}
│ └─vncconfig
├─clock-applet(srvprodh-scm)
├─console-kit-dae───63*[{console-kit-da}]
├─crond
├─dbus-daemon(dbus)
├─dbus-daemon(srvprodh-scm)
├─dbus-launch(srvprodh-scm)
├─devkit-power-da
├─dovecot─┬─anvil(dovecot)
│ └─log
├─fail2ban-server───6*[{fail2ban-serve}]
├─gconfd-2(srvprodh-scm)
├─gdm-user-switch(srvprodh-scm)
├─gnome-keyring-d(srvprodh-scm)───2*[{gnome-keyring-}]
├─gnome-screensav(srvprodh-scm)
├─gnome-settings-(srvprodh-scm)───{gnome-settings}
├─gvfs-gdu-volume(srvprodh-scm)
├─gvfsd(srvprodh-scm)
├─gvfsd-dnssd(srvprodh-scm)
├─gvfsd-metadata(srvprodh-scm)
├─gvfsd-network(srvprodh-scm)
├─gvfsd-trash(srvprodh-scm)
├─hald(haldaemon)─┬─hald-runner(root)─┬─hald-addon-acpi(haldaemon)
│ │ └─hald-addon-inpu
│ └─{hald}
├─httpd───12*[httpd(apache)]
├─irqbalance
├─java───119*[{java}]
├─mailmanctl(mailman)───8*[python]
├─master─┬─pickup(postfix)
│ └─qmgr(postfix)
├─6*[mingetty]
├─2*[miniserv.pl]
├─modem-manager
├─mysqld_safe───mysqld(mysql)───9*[{mysqld}]
├─named(named)───26*[{named}]
├─notification-ar(srvprodh-scm)
├─61*[oracle(oracle)]
├─polkitd
├─portreserve
├─proftpd(nobody)
├─pulseaudio(srvprodh-scm)─┬─gconf-helper
│ └─{pulseaudio}
├─rsyslogd───3*[{rsyslogd}]
├─rtkit-daemon(rtkit)───2*[{rtkit-daemon}]
├─saslauthd───4*[saslauthd]
├─sshd───sshd───bash───pstree
├─2*[sudo───nethogs]
├─tnslsnr(oracle)───2*[{tnslsnr}]
├─trashapplet(srvprodh-scm)
├─udevd───2*[udevd]
├─udisks-daemon───udisks-daemon
├─wnck-applet(srvprodh-scm)
├─wpa_supplicant
└─zabbix_agentd(zabbix)───5*[zabbix_agentd]
# netstat -lputn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:20000 0.0.0.0:* LISTEN 5022/perl
tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 3706/dovecot
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 3879/zabbix_agentd
tcp 0 0 0.0.0.0:995 0.0.0.0:* LISTEN 3706/dovecot
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 3678/mysqld
tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 3805/master
tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN 3534/sshd
tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN 18317/Xvnc
tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN 3706/dovecot
tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 3706/dovecot
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 5031/perl
tcp 0 0 0.0.0.0:6001 0.0.0.0:* LISTEN 18317/Xvnc
tcp 0 0 192.228.106.35:53 0.0.0.0:* LISTEN 15288/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 15288/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 15288/named
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 3805/master
tcp 0 0 :::993 :::* LISTEN 3706/dovecot
tcp 0 0 :::10050 :::* LISTEN 3879/zabbix_agentd
tcp 0 0 :::9091 :::* LISTEN 4830/java
tcp 0 0 :::995 :::* LISTEN 3706/dovecot
tcp 0 0 :::9191 :::* LISTEN 4830/java
tcp 0 0 :::587 :::* LISTEN 3805/master
tcp 0 0 :::9292 :::* LISTEN 3534/sshd
tcp 0 0 :::8686 :::* LISTEN 4830/java
tcp 0 0 :::110 :::* LISTEN 3706/dovecot
tcp 0 0 :::143 :::* LISTEN 3706/dovecot
tcp 0 0 :::4848 :::* LISTEN 4830/java
tcp 0 0 :::80 :::* LISTEN 3862/httpd
tcp 0 0 :::6001 :::* LISTEN 18317/Xvnc
tcp 0 0 :::1521 :::* LISTEN 3919/tnslsnr
tcp 0 0 :::45203 :::* LISTEN 4143/ora_d000_orcl
tcp 0 0 :::3700 :::* LISTEN 4830/java
tcp 0 0 :::53 :::* LISTEN 15288/named
tcp 0 0 :::21 :::* LISTEN 3824/proftpd
tcp 0 0 ::1:953 :::* LISTEN 15288/named
tcp 0 0 :::25 :::* LISTEN 3805/master
tcp 0 0 :::443 :::* LISTEN 3862/httpd
tcp 0 0 :::7676 :::* LISTEN 4830/java
udp 0 0 0.0.0.0:783 0.0.0.0:* 2772/portreserve
udp 0 0 0.0.0.0:10000 0.0.0.0:* 5031/perl
udp 0 0 0.0.0.0:57118 0.0.0.0:* 18615/VirtualBox
udp 0 0 0.0.0.0:20000 0.0.0.0:* 5022/perl
udp 0 0 192.228.106.35:53 0.0.0.0:* 15288/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 15288/named
udp 0 0 :::9496 :::* 4131/ora_lgwr_orcl
udp 0 0 :::53 :::* 15288/named
udp 0 0 :::18751 :::* 4139/ora_mmon_orcl
udp 0 0 ::1:17635 :::* 4143/ora_d000_orcl
udp 0 0 ::1:14184 :::* 4097/ora_pmon_orcl
udp 0 0 ::1:41210 :::* 4145/ora_s000_orcl
Could be DNS amplification. Can you examine named logs that by default land in /var/log/messages?
ASKER
I turned off BIND but the bandwidth usage looks the same.
Are you running Oracle database available to anybody with internet?
Please install some network firewall and follow up its logs. From your netstat is visible that you dont control what is running in the system.
Please install some network firewall and follow up its logs. From your netstat is visible that you dont control what is running in the system.
Once you have the time of day, you then need to find a definitive answer of WHAT is using that bandwidth. I would install Wireshark (www.wireshark.org) and set it up to capture network data during the critical time period(s). (Obviously, you'll need to have lots of disk space for this.) With this data captured, you'll see EXACTLY what is going in and out of the box, and you can track it back to the original application and the machine on the remote side, too.