Solved

Linux VM running slow, receiving INFO: task : blocked for more than 120 seconds errors

Posted on 2016-07-20
17
133 Views
Last Modified: 2016-08-05
Oracle Linux 7.1 Running on VMware vSphere 5.5.

“INFO: task [process]:xxxxx blocked for more than 120 seconds.” Seems like out of memory issues, resource starvation often around disk access. It might be a kernel setting. I don't know.

Here is some logging from the machine:

root@nas01abcxyz001 log]# less /var/log/messages
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811a6554>] mnt_want_write+0x24/0x50
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8119691f>] do_last+0xbaf/0x10c0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81196ee2>] path_openat+0xb2/0x4b0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81192c79>] ? putname+0x29/0x40
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811978cf>] ? user_path_at_empty+0x5f/0x90
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81197a28>] do_filp_open+0x38/0x80
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8157dc9e>] ? _raw_spin_lock+0xe/0x20
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811a453f>] ? __alloc_fd+0xaf/0x140
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81186ef4>] do_sys_open+0xe4/0x1c0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81186ff1>] sys_open+0x21/0x30
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81586239>] system_call_fastpath+0x16/0x1b
Jul 19 07:07:33 nas01abcxyz001 kernel: INFO: task oracle:28959 blocked for more than 120 seconds.
Jul 19 07:07:33 nas01abcxyz001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 07:07:33 nas01abcxyz001 kernel: oracle          D ffff88042fc13100     0 28959      1 0x00000080
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff8802ad2adc38 0000000000000082 ffff880427df2540 ffff8802ad2adfd8
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff8802ad2adfd8 ffff8802ad2adfd8 ffff880427df2540 ffff88082a0d4000
Jul 19 07:07:33 nas01abcxyz001 kernel: 0000000000000001 0000000000000001 0000000000000000 ffff88082a0d4308
Jul 19 07:07:33 nas01abcxyz001 kernel: Call Trace:
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8157c839>] schedule+0x29/0x70
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81189a82>] __sb_start_write+0xb2/0x100
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81081d10>] ? wake_up_bit+0x30/0x30
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8112ab1a>] generic_file_write_iter+0x4a/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffffa0802fed>] ext4_file_write_iter+0x7d/0x310 [ext4]
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8120f349>] ? sys_semtimedop+0x489/0xac0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811873b2>] do_aio_write+0xa2/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811875ad>] do_sync_write+0x9d/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81187c06>] vfs_write+0xa6/0x180
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81187f42>] sys_pwrite64+0x72/0xb0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81586239>] system_call_fastpath+0x16/0x1b
Jul 19 07:07:33 nas01abcxyz001 kernel: INFO: task oracle:28961 blocked for more than 120 seconds.
Jul 19 07:07:33 nas01abcxyz001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 07:07:33 nas01abcxyz001 kernel: oracle          D ffff88042fd13100     0 28961      1 0x00000080
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff880822769c38 0000000000000082 ffff880822698380 ffff880822769fd8
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff880822769fd8 ffff880822769fd8 ffff880822698380 ffff88082a0d4000
Jul 19 07:07:33 nas01abcxyz001 kernel: 0000000000000001 0000000000000001 0000000000000000 ffff88082a0d4308
Jul 19 07:07:33 nas01abcxyz001 kernel: Call Trace:
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8157c839>] schedule+0x29/0x70
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81189a82>] __sb_start_write+0xb2/0x100
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81081d10>] ? wake_up_bit+0x30/0x30
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8112ab1a>] generic_file_write_iter+0x4a/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffffa0802fed>] ext4_file_write_iter+0x7d/0x310 [ext4]
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8120f0cc>] ? sys_semtimedop+0x20c/0xac0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811873b2>] do_aio_write+0xa2/0xd0



I don't know how to resolve this. Please let me know what additional information you need to hep assist me in this. Thanks.
0
Comment
Question by:so_cal
  • 7
  • 6
  • 3
17 Comments
 
LVL 118
ID: 41722042
can you check for snapshots ?

screenshot of the datastore folder, and I can check.
0
 

Author Comment

by:so_cal
ID: 41723374
I'm not sure about the snapshots, but here is a screenshot oft he datastore folder:

helpme2.png
0
 
LVL 118
ID: 41723424
you will have to look in the VM folder on the datastore which contains this VM
0
 

Author Comment

by:so_cal
ID: 41723434
This one?:

 datastore-browser.png
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 41726884
I think your problem is with the whole virtual machine rather than Oracle.
How do you have such a high level of logging from VMWare? That is likely to slow down your VM. especially it will slow it down if you have configured syslogd to sync log messages to disk (which is the default). To avoid syncing to disk, preface the log file path with a minus sign in /etc/syslogd.conf or whatever your system uses. Example
# Everything is logged here.
*.debug                                         -/var/log/debug

Open in new window

1
 
LVL 118
ID: 41726889
there must be another folder on datastore because that folder is missing config files
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 41726897
Are the log entries in the original Q from the VM or the host? Either way, there will be config files, usually in or somewhere below /etc on the system concerned. This is in the Linux file system - you need to get an xterm and investigate. The datastore is just for Oracle, right?
0
 

Author Comment

by:so_cal
ID: 41728075
Duncan, I don't see a syslogd.conf, but I do see a sysctl.conf. Should I use that one?
helpme3.png
Andrew, I've looked through the Datastores above, didn't see one that contains config files.

Duncan, the log entries are from the VM. There are four VMs on the host machine. Here is what's in /etc

[root@nas01abcxyz001 etc]# ls
adjtime                  idmapd.conf               rc0.d
aliases                  init.d                    rc1.d
aliases.db               inittab                   rc2.d
alternatives             inputrc                   rc3.d
anacrontab               iproute2                  rc4.d
asound.conf              iscsi                     rc5.d
audisp                   issue                     rc6.d
audit                    issue.net                 rc.d
avahi                    java                      rc.local
bash_completion.d        jvm                       rcS.d
bashrc                   jvm-commmon               rdma
binfmt.d                 kdump.conf                redhat-release
chkconfig.d              kernel                    request-key.conf
cron.d                   krb5.conf                 request-key.d
cron.daily               kshrc                     resolv.conf
cron.deny                ld.so.cache               rpc
cron.hourly              ld.so.conf                rpm
cron.monthly             ld.so.conf.d              rsyncd.conf
crontab                  libaudit.conf             rsyslog.conf
cron.weekly              libnl                     rsyslog.d
crypttab                 libuser.conf              rwtab
csh.cshrc                locale.conf               rwtab.d
csh.login                localtime                 samba
cups                     login.defs                sasl2
datto                    logrotate.conf            securetty
dbus-1                   logrotate.d               security
default                  lvm                       selinux
depmod.d                 machine-id                services
dhcp                     magic                     sestatus.conf
DIR_COLORS               mailcap                   shadow
DIR_COLORS.256color      makedumpfile.conf.sample  shadow-
DIR_COLORS.lightbgcolor  man_db.conf               shells
dkms                     maven                     skel
dnsmasq.conf             mime.types                ssh
dnsmasq.d                mke2fs.conf               ssl
dracut.conf              modprobe.d                statetab
dracut.conf.d            modules-load.d            statetab.d
e2fsck.conf              motd                      subversion
environment              mtab                      sudo.conf
ethertypes               multipath                 sudoers
exports                  netconfig                 sudoers.d
exports.d                NetworkManager            sudo-ldap.conf
favicon.png              networks                  sysconfig
filesystems              nfsmount.conf             sysctl.conf
firewalld                nsswitch.conf             sysctl.confe
fonts                    nsswitch.conf.bak         sysctl.d
fstab                    openldap                  systemd
gconf                    opt                       system-release
gcrypt                   oracle-release            system-release-cpe
gdbinit                  oraInst.loc               tcsd.conf
gdbinit.d                oratab                    terminfo
ghostscript              os-release                tmpfiles.d
gnupg                    pam.d                     trusted-key.key
GREP_COLORS              passwd                    tuned
groff                    passwd-                   udev
group                    pkcs11                    vconsole.conf
group-                   pki                       virc
grub2.cfg                plymouth                  vmware-tools
grub.d                   pm                        webmin
gshadow                  polkit-1                  wgetrc
gshadow-                 popt.d                    wpa_supplicant
gss                      postfix                   X11
gssproxy                 ppp                       xdg
host.conf                prelink.conf.d            xinetd.d
hostname                 printcap                  yum
HOSTNAME                 profile                   yum.conf
hosts                    profile.d                 yum.repos.d
hosts.allow              protocols
hosts.deny               pulse

Open in new window


Yes, the datastore is just for Oracle. Should I use any xterm? Why do I need it. How would I use it? How is it going to help?
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 500 total points
ID: 41728789
sysctl is something completely different - leave it alone.
The file you want to look at is rsyslog.conf. The example I have is rather old but looks similar to syslogd.conf.
I suggested to get an xterm to give you access to the command line, but you seem to have done that anyway
0
 

Author Comment

by:so_cal
ID: 41732166
So which one do I put a - (minus sign) in front of?:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Open in new window

0
 
LVL 34

Accepted Solution

by:
Duncan Roe earned 500 total points
ID: 41732196
At least and for sure, /var/log/messages. Personally I would also do: /var/log/secure, /var/log/cron, /var/log/spooler & /var/log/boot.log.
None of them have any need to be synced to disk in my opinion, and you may get the added benefit of multiple identical messages being consolidated (i.e. you will get that benefit if you have sequences of identical messages).
1
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 500 total points
ID: 41732204
rsyslogd will only action changes after you make it restart: pkill -HUP rsyslogd
1
 

Author Comment

by:so_cal
ID: 41733540
Ok, done. We'll see if this improves the performance of this VM.
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 41743677
Hey so_cal,

Any progress?
0
 

Author Comment

by:so_cal
ID: 41744567
It seems to have helped. I think I'm going to close this thread now. Thanks Duncan.
0
 

Author Closing Comment

by:so_cal
ID: 41744570
Thank you, it helped.
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
David Varnum recently wrote up his impressions of PRTG, based on a presentation by my colleague Christian at Tech Field Day at VMworld in Barcelona. Thanks David, for your detailed and honest evaluation!
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now