Solved

Linux VM running slow, receiving INFO: task : blocked for more than 120 seconds errors

Posted on 2016-07-20
17
152 Views
Last Modified: 2016-08-05
Oracle Linux 7.1 Running on VMware vSphere 5.5.

“INFO: task [process]:xxxxx blocked for more than 120 seconds.” Seems like out of memory issues, resource starvation often around disk access. It might be a kernel setting. I don't know.

Here is some logging from the machine:

root@nas01abcxyz001 log]# less /var/log/messages
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811a6554>] mnt_want_write+0x24/0x50
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8119691f>] do_last+0xbaf/0x10c0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81196ee2>] path_openat+0xb2/0x4b0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81192c79>] ? putname+0x29/0x40
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811978cf>] ? user_path_at_empty+0x5f/0x90
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81197a28>] do_filp_open+0x38/0x80
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8157dc9e>] ? _raw_spin_lock+0xe/0x20
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811a453f>] ? __alloc_fd+0xaf/0x140
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81186ef4>] do_sys_open+0xe4/0x1c0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81186ff1>] sys_open+0x21/0x30
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81586239>] system_call_fastpath+0x16/0x1b
Jul 19 07:07:33 nas01abcxyz001 kernel: INFO: task oracle:28959 blocked for more than 120 seconds.
Jul 19 07:07:33 nas01abcxyz001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 07:07:33 nas01abcxyz001 kernel: oracle          D ffff88042fc13100     0 28959      1 0x00000080
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff8802ad2adc38 0000000000000082 ffff880427df2540 ffff8802ad2adfd8
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff8802ad2adfd8 ffff8802ad2adfd8 ffff880427df2540 ffff88082a0d4000
Jul 19 07:07:33 nas01abcxyz001 kernel: 0000000000000001 0000000000000001 0000000000000000 ffff88082a0d4308
Jul 19 07:07:33 nas01abcxyz001 kernel: Call Trace:
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8157c839>] schedule+0x29/0x70
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81189a82>] __sb_start_write+0xb2/0x100
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81081d10>] ? wake_up_bit+0x30/0x30
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8112ab1a>] generic_file_write_iter+0x4a/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffffa0802fed>] ext4_file_write_iter+0x7d/0x310 [ext4]
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8120f349>] ? sys_semtimedop+0x489/0xac0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811873b2>] do_aio_write+0xa2/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811875ad>] do_sync_write+0x9d/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81187c06>] vfs_write+0xa6/0x180
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81187f42>] sys_pwrite64+0x72/0xb0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81586239>] system_call_fastpath+0x16/0x1b
Jul 19 07:07:33 nas01abcxyz001 kernel: INFO: task oracle:28961 blocked for more than 120 seconds.
Jul 19 07:07:33 nas01abcxyz001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 07:07:33 nas01abcxyz001 kernel: oracle          D ffff88042fd13100     0 28961      1 0x00000080
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff880822769c38 0000000000000082 ffff880822698380 ffff880822769fd8
Jul 19 07:07:33 nas01abcxyz001 kernel: ffff880822769fd8 ffff880822769fd8 ffff880822698380 ffff88082a0d4000
Jul 19 07:07:33 nas01abcxyz001 kernel: 0000000000000001 0000000000000001 0000000000000000 ffff88082a0d4308
Jul 19 07:07:33 nas01abcxyz001 kernel: Call Trace:
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8157c839>] schedule+0x29/0x70
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81189a82>] __sb_start_write+0xb2/0x100
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff81081d10>] ? wake_up_bit+0x30/0x30
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8112ab1a>] generic_file_write_iter+0x4a/0xd0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffffa0802fed>] ext4_file_write_iter+0x7d/0x310 [ext4]
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff8120f0cc>] ? sys_semtimedop+0x20c/0xac0
Jul 19 07:07:33 nas01abcxyz001 kernel: [<ffffffff811873b2>] do_aio_write+0xa2/0xd0



I don't know how to resolve this. Please let me know what additional information you need to hep assist me in this. Thanks.
0
Comment
Question by:so_cal
  • 7
  • 6
  • 3
17 Comments
 
LVL 118
ID: 41722042
can you check for snapshots ?

screenshot of the datastore folder, and I can check.
0
 

Author Comment

by:so_cal
ID: 41723374
I'm not sure about the snapshots, but here is a screenshot oft he datastore folder:

helpme2.png
0
 
LVL 118
ID: 41723424
you will have to look in the VM folder on the datastore which contains this VM
0
 

Author Comment

by:so_cal
ID: 41723434
This one?:

 datastore-browser.png
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 41726884
I think your problem is with the whole virtual machine rather than Oracle.
How do you have such a high level of logging from VMWare? That is likely to slow down your VM. especially it will slow it down if you have configured syslogd to sync log messages to disk (which is the default). To avoid syncing to disk, preface the log file path with a minus sign in /etc/syslogd.conf or whatever your system uses. Example
# Everything is logged here.
*.debug                                         -/var/log/debug

Open in new window

1
 
LVL 118
ID: 41726889
there must be another folder on datastore because that folder is missing config files
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 41726897
Are the log entries in the original Q from the VM or the host? Either way, there will be config files, usually in or somewhere below /etc on the system concerned. This is in the Linux file system - you need to get an xterm and investigate. The datastore is just for Oracle, right?
0
 

Author Comment

by:so_cal
ID: 41728075
Duncan, I don't see a syslogd.conf, but I do see a sysctl.conf. Should I use that one?
helpme3.png
Andrew, I've looked through the Datastores above, didn't see one that contains config files.

Duncan, the log entries are from the VM. There are four VMs on the host machine. Here is what's in /etc

[root@nas01abcxyz001 etc]# ls
adjtime                  idmapd.conf               rc0.d
aliases                  init.d                    rc1.d
aliases.db               inittab                   rc2.d
alternatives             inputrc                   rc3.d
anacrontab               iproute2                  rc4.d
asound.conf              iscsi                     rc5.d
audisp                   issue                     rc6.d
audit                    issue.net                 rc.d
avahi                    java                      rc.local
bash_completion.d        jvm                       rcS.d
bashrc                   jvm-commmon               rdma
binfmt.d                 kdump.conf                redhat-release
chkconfig.d              kernel                    request-key.conf
cron.d                   krb5.conf                 request-key.d
cron.daily               kshrc                     resolv.conf
cron.deny                ld.so.cache               rpc
cron.hourly              ld.so.conf                rpm
cron.monthly             ld.so.conf.d              rsyncd.conf
crontab                  libaudit.conf             rsyslog.conf
cron.weekly              libnl                     rsyslog.d
crypttab                 libuser.conf              rwtab
csh.cshrc                locale.conf               rwtab.d
csh.login                localtime                 samba
cups                     login.defs                sasl2
datto                    logrotate.conf            securetty
dbus-1                   logrotate.d               security
default                  lvm                       selinux
depmod.d                 machine-id                services
dhcp                     magic                     sestatus.conf
DIR_COLORS               mailcap                   shadow
DIR_COLORS.256color      makedumpfile.conf.sample  shadow-
DIR_COLORS.lightbgcolor  man_db.conf               shells
dkms                     maven                     skel
dnsmasq.conf             mime.types                ssh
dnsmasq.d                mke2fs.conf               ssl
dracut.conf              modprobe.d                statetab
dracut.conf.d            modules-load.d            statetab.d
e2fsck.conf              motd                      subversion
environment              mtab                      sudo.conf
ethertypes               multipath                 sudoers
exports                  netconfig                 sudoers.d
exports.d                NetworkManager            sudo-ldap.conf
favicon.png              networks                  sysconfig
filesystems              nfsmount.conf             sysctl.conf
firewalld                nsswitch.conf             sysctl.confe
fonts                    nsswitch.conf.bak         sysctl.d
fstab                    openldap                  systemd
gconf                    opt                       system-release
gcrypt                   oracle-release            system-release-cpe
gdbinit                  oraInst.loc               tcsd.conf
gdbinit.d                oratab                    terminfo
ghostscript              os-release                tmpfiles.d
gnupg                    pam.d                     trusted-key.key
GREP_COLORS              passwd                    tuned
groff                    passwd-                   udev
group                    pkcs11                    vconsole.conf
group-                   pki                       virc
grub2.cfg                plymouth                  vmware-tools
grub.d                   pm                        webmin
gshadow                  polkit-1                  wgetrc
gshadow-                 popt.d                    wpa_supplicant
gss                      postfix                   X11
gssproxy                 ppp                       xdg
host.conf                prelink.conf.d            xinetd.d
hostname                 printcap                  yum
HOSTNAME                 profile                   yum.conf
hosts                    profile.d                 yum.repos.d
hosts.allow              protocols
hosts.deny               pulse

Open in new window


Yes, the datastore is just for Oracle. Should I use any xterm? Why do I need it. How would I use it? How is it going to help?
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 500 total points
ID: 41728789
sysctl is something completely different - leave it alone.
The file you want to look at is rsyslog.conf. The example I have is rather old but looks similar to syslogd.conf.
I suggested to get an xterm to give you access to the command line, but you seem to have done that anyway
0
 

Author Comment

by:so_cal
ID: 41732166
So which one do I put a - (minus sign) in front of?:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Open in new window

0
 
LVL 34

Accepted Solution

by:
Duncan Roe earned 500 total points
ID: 41732196
At least and for sure, /var/log/messages. Personally I would also do: /var/log/secure, /var/log/cron, /var/log/spooler/var/log/boot.log.
None of them have any need to be synced to disk in my opinion, and you may get the added benefit of multiple identical messages being consolidated (i.e. you will get that benefit if you have sequences of identical messages).
1
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 500 total points
ID: 41732204
rsyslogd will only action changes after you make it restart: pkill -HUP rsyslogd
1
 

Author Comment

by:so_cal
ID: 41733540
Ok, done. We'll see if this improves the performance of this VM.
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 41743677
Hey so_cal,

Any progress?
0
 

Author Comment

by:so_cal
ID: 41744567
It seems to have helped. I think I'm going to close this thread now. Thanks Duncan.
0
 

Author Closing Comment

by:so_cal
ID: 41744570
Thank you, it helped.
0

Featured Post

Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
In this article, I will show you HOW TO: Suppress Configuration Issues and Warnings Alert displayed in Summary status for ESXi 6.5 after enabling SSH or ESXi Shell.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

919 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now