We help IT Professionals succeed at work.

RHEL7 took very long to boot: dracut Warning   messages

sunhux
sunhux asked
on
High Priority
82 Views
Last Modified: 2020-02-24
On my RHEL7 VM, the reboot took about 30-40 mins to boot up
(was told by colleague who supports it, it's about 20 times longer).

 At the console, I could see
"dracut Warning: Cannot umount /oldroot
 dracut Warning: Blocking umount of /oldroot [14015]
   /usr/linb/systemd/systemd-shutdownreboot--log-leval6-log-targetkmsg
 dracut Warning: lrwxrwxrwx.  1  root  0  0 ...
   /proc/14015/exe-> /oldroot/usr/lib/systemd/systemd-shutdown

A few links suggest to disable firewalld but after it boots up, can't
   see that firewalld is running:
$ firewall-cmd --list-all |more
FirewallD is not running

is the dracut message on the console the cause of the slow booting
up or this long booting up is caused by another issue?  How can I
fix this?
Comment
Watch Question

Fractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Output from the following will likely tell you the exact problem.

systemd-analyze --no-pager blame

Open in new window


You can always post your systemd-analyze (as text, not an image) for further assistance.
Julian ParkerSenior Systems Administrator
CERTIFIED EXPERT
Commented:
Why do you have a /oldroot?

Is the server trying to shutdown but you have a link to old root and it may be blocking the restarts.

dracut Warning: lrwxrwxrwx.  1  root  0  0 ...
   /proc/14015/exe-> /oldroot/usr/lib/systemd/systemd-shutdown

Open in new window

Author

Commented:
> Is the server trying to shutdown but you have a link to old root and it may be blocking the restarts.
Think Julian may be right.

Btw, when I login to issue "find / -name oldroot -print", it returns nothing on the 3 VMs.

From the 3 VMs, I  issued the command given by David & none of them showed anything that cause
delays of above 1 minute that could explain the 30 mins booting up time (unless I do a 'Poweroff'
from the console of the VM from vCenter:

.51 VM:
$ systemd-analyze --no-pager blame
         19.714s kdump.service
          2.100s lvm2-pvscan@8:2.service
          1.658s docker.service
          1.388s dcos-net.service
          1.329s postfix.service
          1.191s lvm2-pvscan@8:16.service
           869ms dcos-diagnostics.service
           838ms dev-mapper-rhel\x2droot.device
           825ms dcos-mesos-slave-public.service

.56 VM:
$ systemd-analyze --no-pager blame
         18.990s kdump.service
          1.843s lvm2-pvscan@8:16.service
          1.815s docker.service
          1.507s dcos-net.service
          1.435s lvm2-pvscan@8:2.service
          1.364s postfix.service
           822ms dcos-diagnostics.service
           712ms dcos-mesos-slave.service
           697ms tuned.service
           589ms dcos-adminrouter-agent.service

.57 VM:
$ systemd-analyze --no-pager blame
         19.109s kdump.service
          1.959s docker.service
          1.336s postfix.service
          1.320s dcos-net.service
          1.183s lvm2-pvscan@8:2.service
          1.166s lvm2-pvscan@8:16.service
           813ms dcos-diagnostics.service
           746ms dcos-mesos-slave.service
           666ms tuned.service
           582ms dcos-adminrouter-agent.service

Author

Commented:
On the 3 VMs, while at  / , command below returns nothing:

$ ls -lad |grep -i oldroot

Author

Commented:
If there's an equivalent command for "systemd-analyze"  that breaks down the
timings for each components of shutting down, that may be able to identify
something
David FavorFractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
More difficult for shutdown process.

Possibilities...

1) In one window, do a journalctl -f to trace all systemd logging.

Then hit reboot in another window.

Note: When I do this, I also run a 2x other windows running top + pstree (in a tight while loop). Using top + pstree together during shutdown works well to look for processes failing to stop correctly.

Note: When you do this, you'll have to run a custom sshd on some non-22 port, as sshd is one of the first processes killed during a shutdown, so you must run a custom copy which will live till... the very end of the shutdown process...

2) https://wiki.freedesktop.org/www/Software/systemd/Debugging/#index2h1 may assist.