Solved

NFS client issues connecting to and unmounting NFS server

Posted on 2014-10-31
29
194 Views
Last Modified: 2014-11-30
I recently upgraded my kickstart server from RHEL4.5 to CentOS 6.5.  It used to be fine but now NFS is slow, it freezes and clients cannot unmount.

exports file:
# cat /etc/exports
/opt/kickstart              *(ro,sync)

I also tried but this make no difference:
# cat /etc/exports
/opt/kickstart              *(ro,sync,fsid=0,no_subtree_check,no_root_squash)

client:
# mount
10.10.12.254:/opt/kickstart on /opt/kickstart type nfs  (rw,nolock,udp,vers=4,addr=10.10.12.254,clientaddr=10.10.12.230)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy

I'm not seeing much that helps in logs.

Anyone any ideas on this?
0
Comment
Question by:mickt
  • 16
  • 13
29 Comments
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
Hello,
Could you use lsof command to see if kickstart is in use ?
0
 

Author Comment

by:mickt
Comment Utility
# lsof | grep kickstart
lsof       1104      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
grep       1105      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
lsof       1106      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
ntpd       1981       ntp   19u     IPv4              12671      0t0        UDP kickstart:ntp
ntpd       1981       ntp   20u     IPv4             886021      0t0        UDP kickstart13:ntp
ntpd       1981       ntp   23u     IPv4             886051      0t0        UDP kickstart16:ntp
sshd       4796      root    3r     IPv4             867621      0t0        TCP kickstart:ssh->10.10.14.22:62061 (ESTABLISHED)
bash       4798      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
sshd      18793      root    3r     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
sshd      18795   brendan    3u     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
bash      18796   brendan  cwd       DIR              253,1     4096    6423266 /opt/kickstart/mnt/5.7
0
 

Author Comment

by:mickt
Comment Utility
# showmount -a
All mount points on kickstart:
10.10.12.16:/opt/kickstart/CENTOS7.0-1406-x86_64-ES
10.10.13.131:/opt/kickstart/RHEL5.8-x86_64-ES/Server
10.10.13.82:/opt/kickstart

I also ran scripts on two other servers (10.10.12.222 & 230) and they are still showing mounted client side but not shown here on the server.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
could you logout/kill bash sessions inside /opt/kickstart maybe ?

And would umount force make a difference ?
umount -f /opt/kickstart

Or a lazy umount
umount -l
0
 

Author Comment

by:mickt
Comment Utility
# umount -f /opt/kickstart
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy

and -l doesn't exist.

umounting is less an issue than the freezing as it appears that connection falls of the planet and then cannot be unmounted on client with reboot.
0
 

Author Comment

by:mickt
Comment Utility
that should read without a reboot.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
Could you try to mount your NFS datastore with -o nobarrier option ?
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
Are you using hard mount or soft mount ?
When using hard mount if network is unavailable, application freezing is the expected behaviour but it should resume when the NFS server comes back. Hard mount is strongly suggested.

If mounts are not important like read only, you may use soft mount and see if it makes a difference.
0
 

Author Comment

by:mickt
Comment Utility
-o nobarrier returns:
mount.nfs: an incorrect mount option was specified
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
This nobarrier flag is to use on the (supposed ext4) filesystem of your NFS datastore : the filesystem where you store files on the server.
0
 

Author Comment

by:mickt
Comment Utility
/opt is a partition and /opt/kickstart is the NFS share.

from fstab:
/dev/mapper/optvg-lv_opt /opt                    ext4    defaults        1 2

How can it be set for /opt/kickstart?
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
You woud have to use it for all /opt
If you do have other things running under opt better mount directly your NFS under /opt/kickstart
or create another lv to perform the test.
0
 

Author Comment

by:mickt
Comment Utility
I think the soft option may do the trick.  I tried one execution there and it succeeded without issue.  These are temporary mounts but many so I'll try a few dozen and see how it works.
0
 

Author Comment

by:mickt
Comment Utility
next one has the same issue:

10.10.12.254:/opt/kickstart on /opt/kickstart type nfs (rw,nolock,udp,soft,vers=4,addr=10.10.12.254,clientaddr=10.10.12.222)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 7

Expert Comment

by:Stampel
Comment Utility
be sure not to allow writes on your NFS if using soft mount.
That could lead to corruption.

what OS/distrib are the client using ? Cause even my Fedora 8 has umount -l flag :

[root@ ~]# umount -V
umount (util-linux-ng 2.13.1)

[root@ ~]# cat /etc/fedora-release
Fedora release 8 (Werewolf)

[root@ ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore.  (Requires kernel 2.4.11or later.)
0
 

Author Comment

by:mickt
Comment Utility
added ro as option in addition to the soft option and had a couple more successful executions.  I'll see how it goes next week and get back.
0
 

Author Comment

by:mickt
Comment Utility
# umount -V
umount (util-linux-ng 2.17.2)

I have many client versions but mostly versions of RHEL/CentOS.  The only thing that has changed is the server OS.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
On my Centos 5 the -l flag exists :
[root@centos5 ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem ...
0
 

Author Comment

by:mickt
Comment Utility
I tried nobarrier option for /opt with no success.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
At some point you got it working with the soft option, right ?
What is the difference between the 2 systems ?
Did you try a few dozen as you wished to ?
0
 

Author Comment

by:mickt
Comment Utility
I tried a few but it failed quickly.  The only difference between old and new is the kickstart server OS.  I'm looking at NFS versions now.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
Do you have the possibility to test upgrading the server to 6.6 and get latest updates ?
0
 

Author Comment

by:mickt
Comment Utility
I can but I'd prefer not to, if possible.  May go to 7 if I upgrade.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
Not possible only for a test like using a VM ?
You would not notice so much changes using 6.6 than moving to 7
0
 

Accepted Solution

by:
mickt earned 0 total points
Comment Utility
Seems like forcing to use V3 has resolved the issue.

Resolution:

on server edit /etc/sysconfig/nfs:
RPCNFSDARGS="-N 4"

on client mount with '-o vers=3'

Perhaps there is a resolution for using v4?  But it doesn't matter so much for my system.
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
NFS v4 is Inspired by the AFS protocol  and marks a full break with previous versions.
The entire protocol has been redesigned and the code rewritten.
Using v3 instead of v4 is kinda use a different program but as far as you get it work this way its ok i guess :)
0
 

Author Comment

by:mickt
Comment Utility
I presume it is fully compatible with versions on all older OSes?
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
Yes better compatibility with version 3.
I also had to use version 3 on Raspbian to mount ReadyNAS NV+ on my Raspberry Pi.
0
 

Author Closing Comment

by:mickt
Comment Utility
Worked around the issue.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Suggested Solutions

Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now