NFS client issues connecting to and unmounting NFS server

I recently upgraded my kickstart server from RHEL4.5 to CentOS 6.5.  It used to be fine but now NFS is slow, it freezes and clients cannot unmount.

exports file:
# cat /etc/exports
/opt/kickstart              *(ro,sync)

I also tried but this make no difference:
# cat /etc/exports
/opt/kickstart              *(ro,sync,fsid=0,no_subtree_check,no_root_squash)

client:
# mount
10.10.12.254:/opt/kickstart on /opt/kickstart type nfs  (rw,nolock,udp,vers=4,addr=10.10.12.254,clientaddr=10.10.12.230)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy

I'm not seeing much that helps in logs.

Anyone any ideas on this?
micktAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

StampelCommented:
Hello,
Could you use lsof command to see if kickstart is in use ?
0
micktAuthor Commented:
# lsof | grep kickstart
lsof       1104      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
grep       1105      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
lsof       1106      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
ntpd       1981       ntp   19u     IPv4              12671      0t0        UDP kickstart:ntp
ntpd       1981       ntp   20u     IPv4             886021      0t0        UDP kickstart13:ntp
ntpd       1981       ntp   23u     IPv4             886051      0t0        UDP kickstart16:ntp
sshd       4796      root    3r     IPv4             867621      0t0        TCP kickstart:ssh->10.10.14.22:62061 (ESTABLISHED)
bash       4798      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
sshd      18793      root    3r     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
sshd      18795   brendan    3u     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
bash      18796   brendan  cwd       DIR              253,1     4096    6423266 /opt/kickstart/mnt/5.7
0
micktAuthor Commented:
# showmount -a
All mount points on kickstart:
10.10.12.16:/opt/kickstart/CENTOS7.0-1406-x86_64-ES
10.10.13.131:/opt/kickstart/RHEL5.8-x86_64-ES/Server
10.10.13.82:/opt/kickstart

I also ran scripts on two other servers (10.10.12.222 & 230) and they are still showing mounted client side but not shown here on the server.
0
Powerful Yet Easy-to-Use Network Monitoring

Identify excessive bandwidth utilization or unexpected application traffic with SolarWinds Bandwidth Analyzer Pack.

StampelCommented:
could you logout/kill bash sessions inside /opt/kickstart maybe ?

And would umount force make a difference ?
umount -f /opt/kickstart

Or a lazy umount
umount -l
0
micktAuthor Commented:
# umount -f /opt/kickstart
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy

and -l doesn't exist.

umounting is less an issue than the freezing as it appears that connection falls of the planet and then cannot be unmounted on client with reboot.
0
micktAuthor Commented:
that should read without a reboot.
0
StampelCommented:
Could you try to mount your NFS datastore with -o nobarrier option ?
0
StampelCommented:
Are you using hard mount or soft mount ?
When using hard mount if network is unavailable, application freezing is the expected behaviour but it should resume when the NFS server comes back. Hard mount is strongly suggested.

If mounts are not important like read only, you may use soft mount and see if it makes a difference.
0
micktAuthor Commented:
-o nobarrier returns:
mount.nfs: an incorrect mount option was specified
0
StampelCommented:
This nobarrier flag is to use on the (supposed ext4) filesystem of your NFS datastore : the filesystem where you store files on the server.
0
micktAuthor Commented:
/opt is a partition and /opt/kickstart is the NFS share.

from fstab:
/dev/mapper/optvg-lv_opt /opt                    ext4    defaults        1 2

How can it be set for /opt/kickstart?
0
StampelCommented:
You woud have to use it for all /opt
If you do have other things running under opt better mount directly your NFS under /opt/kickstart
or create another lv to perform the test.
0
micktAuthor Commented:
I think the soft option may do the trick.  I tried one execution there and it succeeded without issue.  These are temporary mounts but many so I'll try a few dozen and see how it works.
0
micktAuthor Commented:
next one has the same issue:

10.10.12.254:/opt/kickstart on /opt/kickstart type nfs (rw,nolock,udp,soft,vers=4,addr=10.10.12.254,clientaddr=10.10.12.222)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy
0
StampelCommented:
be sure not to allow writes on your NFS if using soft mount.
That could lead to corruption.

what OS/distrib are the client using ? Cause even my Fedora 8 has umount -l flag :

[root@ ~]# umount -V
umount (util-linux-ng 2.13.1)

[root@ ~]# cat /etc/fedora-release
Fedora release 8 (Werewolf)

[root@ ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore.  (Requires kernel 2.4.11or later.)
0
micktAuthor Commented:
added ro as option in addition to the soft option and had a couple more successful executions.  I'll see how it goes next week and get back.
0
micktAuthor Commented:
# umount -V
umount (util-linux-ng 2.17.2)

I have many client versions but mostly versions of RHEL/CentOS.  The only thing that has changed is the server OS.
0
StampelCommented:
On my Centos 5 the -l flag exists :
[root@centos5 ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem ...
0
micktAuthor Commented:
I tried nobarrier option for /opt with no success.
0
StampelCommented:
At some point you got it working with the soft option, right ?
What is the difference between the 2 systems ?
Did you try a few dozen as you wished to ?
0
micktAuthor Commented:
I tried a few but it failed quickly.  The only difference between old and new is the kickstart server OS.  I'm looking at NFS versions now.
0
StampelCommented:
Do you have the possibility to test upgrading the server to 6.6 and get latest updates ?
0
micktAuthor Commented:
I can but I'd prefer not to, if possible.  May go to 7 if I upgrade.
0
StampelCommented:
Not possible only for a test like using a VM ?
You would not notice so much changes using 6.6 than moving to 7
0
micktAuthor Commented:
Seems like forcing to use V3 has resolved the issue.

Resolution:

on server edit /etc/sysconfig/nfs:
RPCNFSDARGS="-N 4"

on client mount with '-o vers=3'

Perhaps there is a resolution for using v4?  But it doesn't matter so much for my system.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
StampelCommented:
NFS v4 is Inspired by the AFS protocol  and marks a full break with previous versions.
The entire protocol has been redesigned and the code rewritten.
Using v3 instead of v4 is kinda use a different program but as far as you get it work this way its ok i guess :)
0
micktAuthor Commented:
I presume it is fully compatible with versions on all older OSes?
0
StampelCommented:
Yes better compatibility with version 3.
I also had to use version 3 on Raspbian to mount ReadyNAS NV+ on my Raspberry Pi.
0
micktAuthor Commented:
Worked around the issue.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.