Solved

NFS client issues connecting to and unmounting NFS server

Posted on 2014-10-31
29
196 Views
Last Modified: 2014-11-30
I recently upgraded my kickstart server from RHEL4.5 to CentOS 6.5.  It used to be fine but now NFS is slow, it freezes and clients cannot unmount.

exports file:
# cat /etc/exports
/opt/kickstart              *(ro,sync)

I also tried but this make no difference:
# cat /etc/exports
/opt/kickstart              *(ro,sync,fsid=0,no_subtree_check,no_root_squash)

client:
# mount
10.10.12.254:/opt/kickstart on /opt/kickstart type nfs  (rw,nolock,udp,vers=4,addr=10.10.12.254,clientaddr=10.10.12.230)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy

I'm not seeing much that helps in logs.

Anyone any ideas on this?
0
Comment
Question by:mickt
  • 16
  • 13
29 Comments
 
LVL 7

Expert Comment

by:Stampel
ID: 40415417
Hello,
Could you use lsof command to see if kickstart is in use ?
0
 

Author Comment

by:mickt
ID: 40415425
# lsof | grep kickstart
lsof       1104      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
grep       1105      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
lsof       1106      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
ntpd       1981       ntp   19u     IPv4              12671      0t0        UDP kickstart:ntp
ntpd       1981       ntp   20u     IPv4             886021      0t0        UDP kickstart13:ntp
ntpd       1981       ntp   23u     IPv4             886051      0t0        UDP kickstart16:ntp
sshd       4796      root    3r     IPv4             867621      0t0        TCP kickstart:ssh->10.10.14.22:62061 (ESTABLISHED)
bash       4798      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
sshd      18793      root    3r     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
sshd      18795   brendan    3u     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
bash      18796   brendan  cwd       DIR              253,1     4096    6423266 /opt/kickstart/mnt/5.7
0
 

Author Comment

by:mickt
ID: 40415428
# showmount -a
All mount points on kickstart:
10.10.12.16:/opt/kickstart/CENTOS7.0-1406-x86_64-ES
10.10.13.131:/opt/kickstart/RHEL5.8-x86_64-ES/Server
10.10.13.82:/opt/kickstart

I also ran scripts on two other servers (10.10.12.222 & 230) and they are still showing mounted client side but not shown here on the server.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415432
could you logout/kill bash sessions inside /opt/kickstart maybe ?

And would umount force make a difference ?
umount -f /opt/kickstart

Or a lazy umount
umount -l
0
 

Author Comment

by:mickt
ID: 40415444
# umount -f /opt/kickstart
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy

and -l doesn't exist.

umounting is less an issue than the freezing as it appears that connection falls of the planet and then cannot be unmounted on client with reboot.
0
 

Author Comment

by:mickt
ID: 40415456
that should read without a reboot.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415511
Could you try to mount your NFS datastore with -o nobarrier option ?
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415521
Are you using hard mount or soft mount ?
When using hard mount if network is unavailable, application freezing is the expected behaviour but it should resume when the NFS server comes back. Hard mount is strongly suggested.

If mounts are not important like read only, you may use soft mount and see if it makes a difference.
0
 

Author Comment

by:mickt
ID: 40415684
-o nobarrier returns:
mount.nfs: an incorrect mount option was specified
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415694
This nobarrier flag is to use on the (supposed ext4) filesystem of your NFS datastore : the filesystem where you store files on the server.
0
 

Author Comment

by:mickt
ID: 40415734
/opt is a partition and /opt/kickstart is the NFS share.

from fstab:
/dev/mapper/optvg-lv_opt /opt                    ext4    defaults        1 2

How can it be set for /opt/kickstart?
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415743
You woud have to use it for all /opt
If you do have other things running under opt better mount directly your NFS under /opt/kickstart
or create another lv to perform the test.
0
 

Author Comment

by:mickt
ID: 40415756
I think the soft option may do the trick.  I tried one execution there and it succeeded without issue.  These are temporary mounts but many so I'll try a few dozen and see how it works.
0
 

Author Comment

by:mickt
ID: 40415767
next one has the same issue:

10.10.12.254:/opt/kickstart on /opt/kickstart type nfs (rw,nolock,udp,soft,vers=4,addr=10.10.12.254,clientaddr=10.10.12.222)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy
0
Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

 
LVL 7

Expert Comment

by:Stampel
ID: 40415768
be sure not to allow writes on your NFS if using soft mount.
That could lead to corruption.

what OS/distrib are the client using ? Cause even my Fedora 8 has umount -l flag :

[root@ ~]# umount -V
umount (util-linux-ng 2.13.1)

[root@ ~]# cat /etc/fedora-release
Fedora release 8 (Werewolf)

[root@ ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore.  (Requires kernel 2.4.11or later.)
0
 

Author Comment

by:mickt
ID: 40415868
added ro as option in addition to the soft option and had a couple more successful executions.  I'll see how it goes next week and get back.
0
 

Author Comment

by:mickt
ID: 40415877
# umount -V
umount (util-linux-ng 2.17.2)

I have many client versions but mostly versions of RHEL/CentOS.  The only thing that has changed is the server OS.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415893
On my Centos 5 the -l flag exists :
[root@centos5 ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem ...
0
 

Author Comment

by:mickt
ID: 40418947
I tried nobarrier option for /opt with no success.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40418951
At some point you got it working with the soft option, right ?
What is the difference between the 2 systems ?
Did you try a few dozen as you wished to ?
0
 

Author Comment

by:mickt
ID: 40418954
I tried a few but it failed quickly.  The only difference between old and new is the kickstart server OS.  I'm looking at NFS versions now.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40418956
Do you have the possibility to test upgrading the server to 6.6 and get latest updates ?
0
 

Author Comment

by:mickt
ID: 40418966
I can but I'd prefer not to, if possible.  May go to 7 if I upgrade.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40418995
Not possible only for a test like using a VM ?
You would not notice so much changes using 6.6 than moving to 7
0
 

Accepted Solution

by:
mickt earned 0 total points
ID: 40419130
Seems like forcing to use V3 has resolved the issue.

Resolution:

on server edit /etc/sysconfig/nfs:
RPCNFSDARGS="-N 4"

on client mount with '-o vers=3'

Perhaps there is a resolution for using v4?  But it doesn't matter so much for my system.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40419382
NFS v4 is Inspired by the AFS protocol  and marks a full break with previous versions.
The entire protocol has been redesigned and the code rewritten.
Using v3 instead of v4 is kinda use a different program but as far as you get it work this way its ok i guess :)
0
 

Author Comment

by:mickt
ID: 40421123
I presume it is fully compatible with versions on all older OSes?
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40421224
Yes better compatibility with version 3.
I also had to use version 3 on Raspbian to mount ReadyNAS NV+ on my Raspberry Pi.
0
 

Author Closing Comment

by:mickt
ID: 40472324
Worked around the issue.
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
I had an issue with InstallShield not being able to use Computer Browser service on Windows Server 2012. Here is the solution I found.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now