Solved

NFS client issues connecting to and unmounting NFS server

Posted on 2014-10-31
29
210 Views
Last Modified: 2014-11-30
I recently upgraded my kickstart server from RHEL4.5 to CentOS 6.5.  It used to be fine but now NFS is slow, it freezes and clients cannot unmount.

exports file:
# cat /etc/exports
/opt/kickstart              *(ro,sync)

I also tried but this make no difference:
# cat /etc/exports
/opt/kickstart              *(ro,sync,fsid=0,no_subtree_check,no_root_squash)

client:
# mount
10.10.12.254:/opt/kickstart on /opt/kickstart type nfs  (rw,nolock,udp,vers=4,addr=10.10.12.254,clientaddr=10.10.12.230)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy

I'm not seeing much that helps in logs.

Anyone any ideas on this?
0
Comment
Question by:mickt
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 16
  • 13
29 Comments
 
LVL 7

Expert Comment

by:Stampel
ID: 40415417
Hello,
Could you use lsof command to see if kickstart is in use ?
0
 

Author Comment

by:mickt
ID: 40415425
# lsof | grep kickstart
lsof       1104      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
grep       1105      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
lsof       1106      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
ntpd       1981       ntp   19u     IPv4              12671      0t0        UDP kickstart:ntp
ntpd       1981       ntp   20u     IPv4             886021      0t0        UDP kickstart13:ntp
ntpd       1981       ntp   23u     IPv4             886051      0t0        UDP kickstart16:ntp
sshd       4796      root    3r     IPv4             867621      0t0        TCP kickstart:ssh->10.10.14.22:62061 (ESTABLISHED)
bash       4798      root  cwd       DIR              253,1     4096    6422601 /opt/kickstart/mnt/scripts
sshd      18793      root    3r     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
sshd      18795   brendan    3u     IPv4             753509      0t0        TCP kickstart:ssh->10.10.14.9:51095 (ESTABLISHED)
bash      18796   brendan  cwd       DIR              253,1     4096    6423266 /opt/kickstart/mnt/5.7
0
 

Author Comment

by:mickt
ID: 40415428
# showmount -a
All mount points on kickstart:
10.10.12.16:/opt/kickstart/CENTOS7.0-1406-x86_64-ES
10.10.13.131:/opt/kickstart/RHEL5.8-x86_64-ES/Server
10.10.13.82:/opt/kickstart

I also ran scripts on two other servers (10.10.12.222 & 230) and they are still showing mounted client side but not shown here on the server.
0
Easy, flexible multimedia distribution & control

Coming soon!  Ideal for large-scale A/V applications, ATEN's VM3200 Modular Matrix Switch is an all-in-one solution that simplifies video wall integration. Easily customize display layouts to see what you want, how you want it in 4k.

 
LVL 7

Expert Comment

by:Stampel
ID: 40415432
could you logout/kill bash sessions inside /opt/kickstart maybe ?

And would umount force make a difference ?
umount -f /opt/kickstart

Or a lazy umount
umount -l
0
 

Author Comment

by:mickt
ID: 40415444
# umount -f /opt/kickstart
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy
umount2: Device or resource busy
umount.nfs: /opt/kickstart: device is busy

and -l doesn't exist.

umounting is less an issue than the freezing as it appears that connection falls of the planet and then cannot be unmounted on client with reboot.
0
 

Author Comment

by:mickt
ID: 40415456
that should read without a reboot.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415511
Could you try to mount your NFS datastore with -o nobarrier option ?
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415521
Are you using hard mount or soft mount ?
When using hard mount if network is unavailable, application freezing is the expected behaviour but it should resume when the NFS server comes back. Hard mount is strongly suggested.

If mounts are not important like read only, you may use soft mount and see if it makes a difference.
0
 

Author Comment

by:mickt
ID: 40415684
-o nobarrier returns:
mount.nfs: an incorrect mount option was specified
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415694
This nobarrier flag is to use on the (supposed ext4) filesystem of your NFS datastore : the filesystem where you store files on the server.
0
 

Author Comment

by:mickt
ID: 40415734
/opt is a partition and /opt/kickstart is the NFS share.

from fstab:
/dev/mapper/optvg-lv_opt /opt                    ext4    defaults        1 2

How can it be set for /opt/kickstart?
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415743
You woud have to use it for all /opt
If you do have other things running under opt better mount directly your NFS under /opt/kickstart
or create another lv to perform the test.
0
 

Author Comment

by:mickt
ID: 40415756
I think the soft option may do the trick.  I tried one execution there and it succeeded without issue.  These are temporary mounts but many so I'll try a few dozen and see how it works.
0
 

Author Comment

by:mickt
ID: 40415767
next one has the same issue:

10.10.12.254:/opt/kickstart on /opt/kickstart type nfs (rw,nolock,udp,soft,vers=4,addr=10.10.12.254,clientaddr=10.10.12.222)

# umount /opt/kickstart
umount.nfs: /opt/kickstart: device is busy
umount.nfs: /opt/kickstart: device is busy
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415768
be sure not to allow writes on your NFS if using soft mount.
That could lead to corruption.

what OS/distrib are the client using ? Cause even my Fedora 8 has umount -l flag :

[root@ ~]# umount -V
umount (util-linux-ng 2.13.1)

[root@ ~]# cat /etc/fedora-release
Fedora release 8 (Werewolf)

[root@ ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore.  (Requires kernel 2.4.11or later.)
0
 

Author Comment

by:mickt
ID: 40415868
added ro as option in addition to the soft option and had a couple more successful executions.  I'll see how it goes next week and get back.
0
 

Author Comment

by:mickt
ID: 40415877
# umount -V
umount (util-linux-ng 2.17.2)

I have many client versions but mostly versions of RHEL/CentOS.  The only thing that has changed is the server OS.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40415893
On my Centos 5 the -l flag exists :
[root@centos5 ~]# man umount | grep "\-l"
       -l     Lazy unmount. Detach the filesystem from the filesystem ...
0
 

Author Comment

by:mickt
ID: 40418947
I tried nobarrier option for /opt with no success.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40418951
At some point you got it working with the soft option, right ?
What is the difference between the 2 systems ?
Did you try a few dozen as you wished to ?
0
 

Author Comment

by:mickt
ID: 40418954
I tried a few but it failed quickly.  The only difference between old and new is the kickstart server OS.  I'm looking at NFS versions now.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40418956
Do you have the possibility to test upgrading the server to 6.6 and get latest updates ?
0
 

Author Comment

by:mickt
ID: 40418966
I can but I'd prefer not to, if possible.  May go to 7 if I upgrade.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40418995
Not possible only for a test like using a VM ?
You would not notice so much changes using 6.6 than moving to 7
0
 

Accepted Solution

by:
mickt earned 0 total points
ID: 40419130
Seems like forcing to use V3 has resolved the issue.

Resolution:

on server edit /etc/sysconfig/nfs:
RPCNFSDARGS="-N 4"

on client mount with '-o vers=3'

Perhaps there is a resolution for using v4?  But it doesn't matter so much for my system.
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40419382
NFS v4 is Inspired by the AFS protocol  and marks a full break with previous versions.
The entire protocol has been redesigned and the code rewritten.
Using v3 instead of v4 is kinda use a different program but as far as you get it work this way its ok i guess :)
0
 

Author Comment

by:mickt
ID: 40421123
I presume it is fully compatible with versions on all older OSes?
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40421224
Yes better compatibility with version 3.
I also had to use version 3 on Raspbian to mount ReadyNAS NV+ on my Raspberry Pi.
0
 

Author Closing Comment

by:mickt
ID: 40472324
Worked around the issue.
0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

#Citrix #Citrix Netscaler #HTTP Compression #Load Balance
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question