Solved

NFS mount error - "access denied by server"

Posted on 2014-02-14
20
15,553 Views
Last Modified: 2014-02-16
I have installed and configured NFS on CentOS 6.5.
Here is my /etc/exports file:

/home/user/share 10.10.10.0/24(rw,sync,no_subtree_check)

I edited /etc/sysconfig/nfs and uncommented the following:

LOCKD_TCPPORT=32803
LOCKD_UDPPORT=32769
MOUNTD_PORT=892

After which I added the following iptables rules:

iptables -I INPUT -m state --state NEW -p tcp -m multiport --dport 111,892,2049,32803 -s 192.168.0.0/24 -j ACCEPT
 
iptables -I INPUT -m state --state NEW -p udp -m multiport --dport 111,892,2049,32769 -s 192.168.0.0/24 -j ACCEPT

When I issue the showmount command on the NFS client I get this:

#showmount -e 10.10.10.35
Export list for 10.10.10.35:
/home/user/share 10.10.10.0/24

When I try to mount the share from the NFS client, I get the following error:

#mount -t nfs 10.10.10.35:/home/user/share /mnt/nfs/share/
mount.nfs: access denied by server while mounting 10.10.10.35:/home/user/share

I tried setting the line in /etc/exports which reads 10.10.10.0/24 to 10.10.10.34, which is the IP of the NFS client, but to no avail.

What could be the issue here?
0
Comment
Question by:proteus-IV
  • 8
  • 8
  • 4
20 Comments
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
Hello,

from the things you supplied it is really hard to tell.
I think the firewall is configured correctly, otherwise you would not get the NFS server denial.
But I will take a wild guess and point out a common issue:
Are you trying to mount a NFS share belonging to root?
If so, this is not allowed by default.
Either change the owner on the exported directory or change the export itself to allow root mounts with no_root_squash:
# cat /etc/export
/home/user/share 10.10.10.0/24(rw,sync,no_subtree_check,no_root_squash)
# exortfs -a

Open in new window


Also check the contents of the exporting servers /etc/hosts.deny and /etc/hosts.allow files. (see man host_access for more info). For testing reasons you might want to clear them out.

Then try mounting the share again.

Does this help?
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
Your exports files shares the data with 10.10.10.0/24 while your iptables rules allow access from 192.168.0.0/24

You allow attempts to NFS from 192.168.0.0/24, but only allow 10.10.10/0/24 systems NFS share access (excluding root)

The IPs/IP Range on the firewall from which you allow access need to match the Ip/IP range to which you grant NFS access.
0
 

Author Comment

by:proteus-IV
Comment Utility
The owner and group of the exported directory are both set to nfsnobody.
The user was my regular user, but I found in an article to change the owner to nfsnobody.
It didn't help.
This is what the permissions on the export dir look like:

drwxr-xr-x. 2 nfsnobody nfsnobody 4096 Feb 15 06:43 share

I also already tried adding no_root_squash, but the error persists.
And I already checked /etc/hosts.deny and /etc/hosts.allow, they have no entries besides comments.

Sorry, I copy pasted those iptables rules from a website where I was following a tutorial.
My /etc/sysconfig/iptables actually states:

-A INPUT -s 10.10.10.0/24 -p udp -m state --state NEW -m multiport --dports 111,892,2049,32769 -j ACCEPT
-A INPUT -s 10.10.10.0/24 -p tcp -m state --state NEW -m multiport --dports 111,892,2049,32803 -j ACCEPT
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
I assume you have also tried turning off the firewall on the NSF server?
Please also post a 'chkconfig --list' from that maschine.
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
do you have nfsd, rpc.statd, lockd running on the system?

use
tcpdump -n src ip

See what is going on while at the same time try to capture what data is going from the system on which you want the NFS mount.

This way you can match what you are missing.

Simpler, drop the iptabels on the centos box and see if you can mount the NFS share.

run iptables -L INPUT
trying to make sure your rules are not below an explicit deny rule.
0
 

Author Comment

by:proteus-IV
Comment Utility
When i stopped iptables I got this message when trying to mount the export:

mount.nfs: Connection timed out

I then tried starting iptables again, but got the same "mount.nfs: Connection timed out" error.

Here is the output of "chkconfig --list":

abrt-ccpp            0:off      1:off      2:off      3:on      4:off      5:on      6:off
abrtd                0:off      1:off      2:off      3:on      4:off      5:on      6:off
acpid                0:off      1:off      2:on      3:on      4:on      5:on      6:off
atd                  0:off      1:off      2:off      3:on      4:on      5:on      6:off
auditd               0:off      1:off      2:on      3:on      4:on      5:on      6:off
autofs               0:off      1:off      2:off      3:on      4:on      5:on      6:off
blk-availability      0:off      1:on      2:on      3:on      4:on      5:on      6:off
certmonger           0:off      1:off      2:off      3:on      4:on      5:on      6:off
cgconfig             0:off      1:off      2:off      3:off      4:off      5:off      6:off
cgred                0:off      1:off      2:off      3:off      4:off      5:off      6:off
cpuspeed             0:off      1:on      2:on      3:on      4:on      5:on      6:off
crond                0:off      1:off      2:on      3:on      4:on      5:on      6:off
cups                 0:off      1:off      2:on      3:on      4:on      5:on      6:off
dovecot              0:off      1:off      2:on      3:on      4:on      5:on      6:off
haldaemon            0:off      1:off      2:off      3:on      4:on      5:on      6:off
ip6tables            0:off      1:off      2:on      3:on      4:on      5:on      6:off
iptables             0:off      1:off      2:on      3:on      4:on      5:on      6:off
irqbalance           0:off      1:off      2:off      3:on      4:on      5:on      6:off
kdump                0:off      1:off      2:off      3:on      4:on      5:on      6:off
lvm2-monitor         0:off      1:on      2:on      3:on      4:on      5:on      6:off
mdmonitor            0:off      1:off      2:on      3:on      4:on      5:on      6:off
messagebus           0:off      1:off      2:off      3:off      4:off      5:off      6:off
named                0:off      1:off      2:on      3:on      4:on      5:on      6:off
netconsole           0:off      1:off      2:off      3:off      4:off      5:off      6:off
netfs                0:off      1:off      2:off      3:on      4:on      5:on      6:off
network              0:off      1:off      2:on      3:on      4:on      5:on      6:off
nfs                  0:off      1:off      2:on      3:on      4:on      5:on      6:off
nfslock              0:off      1:off      2:off      3:on      4:on      5:on      6:off
ntpd                 0:off      1:off      2:off      3:off      4:off      5:off      6:off
ntpdate              0:off      1:off      2:off      3:off      4:off      5:off      6:off
numad                0:off      1:off      2:off      3:off      4:off      5:off      6:off
oddjobd              0:off      1:off      2:off      3:off      4:off      5:off      6:off
portreserve          0:off      1:off      2:on      3:on      4:on      5:on      6:off
postfix              0:off      1:off      2:on      3:on      4:on      5:on      6:off
psacct               0:off      1:off      2:off      3:off      4:off      5:off      6:off
quota_nld            0:off      1:off      2:off      3:off      4:off      5:off      6:off
rdisc                0:off      1:off      2:off      3:off      4:off      5:off      6:off
restorecond          0:off      1:off      2:off      3:off      4:off      5:off      6:off
rngd                 0:off      1:off      2:off      3:off      4:off      5:off      6:off
rpcbind              0:off      1:off      2:on      3:on      4:on      5:on      6:off
rpcgssd              0:off      1:off      2:off      3:on      4:on      5:on      6:off
rpcsvcgssd           0:off      1:off      2:off      3:off      4:off      5:off      6:off
rsyslog              0:off      1:off      2:on      3:on      4:on      5:on      6:off
saslauthd            0:off      1:off      2:off      3:off      4:off      5:off      6:off
slapd                0:off      1:off      2:on      3:on      4:on      5:on      6:off
smartd               0:off      1:off      2:off      3:off      4:off      5:off      6:off
squid                0:off      1:off      2:on      3:on      4:on      5:on      6:off
sshd                 0:off      1:off      2:on      3:on      4:on      5:on      6:off
sssd                 0:off      1:off      2:off      3:off      4:off      5:off      6:off
sysstat              0:off      1:on      2:on      3:on      4:on      5:on      6:off
udev-post            0:off      1:on      2:on      3:on      4:on      5:on      6:off
winbind              0:off      1:off      2:off      3:off      4:off      5:off      6:off
wpa_supplicant       0:off      1:off      2:off      3:off      4:off      5:off      6:off
ypbind               0:off      1:off      2:off      3:off      4:off      5:off      6:off

How do I check whether  nfsd, rpc.statd, lockd are running?

I ran "# tcpdump -i eth0 -n src 10.10.10.34 and port 32803" and tried mounting from the NFS client but no packets were appearing on tcpdump.

Here is the output of "iptables -L INPUT":

Chain INPUT (policy DROP)
target     prot opt source               destination        
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:imap state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:pop3 state NEW,ESTABLISHED
ACCEPT     tcp  --  10.10.10.0/24        vault13.wastelands.local tcp spts:1024:65535 dpt:squid state NEW,ESTABLISHED
ACCEPT     udp  --  10.10.10.0/24        anywhere            state NEW multiport dports sunrpc,892,nfs,filenet-rpc
ACCEPT     tcp  --  10.10.10.0/24        anywhere            state NEW multiport dports sunrpc,892,nfs,32803
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
From chkconfig, you are fine.
Did you make any changes in /etc/nfsmounts?

Check the services status with
service nfs status
service nfslock status
service rpcbind status

Open in new window

if the services are running. If not start them with a 'start' or 'restart' instead of status.

If you changed /etc/exports, you either need to restart the nfs service or run 'exportfs -a' as I mentioned above to apply the changes.

Keep iptables off for debugging.

After verifing all services are running, check with netstat for listening ports:
netstat -l|grep nfs

Open in new window

should look something like this:
tcp        0      0 0.0.0.0:nfs             0.0.0.0:*               LISTEN     
tcp6       0      0 [::]:nfs                [::]:*                  LISTEN     
udp        0      0 0.0.0.0:nfs             0.0.0.0:*                          
udp6       0      0 [::]:nfs                [::]:* 

Open in new window


Also try from the client machine, try telneting into one of the TCP ports, for instance:
telnet 10.10.10.35 2049
telnet 10.10.10.35 875

Open in new window

If this does not work or time out, you have a network connection issue.
0
 

Author Comment

by:proteus-IV
Comment Utility
I don't have the file /etc/nfsmounts, only /etc/nfsmount.conf and I didn't make any changes to it.

Here is the ouput of "service nfs status":

rpc.svcgssd is stopped
rpc.mountd (pid 1561) is running...
nfsd (pid 1576 1575 1574 1573 1572 1571 1570 1569) is running...
rpc.rquotad (pid 1557) is running...

"service nfslock status":
rpc.statd (pid  1322) is running...

and "service rpcbind status":

rpcbind (pid  1296) is running...

I ran "exportfs -a" everytime I changed something in /etc/exports.

Here is the ouput of "netstat -l|grep nfs":

tcp        0      0 *:nfs                       *:*                         LISTEN      
tcp        0      0 *:nfs                       *:*                         LISTEN      
udp        0      0 *:nfs                       *:*                                    
udp        0      0 *:nfs                       *:*            

I successfully managed to establish a telnet connection from the client to the server on port 2049, but not on port 875.
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
Sorry, this was my bad; I ment tcp/111; 875 is the quotas.

In brief, here is the NFS mount procedure from the client:
1. Contact server port 111, ask for mountd port
2. Contact server port 2049, ask for nfsd port

If everything is fine on the server, the nfs part from netstat should look like this:
netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:nfs             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:40584           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:rquotad         0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:sunrpc          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:mountd          0.0.0.0:*               LISTEN  

Open in new window

0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
ps -ef | egrep '(nfs|rstat|lock)'

You need to have three processes:
nfsd (server)
rpc.statd
lockd

exportfs -a

Make sure you can mount NFS with iptables off.  Then work the iptables if there are issues after you enable them.
0
Free Gift Card with Acronis Backup Purchase!

Backup any data in any location: local and remote systems, physical and virtual servers, private and public clouds, Macs and PCs, tablets and mobile devices, & more! For limited time only, buy any Acronis backup products and get a FREE Amazon/Best Buy gift card worth up to $200!

 

Author Comment

by:proteus-IV
Comment Utility
I can telnet to port 111 on the server.

Here is the outout of "ps -ef | egrep '(nfs|rstat|lock)'":

root        22     2  0 16:57 ?        00:00:00 [kblockd/0]
root        23     2  0 16:57 ?        00:00:00 [kblockd/1]
root      1586     2  0 16:58 ?        00:00:00 [lockd]
root      1587     2  0 16:58 ?        00:00:00 [nfsd4]
root      1588     2  0 16:58 ?        00:00:00 [nfsd4_callbacks]
root      1589     2  0 16:58 ?        00:00:00 [nfsd]
root      1590     2  0 16:58 ?        00:00:00 [nfsd]
root      1591     2  0 16:58 ?        00:00:00 [nfsd]
root      1592     2  0 16:58 ?        00:00:00 [nfsd]
root      1593     2  0 16:58 ?        00:00:00 [nfsd]
root      1594     2  0 16:58 ?        00:00:00 [nfsd]
root      1595     2  0 16:58 ?        00:00:00 [nfsd]
root      1596     2  0 16:58 ?        00:00:00 [nfsd]
root      2170  1814  0 17:20 pts/1    00:00:00 egrep (nfs|rstat|lock)

Seems that tpc.statd is missing.
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
No, most likely not. The process may as well been called rpc.statd:
$ ps -ef | egrep '(nfs|rpc|lock)'
root       128   121  0  2013 ?        00:00:13 [kblockd/0]
root       129   121  0  2013 ?        00:01:47 [kblockd/1]
root       130   121  0  2013 ?        00:00:24 [kblockd/2]
root       131   121  0  2013 ?        00:00:45 [kblockd/3]
root      3131     1  0 17:35 ?        00:00:00 rpc.rquotad
root      3155   121  0 17:35 ?        00:00:00 [nfsd4]
root      3156     1  0 17:35 ?        00:00:00 [lockd]
root      3157     1  0 17:35 ?        00:00:00 [nfsd]
root      3158     1  0 17:35 ?        00:00:00 [nfsd]
root      3159     1  0 17:35 ?        00:00:00 [nfsd]
root      3160     1  0 17:35 ?        00:00:00 [nfsd]
root      3161     1  0 17:35 ?        00:00:00 [nfsd]
root      3162     1  0 17:35 ?        00:00:00 [nfsd]
root      3163     1  0 17:35 ?        00:00:00 [nfsd]
root      3164     1  0 17:35 ?        00:00:00 [nfsd]
root      3167     1  0 17:35 ?        00:00:00 rpc.mountd
rpc       9269     1  0  2013 ?        00:00:00 portmap
root      9301   121  0  2013 ?        00:00:00 [rpciod/0]
root      9302   121  0  2013 ?        00:00:00 [rpciod/1]
root      9303   121  0  2013 ?        00:00:00 [rpciod/2]
root      9304   121  0  2013 ?        00:00:00 [rpciod/3]
rpcuser   9314     1  0  2013 ?        00:00:00 rpc.statd
root      9444     1  0  2013 ?        00:00:00 rpc.idmapd

Open in new window


What about the telnet test?
I am asking because NFS is really easy to configure on the server side. There may as well be something on your client blocking the connection.

Is it possible to try the mount from another client, even the server should be able to mount it's own share.
0
 

Author Comment

by:proteus-IV
Comment Utility
I tried mounting from the server, this is what I get:

# mount -t nfs 10.0.0.2:/home/user/share /mnt/nfs/share/

mount.nfs: access denied by server while mounting 10.0.0.2:/home/user/share
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
I thought the server was Ip .35; does it has multiple ip's? This can sometimes cause problems if they are in the same subnet.
Did you already run a verbose mount, eg.

mount -v -t nfs (...)

Please post the result.
0
 
LVL 76

Accepted Solution

by:
arnold earned 500 total points
Comment Utility
usually the NFS share is in /exports/ using home/users/share you may be running into access issues there.
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
running into access issues there.
Arnold is making an excellent point: Is selinux interfering? Find out if it is enabled with 'getenforce' and set it to permissive with 'setenforce 0'
0
 

Author Comment

by:proteus-IV
Comment Utility
I changed the IP of the server because i connected it to a DMZ port on my router.
Firewal is off on the router while I am still testing.

I also tried removing and installing nfs-utils nfs-utils-libs again and now ehen I try to start
the nfs service I get the following error message:

# service nfs start
FATAL: Module nfsd not found.
FATAL: Error running install command for nfsd
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
Starting NFS daemon: rpc.nfsd: Unable to access /proc/fs/nfsd errno 2 (No such file or directory).
Please try, as root, 'mount -t nfsd nfsd /proc/fs/nfsd' and then restart rpc.nfsd to correct the problem
                                                           [FAILED]
0
 

Author Comment

by:proteus-IV
Comment Utility
selinux is Disabled.
0
 
LVL 13

Expert Comment

by:Daniel Helgenberger
Comment Utility
selinux is Disabled.
Was it off all the time?
Could it be there was an installation error all the time?
If you can try to reinstall the missing nfsd and with yum reinstall.

Maybe even setup a test VM and redo it entirely.
This is a good basic tutorial for centos/RHEL:
https://www.digitalocean.com/community/articles/how-to-set-up-an-nfs-mount-on-centos-6
0
 

Author Closing Comment

by:proteus-IV
Comment Utility
I created the directories /export/share and ran "chmod 777" on share.
Now I can mount the folder and it is syncing without issues.

Thank you for your help.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now