We help IT Professionals succeed at work.
Private

What can I possibly be doing wrong with this HA design for NFS?

60 Views
Last Modified: 2020-02-25
Hi Experts!

I've been trying to troubleshoot an issue with this setup without success, hopefully you can help me.

With this current setup (diagram attached), I can perform a NAS failover from storage controller SPA to SPB but then I cannot failback from SPB to SPA. During the failback the datastore mounted on ESXi becomes inaccessible, if I failover the NAS again from SPA to SPB the datastore becomes available again automatically.

Here are the scenarios which worked for me, one test at a time:
1) Turning off switch 1, the datastore on ESXi remains mounted, good, turn it back on.
2) Turning off switch 2, the datastore on ESXi remains mounted, good, turn it back on.
3) Failing over the NAS on storage controller SPA to SPB, the datastore on ESXi remains mounted, good.

Here's the scenario I'm having trouble with:
1) Failing back the NAS on storage controller SPB to SPA after a successful failover from the above scenario

I've also noticed that if I leave the NAS server long enough on the SPB I can eventually failback to SPA without issues, but then it reverses path, I cannot go back to SPA. It seems to only failover once, but never failback unless I give it enough time.


Port-channels and vPC are up successfully without inconsistencies.

The switches are Nexus 3k. Storage is EMC Unity.


Any thoughts on what else I can check/test/do? Much appreciated, thank you!


~ Rich



diagram
Comment
Watch Question

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
It suggests your reference on the VMware host is tied to spB based reference.
This is an iscsi based access or FC?
Do you have powerpath to manage the multipath access?

Author

Commented:
This is a NFS, file based access. No iscsi or FC.

Additionally, when the datastore is inaccessible on VMware when a failback happens, the host can ping the NAS server and the ARP immediately switchs to SPA's MAC address, which tells me it's not a preferred path as it can reach SPA.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Does your NFS partition definition include both paths on a single line

You are using the single shifting ip? Double check whether the LuN in question  is setup on the spa port.

Only when spB switches to spa your access is lost.

Suggesting LUN is unavailable on spa.

Both processors are available on the switches. Suggesting you have a single path to the NFS share.
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:

There always is only one MAC address associated with one IP address.   Does your storage server provide VRRP? that would be the correct protocol to failover on IP.

For the NAS there is no reason to "switch" back.  When there is no response the ARP requests by the system will eventualy recover.


The alternative is that the NAS needs to send out "gratitious" ARP when the other link goes down.. to support failover.  This is more or less a hope & prey failover.

(A ping more or less will explicitely request an ARP)  instead of after some timeout. 


The better way is VRRP where each interface has it's own IP address and if one interface fails the IP address will move over to the other interface. After an interface comes online again the VRRP management will failback according to priorities setup etc. 

This is an explicit form of failover & failback, with defined results.


ISCSI with multipath can be an alternative where each interface also get their own IP and the multipath drivers will only use valid paths. (you will have 4 paths between each unit). 

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Nfs can be defined

Server1:/sharename, server2:/sharename  nfs /mountpoint

In the setup, it sounds as the nfs is only exported from spb processor

Since spb is accessible via either switch.

Author

Commented:
@arnold
Please check the question's attached diagram which ilustrates the connections. NEXUS 1 is the preferred path, so when SPA is active the traffic flows through Port-channel 25 on port 1/25 of Nexus 1. When SPB is active the traffic flows through Port-channel 32 on port 1/32 of Nexus 2. They both have different MAC addresse but the same IP, which VMware still has access. VMware has access to both SPA and SPB.

Failback only works after about 20 minutes of being on that same path. Meaning, I can only failback from SPB to SPA after about 20 minutes of having failover from SPA to SPB. If I try to failback before the 20 minutes mark the datastore becomes inaccessible on VMware.

For your second comment, we only have 1 server and one share, like so: 192.168.200.25:/datastore01
The storage array is smart enough to export from both SPA and SPB, but only one is active at a time.

DATASTORE01 is owned by SPA, my failover test consists of changing the ownership to SPB, which works. Then failback by changing ownership to SPA, this is where it doesn't work.


@noci
VRRP is not what we're looking for here, though the EMC Unity storage handles all the failover on their side their own way. I also mentioned that the network test passed, which if a network link goes down the datastore remains accessible to the host, which is the desired outcome. Further, cannot use iSCSI here, only NFS for this setup.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Not sure why not make the datasore01 owned by both processors and let the access be determined?

you are switching ownership versus testing failover.

My prior experience dealt with LUNS and FC infrastructure where zoning was.

The failover test would be to remove the SPA feeds on the switches or from EMC unity and see whether NFS access remains.

Author

Commented:
The EMC Unity doesn't have an option to let datastore01 be owned by both processors, unlike some of the older VNXe arrays.
The manual ownership changes are to simulate the failover process and make sure the paths are working properly, according to EMC changing ownership back and forth should be sufficient.

On the other hand, our setup with iSCSI and FC (which requires zoning) works very well. For this case though, this is a new setup with NFS which is the requirement for this new setup, I know iSCSI or FC would make things easier due to multipath and other things, but can't use them :(

I still don't know why after some time I'm able to failback but not right away. I've been trying to trace the paths and look at ARP to ensure there's no cache of MAC to the IP Address and haven't found issues on that yet.

The setup is very simple, which why is killing me not knowing what is happening.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
The back and forth on ownership the delay might be within the EMC Unity side versus related to arp or host where the NFS soft/hard error threshold ...

Have not dealt with Unity and whether you can bring down the SPA versus doing ownership switches. i.e. let the failover handler deal with spa being taken down.

in short, NFS is to avoid having HBAs added into the vmware hosts...

Author

Commented:
When checking with EMC Unity team, they analyzed the storage array's log and can see that there's no issue with the manual ownership changes and that the controllers are switches pretty quick (seconds), even on the VMware side it takes 12 seconds for VMware to notice the changes, which is their heartbeat frequency time. These vendors always keep pushing to the next guy, so can't trust them 100%, I know.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
my sentiments exactly. a question to emc is whether you can test the operational failure of spa to see whether the failover will take effect. then "recover spa" then fail spb.
Get access with a 7-day free trial.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.