Solved

ESXi to Synology NAS iSCSI slowdown out of nowhere

Posted on 2016-07-29
27
30 Views
Last Modified: 2016-11-22
Out of nowhere I'm experiencing incredibly slow connections between my esxi hosts and my synology NAS.  I have 3 hosts, 2 Mac Minis running in HA and a Dell Server.  My Synology NAS is on the latest firmware with 3 of the links link-aggregated.  I noticed while copying files to one of my VMs over the network (the VM had a share going over AFP) that I was getting maybe 12 MB/s.  Then I noticed some of my other systems were quite sluggish.  I tried creating a new iSCSI LUN with some drives that have not been used on the NAS.  While it did show up on my esxi hosts and I was able to format it, migrating to it has taken too long and keeps timing out.  I've tried restarting the NAS units and the hosts.  All drives in the NAS appear to be healthy.  No configuration changes have been made for at least a month or two but I did check all settings in esxi and on my network switch to no avail.

Any ideas?
0
Comment
Question by:huntson
  • 14
  • 10
  • 3
27 Comments
 
LVL 118
ID: 41734518
any configuration changes, are you using jumbo frames ?

checked all equipment is configured for jubmo frames
0
 
LVL 30

Expert Comment

by:renazonse
ID: 41734521
Are you seeing any packet loss? I'd check your switch configuration to make sure that's still in tact and confirm 100% nothing has grabbed the same IP of the NAS. I had a very similar issue with a NAS for a customer who could never get it working properly where the same IP was assigned to a rogue device.
0
 

Author Comment

by:huntson
ID: 41734528
Thanks for the quick reply.  As was stated in my second to last sentence - no changes were made in at least two months.

I am using jumbo frames and they are enabled everywhere.  I tried turning them off but the Synology Units won't allow me to without breaking apart the cluster and then putting it back together - not something I am keen on doing seeing as this all worked fine and FAST a few days ago.
0
 

Author Comment

by:huntson
ID: 41734529
I'll have to shut down the NAS to test and see if I can ping anything else on the network.  I doubt that's the case but I'll get back to you shortly.
0
 
LVL 118
ID: 41734541
time for some vmkpings from your hosts to your NAS

can you screenshot your  networking on the hosts

not upgraded DSM at all ?
0
 

Author Comment

by:huntson
ID: 41734543
There is no other unit on the network with those IPs.
0
 

Author Comment

by:huntson
ID: 41734546
I have not upgraded DSM on the server in about a month or two as I stated.  Attached are some screenshots from my Dell system.   I'll admit I'm not necessarily the most experienced esxi guy so can you explain vmkpings please?
shot1.PNG
snip-2.PNG
0
 
LVL 30

Expert Comment

by:renazonse
ID: 41734547
Although no changes haven't been physically made I've seen switches lose configuration data in various situations. Confirming the switch configs is definitely the way to go. In esx make sure all the paths to the NAS are live.
0
 

Author Comment

by:huntson
ID: 41734551
Indeed I have checked.  All the ports are set for jumbo frames, the LAGS are up, and vlans are all flat.
0
 
LVL 30

Expert Comment

by:renazonse
ID: 41734554
Is there any packet loss when you start a running ping to the NAS?
0
 

Author Comment

by:huntson
ID: 41734564
Is this the vmkping run from an ssh terminal on the host?
0
 
LVL 118
ID: 41734605
yes, it's the equivalent of ping, but it's a special version which sends traffic via the vmkernel port groups.

so provides a better valid test of reaching the SAN
0
 

Author Comment

by:huntson
ID: 41734615
I see.  Had to look it up.  it's seeing the NAS fine but I can't seem to find an indefinite command - do you know the flag to keep it pinginging until I stop it?

Also I noticed that on my Macmini the jumbo frames aren;'t passing despite the setting.  I'm not really worried about that.  Just an FYI.
0
U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

 
LVL 118
ID: 41734630
could be a sign of mismatched jumbo frames ?
0
 

Author Comment

by:huntson
ID: 41734753
I just remembered what I did to potentially mess things up.  Hate when you forget just when you need to remember!!!

Only 4 TB of the total 5tb of the volume was provisioned.  I thin provisioned an additional TB and added that space on the VMware side.  The migration went well.  Any idea why this would cause that problem?
0
 
LVL 118
ID: 41734791
Is your Synology on the VMware HCL and Supported ?
0
 

Author Comment

by:huntson
ID: 41734800
It is indeed.
0
 
LVL 118
ID: 41734871
have you used all the space on the NAS now ?

e.g. the NAS volume is completely full, as you expanded the LUN, e.g. thin ?
0
 

Author Comment

by:huntson
ID: 41734881
Yes.  I provisioned all the space.
0
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 41734908
Yes, I see to remember an issue with Synology NASes, when you allocate all the space to a LUN, which is thin, provisioned, it goes offline, into a read only mode.

e.g. ALL Volume space, the LUN goes read only, and does not get mounted for protection.

this does appear catch-22, if you have VMs, because you would have to add more physical disk to expand the volume on the NAS.

let me try and find the issue, we had with this before.
0
 
LVL 118
ID: 41734999
0
 
LVL 118
ID: 41735267
was the corruption the cause of the slow down?
0
 

Author Comment

by:huntson
ID: 41735280
This is to be determined.  Once I recreate the volume we can be certain.  Give me a few days.
0
 
LVL 118
ID: 41735324
I think I would pause, and get some backups!
0
 

Author Comment

by:huntson
ID: 41742220
After working on my backup solution and re configuring some items I'm still having speed issues.   I moved everything temporarily to LUNs on another system and then removed all the volumes on the system I was working on and re-created them.

While copying files back through vcenter, according to the resource monitor in Synology I'm only seeing write speeds to the unit approaching 10 MB/s.  Using simple SMB or AFP to shares on the Synology box yields results approaching 80 MB/s.  I forgot what I had done in the path however I'm not sure if I should have a separate target for each system that wants to connect or if I should allow multiple initiators to connect to one target.


Let me know if I can check any other settings to help improve this performance.
0
 

Accepted Solution

by:
huntson earned 0 total points
ID: 41745714
While I still don't have an answer if it's best to have multiple initiators connect to the same target or have separate targets for each initiator I think I found the primary cause of the slow-down.  In addition to the corruption that was experienced/discussed early-on, it would appear that using a file-based LUN is the fastest performer on Synology systems.  This is exactly the opposite of what the prompts tell you (block or volume-based LUNS, according to the UI, perform faster).
0
 

Author Closing Comment

by:huntson
ID: 41772802
I troubleshot and found the issue through repeated testing.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
Finding original email is quite difficult due to their duplicates. From this article, you will come to know why multiple duplicates of same emails appear and how to delete duplicate emails from Outlook securely and instantly while vital emails remai…
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…

912 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now