Vmware vcenter alarms - vmdisklatencyalarm

I keep having a recurring issue that I thought I might get some help from here.

I have A vcenter 6.0 cluster with 3 hosts and approximately 180 VMs. We are in a VDI environment. Horizon View is version 6.1. We use a Dell m1000e chassis with m620 blades on the newest firmware accross the board. Storage is a Nimble CS420-X8 on version 2.2.8. I have Veeam backups to a seperate san and vcloud air as DR (but that is shut off right now, except for a couple of replicaitons).

Every few hours (not the same time of the day nor the same ammount of times a day) we get a major cpu utilization spike on our Nimble as well as our Hosts. This causes the above error: alarm.vmdisklatencyalarm accross multiple guests, sometimes as few as 1 or 2 but usually it's about 60-80.

I have tried turning replication off, backups off, checked against symantec endpoint protectoin, updated firmware, updated software, added the nimble connection manager - everything i can think of and I can't crack it.

Has anyone seen anything like this or have any insight?

I can add details as needed.
tommy porterAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Is this a new implementation ?

Did Nimble size your installation for 180 concurrent virtual machines ?

You've not got virus scanning running a scan, or all clients receiving an email from management at the same time, similar activities all occuring across all VDI machines.

It would seem you are suffering from a broadcast storm, e.g. your datastore cannot cope with the demand, hence the high CPU on the Nimble storage, which will effect the VMs, and cause a latency issues.

We have seen this with all technologies, when VDI causes a broadcast storm, e.g. swaps the datastore with I/O requests,

Nimble Support wll be you best course of action, to diagnose the issue.
tommy porterAuthor Commented:
I have tickets open with all the vendors and they haven't been able to find anything yet.  Just trying all options.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Is performance affected on the VMs ?

Often high loads, and high latency is to be expected.....and is just informative, if service is delivered correctly.

which is why we asked, did Nimble Pre-Sales size accordingly.

and is this a new implementation.
tommy porterAuthor Commented:
I will notice a minute or so of lag before the errors on the VMs when it happens during the day, no clue at night.

I checked with Nimble and they confirmed that it is sized accordingly.

I've only been aware of this as an issue for the last couple months, and we upgraded in June
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you have 180 concurrent VMs in use at this time ?

high CPU on he storage, is a killer and is the issue, which suggests under sized.

I would escalate and contact your TAM at Nimble!

Give them a @tweet on @twitter, that usually get's them pretty responsive, especially if you include competitors in the mix as well!!!!
tommy porterAuthor Commented:
There was a symantec job that was scanning every time that an update came accross - this was causing major load on the entire environment. After I removed Symantec from all the images, it stopped. When i re-installed symantec i tweaked the settings per their VDI best practices.

https://support.symantec.com/en_US/article.TECH173650.html.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
tommy porterAuthor Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for tommymoto215's comment #a41492327

for the following reason:

it's the right solution
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
First response:- I quote

evidence.

"You've not got virus scanning running a scan, or all clients receiving an email from management at the same time, similar activities all occuring across all VDI machines. "

http:#a40946946

Symantec Endpoint is a Virus scanning engine.

I rest my case.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.