High disk latency on one ESXi 5.1 host when running backup

I have strange problem. Few days ago I replaced 1 ESXi host in our cluster with new one.
I'm using the same HBA card, the same cables connected to same ports in Brocade switch.
I am using Veeam 8 for backup. Everything works great with virtual machines on this host, but when backup starts, disk latency is jumping up to 700 DAVG/cmd, when KAVG and QAVG are under 20.
We are using EMC VNX. In the cluster I have 2 more hosts with exactly the same configuration and the same HBA model, but the problem is only on this newest one, so it's definitely not the storage problem.
Fiber cables are tested. Zoning was configured correctly when HBA card was in old host.
I am thinking  about getting brand new HBA and configure zoning again, but it costs a lot and I don't want to spend much money if the problem is in configuration.
Any advices what should I check?
Or how to find out if this latency is zoning problem?
Tomasz CzyzSystem AdministratorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mohammed KhawajaManager - Infrastructure:  Information TechnologyCommented:
Swap HBA with another server and see if the problem follows the server, HBA or cable (assuming you have other cables or switch cables with another host).
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
check you have no dust on the FC cables or in the FC ports on the HBA!

Is the ESX host identical in configuration, memory, processors, cpu, firmware, bios, bios configuration.

FC HBA in the same slot, wrong slot, slow slot, not support slot in the server.
Tomasz CzyzSystem AdministratorAuthor Commented:
@Mohammed I would like to do that, but can't do it without downtime, 1 of 6 hosts is already down and I  can't run all VMs on 4 servers.

@Andrew Thanks for advice. I've checked FC cables with cable tester, but didn't check for dust  on FC ports.
new ESX host is HP Proliant DL360p Gen8, it replaced DL360p G6. 2 other Gen8 servers have the same configuration, only difference is +/- 20GB memory and ESXi version, problem is on 5.1 build 3070626, but others have 1743533.

Strange that even when VMs are using disk quite actively (high read and write/s), the latency is max 20, but when backup starts, it jumps up to even few thousands. Even if backup is done (it takes 2 hours instead of 10-15 minutes), host is hanging. Any command in vSphere results with timeout.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If you have a change of ESXi build, you have a completely different server!

I would check firmware of FC HBA, Storage, Fabric and cross reference with HCL for ESXi 5.1, version you are using!

or rollback to Build 1743533! (first!).

Too many changes at once!

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tomasz CzyzSystem AdministratorAuthor Commented:
I have one host working with the same build without problems, but now I see it has other type of FC HBA.
Thanks for advice, I'll try older build and let you know the results.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
so, it's all different!
Tomasz CzyzSystem AdministratorAuthor Commented:
Last few days I tried following:
- replaced HBA with the same model
- brand new fiber cables
- reconfigured zoning and storage access
- installed lower ESXi version

No change at this point, got same problem.

- forced manual reinstallation of drivers and firmware for HBA
- upgraded flash fw
- updated all firmware from latest SPP
- installed all ESX updates via Update Manager

Now it's better, I used to reproduce problem by starting VM restore to this host, and this time it ran without problems... but last night I've got alerts again that latency was high for around 10 minutes. Good thing is that after those 10 minutes everything got back to normal.

I'm going to test it with more VMs.

HBA I'm using is HPAE312A which according to Qlogic is already End-Of-Life, so I am going to buy new 8Gb dual-port HBA. Any advice for HBA compatible with Proliant DL360p gen8?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We've always preferred Emulex HBA for FC, and Qlogic for iSCSi!

What ever you use, make sure there are on the HCL.
Tomasz CzyzSystem AdministratorAuthor Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for cafejava's comment #a41225786

for the following reason:

Upgrading drives and firmware on HBA and all firmware on HP components solved the problem.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
this was stated in http:#a41200405
Tomasz CzyzSystem AdministratorAuthor Commented:
Upgrading drives and firmware on HBA and all firmware on HP components solved the problem.

Sorry I marked wrong comment as solution before
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.