MSA P2000 G3 - Scrub-vdisk

All,

I was notified of an error on our SAN

Event:

A scrub-vdisk job completed. Errors were found. (number of mirror-verify errors found: 140414, number of media errors found: 0) (vdisk: CYHQSAN01VD02, SN: 00c0ff1a41360000865d8b5200000000)

EVENT ID:#B1340

EVENT CODE:207

EVENT SEVERITY:Error

I logged into the SAN and noticed that the system was in a healthy condition and thought nothing of the error.
A few days later I had to reboot a VM (running on vSphere 6) and the server wouldn’t reboot. After hours of trying to repair windows I restored the VM from a backup but this also wouldn’t boot. I restored once again but this time to another vdisk and all is ok.
Again today I had to restart a VM and the same thing happened. It appears that everything on this vdisk is corrupt.
I cant believe this has happened and the MSA does not report a hardware issue.
Has anyone come across this before? What steps can I perform to keep the data or is it a matter of restoring to another vdisk and then recreated the failed vdisk.
How can I prevent this in future? is SAN snapshots an option?

Thanks in advance


EVENT ID:#B1340

EVENT CODE:207

EVENT SEVERITY:Error

I logged into the SAN and noticed that the system was ina  healthy condition and thought nothing of the error.

A few days later I had to reboot a VM (running on vSphere 6) and the server wouldnt reboot. After hours of trying to repair windows i restored the VM from a backup but this also wouldnt boot. I restored once again but this time to another vdisk and all is ok.
Again today I had to restart a VM and the same thing happened. It appears that everything on this vdisk is currupt.

I cant believe this has happened and the MSA does not report a hardware issue.
HAs any one come across this before. What steps can i perfom to keep the data or is it a matter of restoring to another vdisk and then recreated the failed vdisk.

How can i prevent this in future? is SAN snapshots an option?

Thanks in advance
Rio_10Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, we have seen these errors, I would escalate to HP Storage immediately, and also make sure you are running correct firmwares on controllers and disks.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
andyalderSaggar maker's framemakerCommented:
Any disk or controller replaced recently?
Rio_10Author Commented:
Thanks for the reply, HP have the logs and are investigating. What a day!
Had to restore 9 VMs from the weekends backup, obviously to a different SAN.
Nothing has changed.

Has anyone come across SAN snapshots? they are not configured, I`m wondering if they would be beneficial to use, I could of restored to a previous snapshot to save the current state of the VM rather than losing 2 days worth of data.
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
They are of no benefit if you SAN fails, you need another SAN to send/replicate/mirror the snapshots

We feel your pain.

Been there with at least 5 clients with HP MSA, with a total restore of VMs, our clients no longer use HP.
Rio_10Author Commented:
Are you unable to snapshot to another vdisk on the same SAN?

Its the first time that i have had an issue with HP, but you invest in  having redundant controllers and RAID and then rhis happens. How can the SAN report normal health is beyond me.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Snapshots exist in the volume/vdisk. (same location as the original data)
andyalderSaggar maker's framemakerCommented:
You can ship the snapshots to a second remote mirror of the vdisk, but obviously you would have to have already set that up previously.
Rio_10Author Commented:
I have been working with HP on this issue.
I performed a firmware update over the weekend but the error is still there.

I now have a comact flash error and i will be shutting down and reseating the controllers as per there request to see if it clears the error.
Rio_10Author Commented:
Replaced CF cards and vdisk is ok now, however we had to restore from backups as all the VMs on that vdisk would not boot
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage

From novice to tech pro — start learning today.