Solved

Lefthand P4300 and VMWare vSphere 4 Issue

Posted on 2012-04-11
21
1,116 Views
Last Modified: 2016-11-23
Maybe some of you guys can weigh in on this:

Environment:
Lefthand p4300 starter array with two storage nodes (lefthand1 and lefthand2) set up with two-way replication.  In the lefthand management console, I can see lefthand1, but lefthand2 only shows as its private IP and is unreachable.  Last night, around 10CST, it appears a double drive failure of lefthand2 occurred.  This is a RAID 5 array.

Three Dell R710's serve as hosts (vmhost1, vmhost2, vmhost3) and a fourth R710 serves as the vcenter server.

vSphere 4 is running on all servers in a cluster connected to the iSCSI array via a private network 192.168.1.x

The issue is that one of the hosts can't really see the array (vmhost 2).  I'm guessing it's because of the issue with lefthand2.  I would think that since LH2 is not reachable, it would use the replicated information on LH1 to provide storage resources to the VM array.  However, this does not appear to be the case.

Doing anything in vCenter takes a long time.  I've tried to rescan all storage adapters but it is either locked or taking forever.  I don't know if I should break the 2 node storage management group and see if I can access the datastores on the good array or what.  I've got a a call into HP support and a ticket opened, but have yet to hear anything.  

Anyone seen or heard of something similar before?  Any insight would be greatly appreciated.

Thanks,

Ben
0
Comment
Question by:bwhorton
  • 10
  • 6
  • 4
21 Comments
 
LVL 119
ID: 37833892
what version are you running?

and get it escalted to HP, for the private fixes.
0
 

Author Comment

by:bwhorton
ID: 37833929
The Lefthand is running software version 8.1

Working to get it escalated with HP...
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 37833931
Can you post a screenshot of the HP Console?

When you look at the physical console of both left hand nodes what do you see on the screens?
0
Manage your data center from practically anywhere

The KN8164V features HD resolution of 1920 x 1200, FIPS 140-2 with level 1 security standards and virtual media transmissions at twice the speed. Built for reliability, the KN series provides local console and remote over IP access, ensuring 24/7 availability to all servers.

 
LVL 119
ID: 37833939
8.1 is ancient 9.5 is the latest, and 9.5/9.6 have private fixes from HP, because of issues with VMware vSphere 4.x.
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 37833951
Also how have you got the nodes bonded? And presumably both NIC's are connected to the network?

At you able to ping the bond IP using the network tools in the console from the working node?
0
 

Author Comment

by:bwhorton
ID: 37833966
See attached.  If you need specific information from one or more areas, let me know.
Screen Capture 1
0
 

Author Comment

by:bwhorton
ID: 37833984
Can ping 192.168.1.101 from vcenter which is lefthand1.

Cannot ping 192.168.1.102 from vcenter which is lefthand2


All private iSCSI traffic is connected through a Brocade FES switch.
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 37834005
I will be in front of my left hand console in about half an hour.

What do you see on the physical consoles for the left hand boxes?
0
 

Author Comment

by:bwhorton
ID: 37834021
They are in a different location.  I'm remoted in to the vcenter server where the LH CMC is running.  Do I need to hook up a kvm to the lefthand boxes and let you know what I see or are you referring to the drives/light on the front of the devices?  Sorry for my ignorance.
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 37834099
Well, it's possible if you had 2 drives "fail" that the RAID Controller may be paused waiting for a response.

I had this recently when it though 3 drives had failed.  The drive failing issue is resolved in version 9.x that's not to say yours may not have actually failed but it is a known issue with left hand falsely reporting drive failures.

On 2 of my nodes I had around 12 disks fail in the space of 2 months.  As I replace 1 from each RAID array every 4 months it's unlikely it's down to "batches".  After an upgrade the issue seems to have slowed down.
0
 

Author Comment

by:bwhorton
ID: 37834109
So are you suggesting an upgrade to the software version as a first step to see if it resolves?
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 37834186
Not yet.  Let's get the other node back online first.

You are going to need to get a screen attached to see what's going on.
0
 

Author Comment

by:bwhorton
ID: 37834198
There's a screen shot above, but I'm guessing you need a specific area for me to capture.  Just let me know which and I'll do that.  Thanks
0
 
LVL 74

Expert Comment

by:Glen Knight
ID: 37834216
Sorry, I mean the physical box.  Monitor/keyboard/mouse or KVM depending how you are setup
0
 

Author Comment

by:bwhorton
ID: 37834423
Working with HP engineer on phone now.  First obvious issue is that it appears lh2 ip settings are gone.  Will post more when I learn more. Thanks
0
 

Accepted Solution

by:
bwhorton earned 0 total points
ID: 37835029
I am closing this post.  HP support determined that a drive failure, combined with a software failure (possibly firmware related, but couldn't be sure until logs were reviewed) created system instability in one node.  That node happened to also run the FOM so it couldn't create a quorum as intended.  We removed the FOM from the management group, created a virtual manager, and it created the quorum.  We also had to reseat the controller card in the failed node to get it to re-initialize which removed the software "hang".  

Since HP actually provided the solution, but demazter and hanccocka were quick to respond and offer support, I will award a splitting of the points to both of you.

Thanks!

Ben
0
 
LVL 119
ID: 37835059
no point split here, but dont worry about it!
0
 

Author Comment

by:bwhorton
ID: 37835065
hancocka, i opened a ticket with the mod when I saw that points weren't split.  They should be addressing it shortly.  Thanks!
0
 
LVL 119
ID: 37835074
thats jolly good of you! thanks
0
 

Author Closing Comment

by:bwhorton
ID: 37854909
Self-supported with guidance from HP Support Engineers to diagnose and remedy the problem.
0

Featured Post

Create the perfect environment for any meeting

You might have a modern environment with all sorts of high-tech equipment, but what makes it worthwhile is how you seamlessly bring together the presentation with audio, video and lighting. The ATEN Control System provides integrated control and system automation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If your vDisk VHD file gets deleted from the image store accidentally or on purpose, you won't be able to remove the vDisk from the PVS console. There is a known workaround that is solid.
HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
Teach the user how to convert virtaul disk file formats and how to rename virtual machine files on datastores. Open vSphere Web Client: Review VM disk settings: Migrate VM to new datastore with a thick provisioned (lazy zeroed) disk format: Rename a…
In this video tutorial I show you the main steps to install and configure  a VMware ESXi6.0 server. The video has my comments as text on the screen and you can pause anytime when needed. Hope this will be helpful. Verify that your hardware and BIO…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question