Xen Server Storage Issue

Posted on 2012-12-22
Last Modified: 2013-01-06

I've just upgraded a XenServer host from 5.5.0 to 5.6.0 SP2, however on restart, I get the following message in XenCenter:

Failed to attached storage on server start

This refers to a 2 TB lun on a NetApp filer via Fibre Channel. Other hosts in the pool (not yet upgraded) can see the lun ok, yet it's status is broken as the host reports to be "unplugged" from the storage.

This is 1 of 8 luns accessible - other FC luns are ok.

No idea what to do now, so any help would be greatly appreciated!

Question by:gmbaxter
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 4

Expert Comment

ID: 38716347
I'm sure you tried right clicking the storage and elect "repair". What did that do?
Have you checked the log files on your upgraded host?
You'll find the important ones here:


You can use a tool like WinSCP to connect to the Dom0 and look at the content.
Check for any errors, especially around boot time, and post back.
LVL 11

Author Comment

ID: 38716714
Yes, I tried repair - the error mentioned PBDs

/var/log/messages mentioned:
very big device

xensource.log extract attached, where it tries to attach the SR. PBD is mentioned in here also.

Thanks for the quick reply by the way.

Expert Comment

ID: 38717173
Try to look up the messages in SMLog, right around the time the xensource log reported the error. SMlog is located in the same folder /var/log/
I forgot that storage operations are mostly logged in SMLog.
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

LVL 11

Author Comment

ID: 38717386
Extract below, but the full file is attached:

[14279] 2012-12-22 22:50:51.246524	SUCCESS
[14279] 2012-12-22 22:50:51.246669	Synching Metadata volume
[14279] 2012-12-22 22:50:51.246766	['/usr/sbin/lvchange', '-ay', '/dev/VG_XenStorage-25b74c46-b305-c164-1070-6bf5abb49c5b/MGT']
[14279] 2012-12-22 22:50:51.341808	SUCCESS
[14279] 2012-12-22 22:50:51.342376	lock: released /var/lock/sm/25b74c46-b305-c164-1070-6bf5abb49c5b/sr
[14279] 2012-12-22 22:50:51.351858	***** sr_attach: EXCEPTION exceptions.AssertionError, 
  File "/opt/xensource/sm/", line 94, in run
    return self._run_locked(sr)
  File "/opt/xensource/sm/", line 131, in _run_locked
    return self._run(sr, target)
  File "/opt/xensource/sm/", line 223, in _run
    return sr.attach(self.params['sr_uuid'])
  File "/opt/xensource/sm/LVMoHBASR", line 124, in attach
    LVHDSR.LVHDSR.attach(self, sr_uuid)
  File "/opt/xensource/sm/", line 309, in attach
  File "/opt/xensource/sm/", line 201, in _checkMetadataVolume
  File "/opt/xensource/sm/", line 211, in _synchMetaData
    xml = metadata.retrieveXMLfromFile(self.mdpath)
  File "/opt/xensource/sm/", line 103, in retrieveXMLfromFile
  File "/opt/xensource/sm/", line 41, in _testHdr
    assert(hdr[2] <= MD_MAJOR)

[14279] 2012-12-22 22:50:51.357473	lock: closed /var/lock/sm/25b74c46-b305-c164-1070-6bf5abb49c5b/sr
[14451] 2012-12-22 22:50:51.373612	LVMCache created for VG_XenStorage-76c86946-ce7a-6690-ba60-2692a0b5e303
<14451> 2012-12-22 22:50:51.514441	Host f8b94db1-26eb-4bcb-96ac-293075c52204 not enabled
<14451> 2012-12-22 22:50:51.514677	Pool is not ready, exiting
<14451> 2012-12-22 22:50:51.514750	SR 76c8 ('NetApp 2.5TB 1st floor') (0 VDIs in 0 VHD trees): no changes
[14491] 2012-12-22 22:50:51.699425	LVMCache created for VG_XenStorage-c06d1738-f71a-4b4a-d7df-3685db30e2a8
[14491] 2012-12-22 22:50:51.708770	['/usr/sbin/vgs', 'VG_XenStorage-c06d1738-f71a-4b4a-d7df-3685db30e2a8']
[14491] 2012-12-22 22:50:51.841909	SUCCESS
[14491] 2012-12-22 22:50:51.842105	lock: acquired /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:51.842172	LVMCache: will initialize now
[14491] 2012-12-22 22:50:51.842218	LVMCache: refreshing
[14491] 2012-12-22 22:50:51.842288	['/usr/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-c06d1738-f71a-4b4a-d7df-3685db30e2a8']
[14491] 2012-12-22 22:50:51.921928	SUCCESS
[14491] 2012-12-22 22:50:51.922361	lock: released /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:51.922611	lock: closed /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:51.922684	LVMCache created for VG_XenStorage-c06d1738-f71a-4b4a-d7df-3685db30e2a8
[14491] 2012-12-22 22:50:51.932351	['/usr/sbin/vgs', 'VG_XenStorage-c06d1738-f71a-4b4a-d7df-3685db30e2a8']
[14491] 2012-12-22 22:50:52.008071	SUCCESS
[14491] 2012-12-22 22:50:52.008258	lock: acquired /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:52.008326	LVMCache: will initialize now
[14491] 2012-12-22 22:50:52.008371	LVMCache: refreshing
[14491] 2012-12-22 22:50:52.008442	['/usr/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-c06d1738-f71a-4b4a-d7df-3685db30e2a8']
[14491] 2012-12-22 22:50:52.147745	SUCCESS
[14491] 2012-12-22 22:50:52.148196	lock: released /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:52.148327	lock: acquired /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:52.148461	sr_attach {'sr_uuid': 'c06d1738-f71a-4b4a-d7df-3685db30e2a8', 'subtask_of': 'OpaqueRef:94fadc0d-5104-cea7-8f83-46c3f9ab1b5c', 'args': [], 'host_ref': 'OpaqueRef:a69330e3-757f-8ff5-6099-d397d54caf85', 'session_ref': 'OpaqueRef:e1791520-24a6-f91e-3670-373e4153111f', 'device_config': {'device': '/dev/disk/mpInuse/360a98000572d5038636f58795a6c3753', 'SCSIid': '360a98000572d5038636f58795a6c3753', 'SRmaster': 'true'}, 'command': 'sr_attach', 'sr_ref': 'OpaqueRef:ac0b5d46-1ac7-7051-83c0-d1e720b5779f'}
[14491] 2012-12-22 22:50:52.148766	lock: acquired /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/running
[14491] 2012-12-22 22:50:52.148830	lock: acquired /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:52.158463	RESET for SR c06d1738-f71a-4b4a-d7df-3685db30e2a8 (master: True)
[14491] 2012-12-22 22:50:52.195078	lock: released /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:52.195159	lock: released /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/running
[14491] 2012-12-22 22:50:52.195211	lock: closed /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/sr
[14491] 2012-12-22 22:50:52.195261	lock: closed /var/lock/sm/c06d1738-f71a-4b4a-d7df-3685db30e2a8/running
[14491] 2012-12-22 22:50:52.202113	set_dirty 'OpaqueRef:ac0b5d46-1ac7-7051-83c0-d1e720b5779f' succeeded
[14491] 2012-12-22 22:50:52.202177	MPATH: multipath activate called
[14491] 2012-12-22 22:50:52.202227	['/opt/xensource/sm/mpathHBA', 'enable']
[14491] 2012-12-22 22:50:52.221720	SUCCESS

Open in new window

LVL 11

Author Comment

ID: 38717394
From /var/log/messages

Dec 22 22:50:51 bcxenhostb3 fe: 14283 (/opt/xensource/sm/LVMoHBASR <methodCall><methodName>sr_scan</methodName><para...) exitted with code 0
Dec 22 22:50:51 bcxenhostb3 fe: 14279 (/opt/xensource/sm/LVMoHBASR <methodCall><methodName>sr_attach</methodName><pa...) exitted with code 1
Dec 22 22:50:51 bcxenhostb3 xapi: [error|bcxenhostb3|0 thread_zero|creating storage D:aeefcbfe0dce|xapi] Could not plug in pbd '8f3869e9-38c6-e041-da4e-5a21ae84dc61': Api_errors.Server_error("SR_BACKEND_FAILURE", _)
Dec 22 22:50:52 bcxenhostb3 multipathd: sdam: add path (operator)
Dec 22 22:50:52 bcxenhostb3 multipathd: sdam: spurious uevent, path already in pathvec

Expert Comment

ID: 38717400
Unfortunately, there is not much info in the file that I could help you with. I would highly suggest you open an account with the official citrix forum and post your file outputs there if not already done so.
There are Citrix employees on these forums that might be able to help you further.

Also, I assume you disabled HA on your Server before doing the upgrade? How about Multipathing? If Multipathing is still active on your xenservers, try disabling that on all of them and then try to attach the storage again.
LVL 11

Author Comment

ID: 38718390
Thanks, I have posted all of this information over there to see if anything comes out of it,

HA isn't licensed as it is the free version of xenserver, unsure on multipathing? This only affects the one I upgraded to 5.6 from 5.5, so I still have the option to rollback, and try again I suppose.

Expert Comment

ID: 38718587
Yes, you can start over, especially if you have run a "Xe pool-dump-database" and saved the resulting file somewhere offsite.
Then you can simply reinstall the old version and run the restore of the PoolDB on that host.
LVL 11

Author Comment

ID: 38718594
Do i have to restore the DB? I have reverted the pool master back to 5.5 but still have the SR issue
LVL 19

Expert Comment

ID: 38719732
check if the Ip address of the storage or the account used is changed.
LVL 11

Author Comment

ID: 38719915
Its FC storage
LVL 11

Accepted Solution

gmbaxter earned 0 total points
ID: 38734345
Putting the host into maintenance mode and then exiting maintenance mode sorted the issue. Unable to do this in 5.6 though.
LVL 11

Author Closing Comment

ID: 38748229
Correct steps to resolve the sr not appearing.

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, I will show you HOW TO: Perform a Physical to Virtual (P2V) Conversion the easy way from a computer backup (image).
In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
In this video tutorial I show you the main steps to install and configure  a VMware ESXi6.0 server. The video has my comments as text on the screen and you can pause anytime when needed. Hope this will be helpful. Verify that your hardware and BIO…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

756 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question