Solved

Lost Partition on Citrix XenServer

Posted on 2014-04-28
4
2,520 Views
1 Endorsement
Last Modified: 2014-05-01
We have 2 Servers with Citrix XEN 6.2. Both today suddenly both had troubles.
Both Server are Stand Alone with a Raid1 Disk for the System and a Raid5 Disk for the VM's.

It seems that both machines dropped the connection to the Raid5 Drive.

The Raid5 Disk is unplugged. When wie try pbd-plug [UUID] we get following error

Error code: SR_BACKEND_FAILURE_52
Error parameters: , Logical Volume mount/activate error [opterr=Unable to activa                                                                                                                                                             te LV. Errno is 5]

an fdisk -l tells us that there is no partition on the /dev/sdb device.

[root@hskxen02 ~]# fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 146.2 GB, 146263769088 bytes
256 heads, 63 sectors/track, 17712 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       17713   142835711+  ee  EFI GPT

Disk /dev/sdb: 1316.3 GB, 1316373921792 bytes
255 heads, 63 sectors/track, 160040 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Please Help!!
1
Comment
Question by:WaibelIT
  • 2
  • 2
4 Comments
 
LVL 10

Expert Comment

by:acbxyz
Comment Utility
Did you setup the local storage as ext3 ("optimized for XenDesktop") or lvm?
With lvm there won't be partition entries because xenserver (at least 5.5 did) makes the whole disk a physical volume using lvm2. Some commands that may help (comments after #):
pvscan # rescan all disks for physical volumes, this should show sdb and list a VG_XenStorage-*
lvscan # scan for logical volumes. You can ignore the active/inactive-state. xenserver manages this itself
xe sr-list type=lvm
xe sr-scan uuid=xxxx-xxxx-xxxx-xxxx
xe vdi-list sr-uuid=xxxx-xxxx-xxxx-xxxx
0
 

Author Comment

by:WaibelIT
Comment Utility
Hello acbxyz,

please view ther results:
i tried to destroy and recreate the pbd of the raid5 already. i tried it before on a test server. when i try to attache the disk pbd with pbd-plug i get the error as described prior. no matter if i use /dev/sdb as device or /dev/disk/by-id/scisi..........


xe pbd-list
uuid ( RO)                  : db3590c3-c709-c9d0-ed8b-9d2c0984aeb6
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): 6f900dbc-ff24-f1a3-b6b5-4721171bb868
         device-config (MRO): type: nfs_iso; location: 192.168.50.50:/NFS_ISO
    currently-attached ( RO): true


uuid ( RO)                  : 0eea2a16-554a-6e6f-5f65-f8305a6454d9
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): 283656ca-c589-8036-4d71-618891e22cf0
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): true


uuid ( RO)                  : d9d1faae-cc74-a625-e1e2-0e96097f5c2c
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): 3bb63be3-adb8-fc7b-ae58-da2ef9361d9a
         device-config (MRO): device: /dev/sdb
    currently-attached ( RO): false


uuid ( RO)                  : 95021b09-98be-2d62-e75a-19291dae9de7
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): ab352e51-0239-c682-3b32-d619a18b8d2c
         device-config (MRO): serverpath: /NFS_VHD; server: hskna01.hsk.local; options:
    currently-attached ( RO): true


uuid ( RO)                  : 134d9204-aa65-0280-c10d-f8d81fb3a33f
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): 012c1056-022f-ee72-a90e-017ff94c8fe5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): true


uuid ( RO)                  : e561720c-3f44-6a71-1c75-27f6ada7284e
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): aef18868-9f83-7746-9002-4436921ea563
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): true


uuid ( RO)                  : f631d083-3e2a-a695-dd0e-6b750a66e76b
             host-uuid ( RO): 967eadc3-1cce-43b7-a13c-b32bd9f422ea
               sr-uuid ( RO): f0f4b7c9-58c0-cf53-2270-e5038a6472f6
         device-config (MRO): device: /dev/disk/by-id/scsi-36003005701020f9017b41cbb0e07864f-part3
    currently-attached ( RO): true

xe sr-list
uuid ( RO)                : 3bb63be3-adb8-fc7b-ae58-da2ef9361d9a
          name-label ( RW): RAID5
    name-description ( RW): High Speed - SAS XenServer Store
                host ( RO): hskxen02.hsk.local
                type ( RO): ext
        content-type ( RO): user


uuid ( RO)                : 012c1056-022f-ee72-a90e-017ff94c8fe5
          name-label ( RW): XenServer Tools
    name-description ( RW): XenServer Tools ISOs
                host ( RO): hskxen02.hsk.local
                type ( RO): iso
        content-type ( RO): iso


uuid ( RO)                : f0f4b7c9-58c0-cf53-2270-e5038a6472f6
          name-label ( RW): RAID1
    name-description ( RW):
                host ( RO): hskxen02.hsk.local
                type ( RO): ext
        content-type ( RO): user


pvscan
  PV /dev/sda3   VG XSLocalEXT-f0f4b7c9-58c0-cf53-2270-e5038a6472f6   lvm2 [128.21 GB / 0    free]
  Total: 1 [128.21 GB] / in use: 1 [128.21 GB] / in no VG: 0 [0   ]


lvscan
  ACTIVE            '/dev/XSLocalEXT-f0f4b7c9-58c0-cf53-2270-e5038a6472f6/f0f4b7c9-58c0-cf53-2270-e5038a6472f6' [128.21 GB] inherit
0
 
LVL 10

Accepted Solution

by:
acbxyz earned 500 total points
Comment Utility
If even pvscan doesn't recognize the disk xenserver can't either.
The entries below /dev/disk/by-id are just symbolic links to /dev/sdx so using these always gives the same result.

To check the content of disks and partitions you can use this command "blkid /dev/sd*" (without quotes, of course).
Another way is to boot the machine from a live linux system (cd or usb dongle) and try to repair it from there. I prefer grml nowadays and check.
A live system has more tools and a newer kernel than xenserver itself.
0
 

Author Closing Comment

by:WaibelIT
Comment Utility
The Disk could not be recovered. I had a Citrix engeneer working on both Servers vor about 6 hours. On one of the serves he was able to make the Storage accessible again. But sadly the Disk was  empty. We still can not explain how it was possible that two servers at the same time crushed the filesystem. Looking on the protocols of the Hardware nothing special happened. No power loss and no Raid-Error.

Just if some one runs in a similiar problem i post the summery from the citrix engeneer:

Issue Description: cannot re-attach the  SR
Environmental Details: 6.2  

Troubleshooting Steps followed:
========
¿      The error message  found in the logs  :

EXT3-fs error (device dm-1): ext3_valid_block_bitmap: Invalid block bitmap - block_group = 27, block = 885761
Apr 28 12:00:35 hskxen01 kernel: [139021.831759] journal_bmap: journal block not found at offset 31756 on dm-1
Apr 28 12:00:35 hskxen01 kernel: [139021.831776] Aborting journal on device dm-1.
Apr 28 12:00:35 hskxen01 kernel: [139021.838097] ext3_abort called.
Apr 28 12:00:35 hskxen01 kernel: [139021.838112] EXT3-fs error (device dm-1): ext3_journal_start_sb: Detected aborted journal

¿      The PBD plug failed with the error message” Volume Group  XS-Local-**** is not available” .
¿      Recreated the PV and restored  VG on the top.
¿      The PBD plug failed with the fsck error. Below steps fixed the issue:


dumpe2fs /dev/XSLocal-**** | grep "block size" –I  -¿ note down the block size.

Block size:               4096

mke2fs -S -b 4096 -v /dev/XSLocal--***

e2fsck -y -f -v -C 0 /dev/XSLocal--***

tune2fs –j /dev/XSLOCAL--***  ¿ to rebuild the journal.

Since, we have been able to connect the Storage repository, I would now proceed with the case closure.
However if you any questions, please feel free to let me know.
0

Featured Post

NetScaler Deployment Guides and Resources

Citrix NetScaler is certified to support many of the most commonly deployed enterprise applications. Deployment guides provide in-depth recommendations on configuring NetScaler to meet specific application requirements.

Join & Write a Comment

After several days of searching and hunting for limited documentation, I wanted to share this guide to hopefully save someone the hassle of trying to figure this out on their own. I have tested this on Xendesktop 7.1 and PS 4.5 running simultaneous…
Several part series to implement Internet Explorer 11 Enterprise Mode
How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video. We have explained the difference between…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now