[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Cannot find files on iSCSI target

Posted on 2009-02-18
51
Medium Priority
?
3,402 Views
Last Modified: 2013-11-14
We are using CentOS (5.1) servers and a Dell MD3000i storage unit (iSCSI SAN) to run VMWare virtual machines. Everything was running well until we powered down the devices due to scheduled electrical maintenance. After powering on the devices, they would not boot and could not connect to the iSCSI volumes although the initiators appear to be connecting to the target. Prior to the shutdown  which was done properly  no changes were made to any of the configurations.

At this point, I have the CentOS servers running and the iSCSI SAN running. The initiators appear to be connecting to the target. However, I cannot figure out how to access any of the virtual machines on the target.

Boot Up Problem:

The following error was generated at boot-up:

*******************************************
Checking filesystems
/dev/VolGroup00/LogVol00: clean, 113206/8355840 files, 2005677/8355840 blocks
/boot: clean, 35/25584 files, 18099/101984 blocks
fsck.ext3: No such file or directory while trying to open /dev/VolGroup01/LogVol00
/dev/VolGroup01/LogVol00:
The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck b 8193 <device>

*******************************************

I made the following modification to the fstab file in order to get the server to boot. Please note that no lines were deleted from the fstab file:

*******************************************
[root@POCVMHOST11 ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
#/dev/VolGroup01/LogVol00               /VMWare         ext3    defaults       1 2
*******************************************

So at this point, the server boots but I cannot access any of the volumes on the iSCSI target. It appears that the mount point for the target is /VMWare on /dev/ VolGroup01/LogVol00 and I can mount it however, I do not see any data:

*******************************************
[root@POCVMHOST11 ~]# mount t ext3 /dev/VolGroup01/LogVol00 /VMWare
[root@POCVMHOST11 ~]# fdisk l
Disk /dev/cciss/c0d0: 36.4 GB, 36414750720 bytes
255 heads, 32 sectors/track, 8716 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1   *           1          25      101984   83  Linux
/dev/cciss/c0d0p2              26        8716    35459280   8e  Linux LVM

Disk /dev/sda: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       52217   419430399+  8e  Linux LVM
*******************************************

The volume appears to be mounted but I cannot find any data. When I try to unmount, I get strange results. Below is a sequence of commands that I tried:

*******************************************
[root@POCVMHOST11 ~]# umount /dev/VolGroup01/LogVol00 /Vmware
umount: /dev/VolGroup01/LogVol00: not found
umount: /Vmware: not found

[root@POCVMHOST11 ~]# fdisk -l

Disk /dev/cciss/c0d0: 36.4 GB, 36414750720 bytes
255 heads, 32 sectors/track, 8716 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1   *           1          25      101984   83  Linux
/dev/cciss/c0d0p2              26        8716    35459280   8e  Linux LVM

Disk /dev/sda: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       52217   419430399+  8e  Linux LVM

[root@POCVMHOST11 ~]# umount /dev/sda1
umount: /dev/sda1: not mounted

[root@POCVMHOST11 ~]# mount /dev/sda1
mount: can't find /dev/sda1 in /etc/fstab or /etc/mtab

*******************************************
It appears that I need to add /dev/sda1 into the fstab file but it wasnt there before so I shouldnt have to add it.

I am not sure where to go from here. I am pretty sure that the initiators on the CentOS machines are accessing the target without issue. When I use the iscsiadm discovery command, I can find the points:

*******************************************
[root@POCVMHOST11 ~]#  iscsiadm -m discovery -t sendtargets -p x.x.x.11

x.x.x.11:3260,1 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.x.12:3260,1 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.x.13:3260,2 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.x.14:3260,2 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
*******************************************



0
Comment
Question by:Jeremy_in_Japan
  • 21
  • 20
  • 9
  • +1
51 Comments
 
LVL 16

Expert Comment

by:ai_ja_nai
ID: 23668542
Have you started lvm? if you don't, the volums it manages won't be brought up
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23671078
HI

can i see your lvs and vgs out put please
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23677704
Thank you for your comments.

ai_ja_nai:

LVM is a service or daemon? How do you "start" it?

fosiul01:

Here is the output plus some more from pvdisplay and vgdisplay:

[root@POCVMHOST11 ~]# lvs -o +devices
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Devices      
  LogVol00 VolGroup00 -wi-ao  31.88G                               /dev/cciss/c0d0p2(0)
  LogVol01 VolGroup00 -wi-ao   1.94G                               /dev/cciss/c0d0p2(1020)
  LogVol00 VolGroup01 -wi--- 400.00G                               /dev/sda1(0)

[root@POCVMHOST11 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2
  Found volume group "VolGroup01" using metadata type lvm2

[root@POCVMHOST11 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/cciss/c0d0p2
  VG Name               VolGroup00
  PV Size               33.82 GB / not usable 4.20 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              1082
  Free PE               0
  Allocated PE          1082
  PV UUID               X80Wp4-G0QN-C4dk-W5LJ-f8IE-VMN7-hvxPNe

  --- Physical volume ---
  PV Name               /dev/sda1
  VG Name               VolGroup01
  PV Size               400.00 GB / not usable 4.00 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              102399
  Free PE               0
  Allocated PE          102399
  PV UUID               Dhk8Tb-82pF-25Zj-Hzon-36V8-4ugH-3St8KY

[root@POCVMHOST11 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               33.81 GB
  PE Size               32.00 MB
  Total PE              1082
  Alloc PE / Size       1082 / 33.81 GB
  Free  PE / Size       0 / 0
  VG UUID               65R6yd-lIYh-u443-kAJH-2ZDK-vDsN-KAKZf3

  --- Volume group ---
  VG Name               VolGroup01
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                256
  Cur LV                1
  Open LV               0
  Max PV                256
  Cur PV                1
  Act PV                1
  VG Size               400.00 GB
  PE Size               4.00 MB
  Total PE              102399
  Alloc PE / Size       102399 / 400.00 GB
  Free  PE / Size       0 / 0
  VG UUID               WI6Saq-2Oht-ewzr-eIeM-6tyq-SIyP-E6EBMl


0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 29

Expert Comment

by:fosiul01
ID: 23679259
lvm is running fine in your mechine.

so you saying, if you enabled this line #/dev/VolGroup01/LogVol00               /VMWare         ext3    defaults       1 2

server would not boot ??

whats the output of df -h

does it show the VolGroup01/Logvol00 in their ??


you might have to run e2fsck on that disk, but let me have a look something on my pc
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23679329
Thank you for your reply.

Yes, that is correct - if I enable that line, the server will not boot. Here is the output - it does not show the volume. I tried running the e2fsck command as written in the error message at boot (seem boot issue at the top of my thread) but it did not work using the superblock 8193.

[root@POCVMHOST11 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       31G  6.7G   23G  23% /
/dev/cciss/c0d0p1      97M   15M   77M  16% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm

0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23679365
your volgroup and logical vol all there, but its unable to read, give 10 minutes, i am trying to create samething on my pc to see what happended
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23679413
Thank you again. I also tried running the following commands - I am not sure if it helps or not though since it is an iSCSI volume and not physical SCSI volume:

[root@POCVMHOST11 ~]# SMdevices
PowerVault Modular Disk Storage Manager Devices, Version 09.17.A6.01
Built Tue Mar 20 15:31:11 CST 2007
Copyright 2005-2006 Dell Inc. All rights reserved.  Use is subject to license terms

  /dev/sda (/dev/sg0) [Storage Array WEST-MD3000i-01, Virtual Disk DG01-RAID10-LUN00, LUN 0, Virtual Disk ID <6001ec9000def63a000007ec4887594c>, Preferred Path (Controller-0): In Use]

[root@POCVMHOST11 ~]# SMagent start
SMagent started.
[root@POCVMHOST11 ~]# PowerVault Modular Disk Storage Manager Agent, Version 09.17.A6.05
Built Tue Oct 09 11:43:08 CDT 2007
Copyright 2005-2006 Dell Inc. All rights reserved.  Use is subject to license terms
Checking device /dev/sda (/dev/sg0) : Skipping

The SMagent service attempted to start but terminated. Possible causes include:  [1] The access virtual disk is disabled in NVSRAM. [2] No RAID controller modules with compatible firmware were found. [3] The default logical unit number (LUN) assigned to the access virtual disk (typically 7 or 31) is not in the range supported by the host or is already in use by the host for another device. See your storage management documentation for more details.

The Host Context Agent did NOT start because 0 RAID controller modules were found.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23679681
i am not that good in raid and iscsi ...
but one think, your server is recognized the harddrive and showing the size every thing, but its unable to read ...

look at this line

  LogVol00 VolGroup01 -wi--- 400.00G                               /dev/sda1(0)

specially -wi---  

myone  logVol01  VolGroup01 -wi-ao  80.00M
but your one is missing  that...

what the output of pvscan   ??

also : if you execute tthis command

mount /dev/VolGroup01/LogVol00       /VMWare

what it does ??
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23679716
Here is the output:

[root@POCVMHOST11 ~]# pvscan
  PV /dev/cciss/c0d0p2   VG VolGroup00   lvm2 [33.81 GB / 0    free]
  PV /dev/sda1           VG VolGroup01   lvm2 [400.00 GB / 0    free]
  Total: 2 [433.81 GB] / in use: 2 [433.81 GB] / in no VG: 0 [0   ]

[root@POCVMHOST11 ~]# mount /dev/VolGroup01/LogVol00 /VMWare
mount: special device /dev/VolGroup01/LogVol00 does not exist
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23679774
omm type

updatedb
then

locate LogVol00


and go to /dev/VolGroup01/

ls -al

check what output u see.
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23679812
Here is the output:

[root@POCVMHOST11 ~]# updatedb
[root@POCVMHOST11 ~]# locate LogVol00
/dev/VolGroup00/LogVol00
/dev/mapper/VolGroup00-LogVol00

[root@POCVMHOST11 ~]# cd /dev/VolGroup01
-bash: cd: /dev/VolGroup01: No such file or directory
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23679819
ok then do this

mount /dev/mapper/VolGroup00-LogVol00  /VMWare

see if this works....
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23679856
Okay that seemed to have mounted - but it looks like there is a folder with the same name on the server partition:

[root@POCVMHOST11 ~]# mount /dev/mapper/VolGroup00-LogVol00 /VMWare
[root@POCVMHOST11 ~]# find / -name *.vmx
find: Filesystem loop detected; `/VMWare' has the same device number and inode as a directory which is 1 level higher in the filesystem hierarchy.
/var/lib/vmware/Virtual Machines/POCDEV01V/Windows Server 2003 Standard Edition.vmx

Perhaps I need to rename that folder on the server partition?
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23679894
so you can see all your file in /vmware mounted directory ?? are you expecting those files in /Vmware ??
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23680245
Actually, I am not sure where they are - I have obviously assumed wrongly that the mount point is /VMWare. It looks like all of the CentOS servers have a /VMWare folder on the server partition.

Can I umount and then mount on a different point or do I have to know what the mount point is?
0
 
LVL 29

Assisted Solution

by:fosiul01
fosiul01 earned 800 total points
ID: 23680302
haah LOl!! i am dead then

"It looks like all of the CentOS servers have a /VMWare folder on the server partition. " No , its not, you must of created VMware directory in one time in your server

let me try to explain you.

 /dev/VolGroup01/LogVol00               /VMWare         ext3    defaults       1 2

You have created a directory /Vmware, and then you add added a Big hardrive and you have mounted the whole hardrive in that /VMware directory.

now if you unmount /VMware directory, you will see still that directory their but there is no files, because you just unmounted.
now if you create another directory suppose /testvm

now if you mount
mount /dev/mapper/VolGroup00-LogVol00 /testvm

you will see same file in testvm because you just monted the same data at testvm




0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23680583
Okay - the /VMWare folder was probably created with the mkdir command when the server was built. (I didn't make this server - that is why I have no clue where anything is) So, that is why there is a conflict when we mount /VMWare with the iSCSI volume, right?

However, I just tried to umount and got the following error:

[root@POCVMHOST11 ~]# umount /dev/mapper/VolGroup00-LogVol00 /VMWare
umount: /dev/mapper/VolGroup00-LogVol00: not mounted
umount: /dev/mapper/VolGroup00-LogVol00: not mounted

And when I try mounting testvm, I get this:

[root@POCVMHOST11 ~]# mount /dev/mapper/VolGroup00-LogVol00 /testvm
mount: mount point /testvm does not exist

So without knowing the mount point, there is no way to access the data on the iSCSI volume?
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23680642
" without knowing the mount point, there is no way to access the data on the iSCSI volume?" : No , you can create any mount point as you wish. as long as your harddrive has data it will mount to any mount point, also you can mount same hardrive to 2 different mount point, it will show you same data

about this error : mount: mount point /testvm does not exist

its because you didnot create any directory call /testvm

you need to create the directory, and have to define the path in here
i am assuing you have created /testvm in root directory[ first level directory] [ not under /root]
mount /dev/mapper/VolGroup00-LogVol00 /testvm

about this error :
umount: /dev/mapper/VolGroup00-LogVol00: not mounted
umount: /dev/mapper/VolGroup00-LogVol00: not mounted

some times it give this error, but it does unmount the directory [ you can check by going cd /Vmware , you will see no data in their, that mean it did umount ]

0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23680836
Okay - I am spinning in circles because I don't know what I am doing. I tried creating the testvm directory and mounting again but basically got the same result:

[root@POCVMHOST11 Virtual Machines]# find / -name *.vmx
find: Filesystem loop detected; `/testvm' has the same device number and inode as a directory which is 1 level higher in the filesystem hierarchy.
/var/lib/vmware/Virtual Machines/POCDEV01V/Windows Server 2003 Standard Edition.vmx

[root@POCVMHOST11 Virtual Machines]# umount /dev/mapper/VolGroup00-LogVol00 /testvm
umount: /testvm: device is busy
umount: /testvm: device is busy
umount: /: device is busy
umount: /testvm: device is busy
umount: /testvm: device is busy

This is strange - it is like the testvm folder is mounting itself:

[root@POCVMHOST11 testvm]# cd ..
[root@POCVMHOST11 /]# pwd
/
[root@POCVMHOST11 /]# ls -la
total 182
drwxr-xr-x  26 root root  4096 Feb 19 22:00 .
drwxr-xr-x  26 root root  4096 Feb 19 22:00 ..
-rw-r--r--   1 root root     0 Feb 18 16:51 .autofsck
-rw-r--r--   1 root root     0 Jun  8  2008 .autorelabel
drwxr-xr-x   2 root root  4096 Jun  7  2008 bin
drwxr-xr-x   4 root root  1024 Jul 23  2008 boot
-rw-rw-r--   1 root root  4055 Jul 23  2008 console.txt
drwxr-xr-x  12 root root  3780 Feb 18 16:51 dev
drwxr-xr-x  97 root root 12288 Feb 19 22:00 etc
drwxr-xr-x   4 root root  4096 Jul 23  2008 home
drwxr-xr-x  14 root root  4096 Jun  7  2008 lib
drwx------   2 root root 16384 Jun  8  2008 lost+found
drwxr-xr-x   3 root root  4096 Jul 22  2008 media
drwxr-xr-x   2 root root     0 Feb 18 16:51 misc
drwxr-xr-x   2 root root  4096 Mar 30  2007 mnt
drwxr-xr-x   2 root root     0 Feb 18 16:51 net
drwxr-xr-x   4 root root  4096 Jul 23  2008 opt
dr-xr-xr-x 139 root root     0 Feb 18 16:51 proc
drwxr-x---  20 root root  4096 Feb 18 12:56 root
drwxr-xr-x   2 root root 12288 Jul 24  2008 sbin
drwxr-xr-x   2 root root  4096 Jun  8  2008 selinux
drwxr-xr-x   2 root root  4096 Mar 30  2007 srv
drwxr-xr-x  11 root root     0 Feb 18 16:51 sys
drwxr-xr-x  26 root root  4096 Feb 19 22:00 testvm
drwxrwxrwt   7 root root  4096 Feb 19 04:02 tmp
drwxr-xr-x  15 root root  4096 Jul 23  2008 usr
drwxr-xr-x  27 root root  4096 Feb 18 17:00 var
drwxr-xr-x   2 root root  4096 Feb 19 18:08 .vmware
drwxrwxr-x   3 root root  4096 Feb 19 22:03 VMWare

[root@POCVMHOST11 /]# cd testvm/

[root@POCVMHOST11 testvm]# ls -la
total 220
drwxr-xr-x 26 root root  4096 Feb 19 22:00 .
drwxr-xr-x 26 root root  4096 Feb 19 22:00 ..
-rw-r--r--  1 root root     0 Feb 18 16:51 .autofsck
-rw-r--r--  1 root root     0 Jun  8  2008 .autorelabel
drwxr-xr-x  2 root root  4096 Jun  7  2008 bin
drwxr-xr-x  2 root root  4096 Jun  8  2008 boot
-rw-rw-r--  1 root root  4055 Jul 23  2008 console.txt
drwxr-xr-x  4 root root  4096 Jun  8  2008 dev
drwxr-xr-x 97 root root 12288 Feb 19 22:00 etc
drwxr-xr-x  4 root root  4096 Jul 23  2008 home
drwxr-xr-x 14 root root  4096 Jun  7  2008 lib
drwx------  2 root root 16384 Jun  8  2008 lost+found
drwxr-xr-x  3 root root  4096 Jul 22  2008 media
drwxr-xr-x  2 root root  4096 Nov 12  2007 misc
drwxr-xr-x  2 root root  4096 Mar 30  2007 mnt
dr-xr-xr-x  2 root root  4096 Jun  8  2008 net
drwxr-xr-x  4 root root  4096 Jul 23  2008 opt
drwxr-xr-x  2 root root  4096 Jun  8  2008 proc
drwxr-x--- 20 root root  4096 Feb 18 12:56 root
drwxr-xr-x  2 root root 12288 Jul 24  2008 sbin
drwxr-xr-x  2 root root  4096 Jun  8  2008 selinux
drwxr-xr-x  2 root root  4096 Mar 30  2007 srv
drwxr-xr-x  2 root root  4096 Jun  8  2008 sys
drwxr-xr-x  2 root root  4096 Feb 19 22:00 testvm
drwxrwxrwt  7 root root  4096 Feb 19 04:02 tmp
drwxr-xr-x 15 root root  4096 Jul 23  2008 usr
drwxr-xr-x 27 root root  4096 Feb 18 17:00 var
drwxr-xr-x  2 root root  4096 Feb 19 18:08 .vmware
drwxrwxr-x  3 root root  4096 Feb 19 22:03 VMWare

0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23680939
taht because we are mounting the wrong one!!!!
its not this
mount /dev/mapper/VolGroup00-LogVol00  /VMWare

it should be

mount /dev/mapper/VolGroup01-LogVol00  /VMWare


because your having problem with VolGroup01-LogVol00   not VolGroup00-LogVol00

can past the out put of

cd /dev/mapper
then
ls
copy and past the out put here ..full output
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23681000
Sorry - I have been working all day and my mind is just scattered...

[root@POCVMHOST11 testvm]# umount /dev/mapper/VolGroup00-LogVol00 /testvm
umount: /testvm: device is busy
umount: /testvm: device is busy
umount: /: device is busy
umount: /testvm: device is busy
umount: /testvm: device is busy

[root@POCVMHOST11 testvm]# cd /dev/mapper/

[root@POCVMHOST11 mapper]# ls
control  VolGroup00-LogVol00  VolGroup00-LogVol01
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23681083
ommmm i am worried now, you should of see some file call

 VolGroup01-LogVol00


how those files has been deleted ??


do this :
locate VolGroup01

show me the out put here ..
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23681113
Here you go:

[root@POCVMHOST11 mapper]# locate VolGroup01
/etc/lvm/archive/VolGroup01_00000.vg
/etc/lvm/archive/VolGroup01_00001.vg
/etc/lvm/backup/VolGroup01
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23681140
wait let me see if i can delete my VolGroup01 and can reput in from backup

let see if it works
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23682316
sorry i broke my system !!
let me fix my one then i will look at your one , 30 min please
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23683576
HI sorry for late, my Vm pc is broke down it would reboot because of i deleted some VolGroup to test, for some reason it not recognized my cd room so i cant edit any file and my office hour is finished

I understand your problem is urgent, then you can open another queston ,

i will be online after couple hours from home, then i can try again,

but as i said,  your Volgroup01 is missing, so need find a way to place it from backup , which i never see before so without trying in my pc i cant give you a solution.

so you can open another question or i will look it up again after going home,

sorry for this problem
 
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23690145
Ok good morning

have you given up with this problem ??

my pc is working now!!

anyway

can you try with this command : but make sure you dont have any VolGroup01 file in /dev/mapper

vgcfgrestore VolGroup01

it should restore VolGroup01 and its logical volume in /dev/mapper files ... Finger Cross

[ how importnat those data on that harddrive ??]


0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23690234
Thanks for your reply - actually, I posted a question on Dell's Enterprise Technology site for specific help on the Dell MD3000i storage.

As for the data, it is a "test" environment for development however, I really don't want to zap any of the data because there is an Exchange VM on it and I really do not want to have to configure that from scratch.

If this is not a sure thing, it is probably best that I wait. Besides, there is VolGroup01 in /dev/mapper anyway:

[root@POCVMHOST11 dev]# ls /dev/mapper/
control  VolGroup00-LogVol00  VolGroup00-LogVol01
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23690280
yes, try to speak with them first see if this is any hardware related problem

but from my point of view you dont have   VolGroup01 in /dev/mapper  [ which is the problem so may be at last you will have to rebackup from the command i said]


your comments : Besides, there is VolGroup01 in /dev/mapper anyway: . where are you sing VolGroup01 in Dev/mapper file ??

if you would of seen then your problem is solved!! but you said you dont have...


from this output

[root@POCVMHOST11 dev]# ls /dev/mapper/
control  VolGroup00-LogVol00  VolGroup00-LogVol01

where is your VolGroup01  ??  
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23690326
Crap - sorry, I have been looking at VolGroup and LogVol so many times from the past few days and I mixed up the letters and numbers. Sorry!

Okay - I haven't received any replies from anyone on Dell's site yet - if I don't hear anything, I will give it a try.

I think that the SMagent is supposed start (according to Dell's documentation anyway) - the servers are logging into the target just fine. I doubt that it is a hardware or compatibility issue - it is mostly me, not understanding how and where the volumes are supposed to be mapped.

Anyway, I will post again as soon as find out something from Dell!
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23690334
yes, i dont belived as well its not hardware because fdisk -l can see your hardrive

its because some how your VolGroup01 has been deleted,

let me know what happeded, then we will try again .
0
 
LVL 19

Expert Comment

by:jools
ID: 23696360
This looks like and intresting thread...

I think it's time for a recap, this is my understanding;

Your server sees the iSCSI drive on /dev/sda, no configuration has changed and this is still the same disk.

The system boots up with the LVM entry for this mount point commented out but when it is enabled you get errors because the system cannot find the file systems. LVM for rootvg is OK.

The system can see the disk /dev/sda but cannot see anything on it, there is a partition table on the disk with the type 8e and it all looks fine and dandy.

Now it seems to have got worse.

Can you backup the backup files /etc/lvm/backup.

...Check you're happy running the following commands before you run them...
What happens if you run vgscan? This should try to import the vg information.
Can you repost the pvs, vgs and lvs output as well as fdisk -l.


0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23698705
Thanks Jools.

Yeah, I will have a story written here by the time we are done! :)

I made a backup of /etc/lvm/backup. Here is the output:

[root@POCVMHOST11 /]# ls ExpertsHelp/
RAID10  VolGroup00  VolGroup01

[root@POCVMHOST11 /]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2
  Found volume group "VolGroup01" using metadata type lvm2

[root@POCVMHOST11 /]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/cciss/c0d0p2
  VG Name               VolGroup00
  PV Size               33.82 GB / not usable 4.20 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              1082
  Free PE               0
  Allocated PE          1082
  PV UUID               X80Wp4-G0QN-C4dk-W5LJ-f8IE-VMN7-hvxPNe

  --- Physical volume ---
  PV Name               /dev/sda1
  VG Name               VolGroup01
  PV Size               400.00 GB / not usable 4.00 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              102399
  Free PE               0
  Allocated PE          102399
  PV UUID               Dhk8Tb-82pF-25Zj-Hzon-36V8-4ugH-3St8KY

[root@POCVMHOST11 /]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               33.81 GB
  PE Size               32.00 MB
  Total PE              1082
  Alloc PE / Size       1082 / 33.81 GB
  Free  PE / Size       0 / 0
  VG UUID               65R6yd-lIYh-u443-kAJH-2ZDK-vDsN-KAKZf3

  --- Volume group ---
  VG Name               VolGroup01
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                256
  Cur LV                1
  Open LV               0
  Max PV                256
  Cur PV                1
  Act PV                1
  VG Size               400.00 GB
  PE Size               4.00 MB
  Total PE              102399
  Alloc PE / Size       102399 / 400.00 GB
  Free  PE / Size       0 / 0
  VG UUID               WI6Saq-2Oht-ewzr-eIeM-6tyq-SIyP-E6EBMl

[root@POCVMHOST11 /]# lvs -o +devices

  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Devices                                                    
  LogVol00 VolGroup00 -wi-ao  31.88G                               /dev/cciss/c0d0p                                           2(0)
  LogVol01 VolGroup00 -wi-ao   1.94G                               /dev/cciss/c0d0p                                           2(1020)
  LogVol00 VolGroup01 -wi--- 400.00G                               /dev/sda1(0)                                              

[root@POCVMHOST11 /]# iscsiadm -m node

x.x.213.13:3260,2 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.213.11:3260,1 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.213.12:3260,1 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.213.14:3260,2 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223

[root@POCVMHOST11 /]# SMagent start
SMagent started.
[root@POCVMHOST11 /]# PowerVault Modular Disk Storage Manager Agent, Version 09.17.A6.05
Built Tue Oct 09 11:43:08 CDT 2007
Copyright 2005-2006 Dell Inc. All rights reserved.  Use is subject to license terms
Checking device /dev/sda (/dev/sg0) : Skipping

The SMagent service attempted to start but terminated. Possible causes include:  [1] The access virtual disk is disabled in NVSRAM. [2] No RAID controller modules with compatible firmware were found. [3] The default logical unit number (LUN) assigned to the access virtual disk (typically 7 or 31) is not in the range supported by the host or is already in use by the host for another device. See your storage management documentation for more details.

The Host Context Agent did NOT start because 0 RAID controller modules were found.

0
 
LVL 19

Accepted Solution

by:
jools earned 1200 total points
ID: 23698807
It seems that your LV is not active. Try running `vgchange -a y` then try and mount the LV.

I am a little concerned about the SMagent errors but as I do not have a system like yours available to me to test I cant verify if this is a problem or not, it could be that the agent is already started. Can you `restart` to see if it comes up clean.
0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23698837
Should I mount using this command?

mount /dev/sda


0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23699139
Thanks Jools to keep my word and attending this question.[ i asked another guy but he didnot] anyway thanks, this guy needs some experienced help which i dont have..

but most worring thing is, he has this

LogVol00 VolGroup01 -wi--- 400.00G            dev/sda1(0)              

but he does not have any VolGroup01 in /dev/mapper file ....

                               


0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23699196
I can give you the output of the other servers too if you think it will be helpful. I have done any of the commands above on those servers.

Thanks again for your help and time - I really appreciate it.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23699222
HI please wait for jools see what he suggest then will take from their..
0
 
LVL 19

Expert Comment

by:jools
ID: 23699451
No,

mount  /dev/VolGroup01/LogVol00 <mountpoint>
...or you can just uncomment the line in your fstab file and run `mount -a`
0
 
LVL 19

Expert Comment

by:jools
ID: 23699457
sorry to keep reposting.... did the vgchange work?

make sure you run it and rerun the lvs command you did before,

0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23699458
/dev/VolGroup01/LogVol00 that the problem ...

his VolGroup01 is missing totaly

only VolGroup01 he has is only /etc/lvm/backup directory
0
 
LVL 19

Expert Comment

by:jools
ID: 23699463
Hi fosiul, running the vgchange should bring all that back in... one step at a time and all that. be patient :-)

0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23699754
Thanks again for your help. Here is the result and there is great news...

[root@POCVMHOST11 ~]# vgchange -a y
  2 logical volume(s) in volume group "VolGroup00" now active
  1 logical volume(s) in volume group "VolGroup01" now active

[root@POCVMHOST11 ~]# vi /etc/fstab

[root@POCVMHOST11 ~]# cat /etc/fstab

/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
/dev/VolGroup01/LogVol00                /VMWare         ext3    defaults        1 2

[root@POCVMHOST11 ~]# mount -a

[root@POCVMHOST11 ~]# lvs -o +devices

  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Devices
  LogVol00 VolGroup00 -wi-ao  31.88G                               /dev/cciss/c0d0p2(0)
  LogVol01 VolGroup00 -wi-ao   1.94G                               /dev/cciss/c0d0p2(1020)
  LogVol00 VolGroup01 -wi-ao 400.00G                               /dev/sda1(0)

[root@POCVMHOST11 ~]# SMagent start

SMagent started.
[root@POCVMHOST11 ~]# PowerVault Modular Disk Storage Manager Agent, Version 09.17.A6.05
Built Tue Oct 09 11:43:08 CDT 2007
Copyright 2005-2006 Dell Inc. All rights reserved.  Use is subject to license terms
Checking device /dev/sda (/dev/sg0) : Skipping

The SMagent service attempted to start but terminated.....

I can now see the VM on the iSCSI storage! Yeah!

So, I should do the same for the other servers as well, right?

1. vgchange -a y

2. <uncomment /etc/fstab>  mount -a

3. lvs -o +devices


0
 
LVL 19

Expert Comment

by:jools
ID: 23699782
Looks that way, something deactivated the vg, I've not used iscsi on Linux as yet but you should try rebooting on this system first.

using vgchange just allowed the kernel to see the LVM which then created the device files.

The SMagent could be a red herring here so just ignore it for now.

0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23699833
Okay - the server will not reboot now like before.
0
 
LVL 19

Expert Comment

by:jools
ID: 23699854
Not out of the woods quite yet then.

If you can boot into single user mode, type `vgchange -a y` then type `init 3`.
If not then boot into emergency mode.

You will need to add a `vgchange -a y` into the start scripts before it runs mount but after LVM is initialised. Have a look in /etc/rc.d/rc.sysinit to see where these are initialised. I don't have access to a system to help today.
0
 
LVL 19

Expert Comment

by:jools
ID: 23699880
actually, I just logged into a work system and found this line in rc.sysinit

> [root@dba rc.d]# grep lvm *
> rc.sysinit:     if [ -x /sbin/lvm.static ]; then
> rc.sysinit:             action $"Setting up Logical Volume Management:" /sbin/lvm.static vgchange -a y --ignorelockingfailure

The file /sbin/lvm.static exists on my system, I guess it will on yours too. Check the init scripts for when the iSCSI stuff is loaded, you may have to tweak your system so that when the vgchange runs it can see the disks on the iSCSI. I guess at the moment, when it runs it does not know about it.

This doc; http://www.cyberciti.biz/tips/rhel-centos-fedora-linux-iscsi-howto.html suggests that there is an iscsi start script, can you guess where I'm going with this... :-)

0
 
LVL 3

Author Comment

by:Jeremy_in_Japan
ID: 23708333
Thanks again Jools.

Yes, I have the lvm.static file - I will have to experiment with the login scripts.  For now, I can access and start/stop the VMs so I should be okay. I was able to do the same process for all of the other servers except for one.

There are five CentOS servers that connect to the iSCSI target:

Server11
Server12
Server13
Server14
Server15

There are five host-to-virtual desk mappings:

DG01-RAID10-LUN00 accessbile by Server11
DG01-RAID10-LUN01 accessbile by Server12
DG01-RAID10-LUN02 accessbile by Server13
DG02-RAID05-LUN00 accessbile by Server14
DG02-RAID05-LUN01 accessbile by Server15

However, there only appears to be four ports configured - perhaps this doesn't matter?

Login to [iface: default, target: iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223, portal: x.x.x.13,3260]: successful
Login to [iface: default, target: iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223, portal: x.x.x.11,3260]: successful
Login to [iface: default, target: iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223, portal: x.x.x.12,3260]: successful
Login to [iface: default, target: iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223, portal: x.x.x.14,3260]: successful

Server12 is the one that is having a problem - according to a list that I have, there are no VMs listed in its storage. It appears that the disk has not been configured:

***************************************
[root@POCVMHOST12 /]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2

[root@POCVMHOST12 /]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/cciss/c0d0p2
  VG Name               VolGroup00
  PV Size               33.82 GB / not usable 4.20 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              1082
  Free PE               0
  Allocated PE          1082
  PV UUID               X80Wp4-G0QN-C4dk-W5LJ-f8IE-VMN7-hvxPNe

[root@POCVMHOST12 /]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               33.81 GB
  PE Size               32.00 MB
  Total PE              1082
  Alloc PE / Size       1082 / 33.81 GB
  Free  PE / Size       0 / 0
  VG UUID               65R6yd-lIYh-u443-kAJH-2ZDK-vDsN-KAKZf3

[root@POCVMHOST12 /]# lvs -o +devices
  LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Devices
  LogVol00 VolGroup00 -wi-ao 31.88G                               /dev/cciss/c0d0p2(0)
  LogVol01 VolGroup00 -wi-ao  1.94G                               /dev/cciss/c0d0p2(1020)
[root@POCVMHOST12 /]# fdisk -l

Disk /dev/cciss/c0d0: 36.4 GB, 36414750720 bytes
255 heads, 32 sectors/track, 8716 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1   *           1          25      101984   83  Linux
/dev/cciss/c0d0p2              26        8716    35459280   8e  Linux LVM

Disk /dev/sda: 429.4 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sda doesn't contain a valid partition table

[root@POCVMHOST12 /]# iscsiadm -m node
x.x.x.13:3260,2 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.x.11:3260,1 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.x.12:3260,1 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223
x.x.x.14:3260,2 iqn.1984-05.com.dell:powervault.6001ec9000def63a0000000048746223

[root@POCVMHOST12 /]# ls /dev/mapper
control  VolGroup00-LogVol00  VolGroup00-LogVol01

[root@POCVMHOST12 /]# locate VolGroup01
[root@POCVMHOST12 /]#

***************************************

In order to keep the same configuration as the other servers, how can I create the VolGroup01? Should I open this as a new question? I would like to give you and fosiul01 500 points for all of your help.
0
 
LVL 19

Expert Comment

by:jools
ID: 23708815
To be fair I think you're looking at another Q for that I'm afraid. I will say you need to be 100% sure about the configuration, it'll be very difficult diagnosing a LVM disk problem if the disk doesnt exist on the server.... vgscan would usually pick this up!

Did you manage to sort out the configuration when the systems reboot?

You only have 500 points that can be assigned per question so you'll either have to divide it up as you see fit or assign all the points to one person and then open a separate question that the points will be for the other person, however, they are your points so it's up to you.
0
 
LVL 3

Author Closing Comment

by:Jeremy_in_Japan
ID: 31548169
Thanks again for all of your time - I was able to find the VMs and get what I needed done so thank you. I have some test servers so I will definitely start educating myself on iSCSI. I wish that I could award more points - thank you again.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 23709136
HI good morning
sorry due to weakend i didnot able to see any thing here

let me read the solution.

0

Featured Post

Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many businesses neglect disaster recovery and treat it as an after-thought. I can tell you first hand that data will be lost, hard drives die, servers will be hacked, and careless (or malicious) employees can ruin your data.
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses
Course of the Month17 days, 16 hours left to enroll

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question