Can't unmirror hdisk3 from vg.

dfke
dfke used Ask the Experts™
on
On one system I have a defective hdisk3.

When issuing a unmirror secondvg hdisk3 I get:

0516-076 lreducelv: Cannot remove last good copy of stale partition.
        Resynchronize the partitions with syncvg and try again.
0516-922 rmlvcopy: Unable to remove logical partition copies from
        logical volume loglv00.
0516-1135 unmirrorvg: The unmirror of the volume group failed.
        The volume group is still partially or fully mirrored.

I never came across this message before.

hdisk2 and hdisk3 are in a software mirror. And from the looks of it it's still mirrored:

:/# lspv -l hdisk2            
hdisk2:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
lv01                  535   535   110..108..109..109..99 /1
loglv00               1     1     00..01..00..00..00    N/A
:/# lspv -l hdisk3
hdisk3:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
lv01                  535   535   110..108..109..109..99 /1
loglv00               1     1     00..01..00..00..00    N/A

/# lsvg -l secondvg
secondvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv00             jfslog     1     2     2    open/stale    N/A
lv01                jfs        535   1070  2    open/stale    /1


It says I need to synchronize the partitions. How do I sync?

syncvg -p hdisk2 hdisk3

or

syncvg -l lv01


Full layout of the disks as followed:

------ PHYSICAL DISKS ------
     
hdisk2          0058a00f1eac2217                    secondvg      
hdisk3          0058a00f1eb13c34                    secondvg    

------ VOLUME GROUP INFO (secondvg) ------
VOLUME GROUP:   secondvg                 VG IDENTIFIER:  0058a00f00004c00000000fa1eac36b5
VG STATE:       active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:  read/write               TOTAL PPs:      1092 (139776 megabytes)
MAX LVs:        256                      FREE PPs:       20 (2560 megabytes)
LVs:            2                        USED PPs:       1072 (137216 megabytes)
OPEN LVs:       2                        QUORUM:         1
TOTAL PVs:      2                        VG DESCRIPTORS: 3
STALE PVs:      2                        STALE PPs:      388
ACTIVE PVs:     1                        AUTO ON:        yes
MAX PPs per PV: 1016                     MAX PVs:        32
LTG size:       128 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:      no                                      

------ LOGICAL VOLUME NAME (secondvg) ------
secondvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv00             jfslog     1     2     2    open/stale    N/A
lv01                jfs        535   1070  2    open/stale    /1

------ DISK STATISTICS (hdisk2) ------
PHYSICAL VOLUME:    hdisk2                   VOLUME GROUP:     secondvg
PV IDENTIFIER:      0058a00f1eac2217 VG IDENTIFIER     0058a00f00004c00000000fa1eac36b5
PV STATE:           active                                    
STALE PARTITIONS:   1                        ALLOCATABLE:      yes
PP SIZE:            128 megabyte(s)          LOGICAL VOLUMES:  2
TOTAL PPs:          546 (69888 megabytes)    VG DESCRIPTORS:   2
FREE PPs:           10 (1280 megabytes)      HOT SPARE:        no
USED PPs:           536 (68608 megabytes)                      
FREE DISTRIBUTION:  00..00..00..00..10                        
USED DISTRIBUTION:  110..109..109..109..99                    

------ FILESYSTEMS (hdisk2) ------
hdisk2:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
lv01                  535   535   110..108..109..109..99 /1
loglv00               1     1     00..01..00..00..00    N/A

------ FREE SPACE (hdisk2) ------
hdisk2:
PP RANGE  STATE   REGION        LV NAME             TYPE       MOUNT POINT
  1-110   used    outer edge    lv01                jfs        /1
111-111   stale   outer middle  loglv00             jfslog     N/A
112-219   used    outer middle  lv01                jfs        /1
220-328   used    center        lv01                jfs        /1
329-437   used    inner middle  lv01                jfs        /1
438-536   used    inner edge    lv01                jfs        /1
537-546   free    inner edge                                  


------ DISK STATISTICS (hdisk3) ------
PHYSICAL VOLUME:    hdisk3                   VOLUME GROUP:     secondvg
PV IDENTIFIER:      0058a00f1eb13c34 VG IDENTIFIER     0058a00f00004c00000000fa1eac36b5
PV STATE:           missing                                    
STALE PARTITIONS:   387                      ALLOCATABLE:      yes
PP SIZE:            128 megabyte(s)          LOGICAL VOLUMES:  2
TOTAL PPs:          546 (69888 megabytes)    VG DESCRIPTORS:   1
FREE PPs:           10 (1280 megabytes)      HOT SPARE:        no
USED PPs:           536 (68608 megabytes)                      
FREE DISTRIBUTION:  00..00..00..00..10                        
USED DISTRIBUTION:  110..109..109..109..99                    

------ FILESYSTEMS (hdisk3) ------
hdisk3:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
lv01                  535   535   110..108..109..109..99 /1
loglv00               1     1     00..01..00..00..00    N/A

------ FREE SPACE (hdisk3) ------
hdisk3:
PP RANGE  STATE   REGION        LV NAME             TYPE       MOUNT POINT
  1-4     stale   outer edge    lv01                jfs        /1
  5-5     used    outer edge    lv01                jfs        /1
  6-6     stale   outer edge    lv01                jfs        /1
  7-7     used    outer edge    lv01                jfs        /1
  8-8     stale   outer edge    lv01                jfs        /1
  9-9     used    outer edge    lv01                jfs        /1
 10-18    stale   outer edge    lv01                jfs        /1
 19-19    used    outer edge    lv01                jfs        /1
 20-24    stale   outer edge    lv01                jfs        /1
 25-25    used    outer edge    lv01                jfs        /1
 26-26    stale   outer edge    lv01                jfs        /1
 27-27    used    outer edge    lv01                jfs        /1
 28-61    stale   outer edge    lv01                jfs        /1
 62-62    used    outer edge    lv01                jfs        /1
 63-80    stale   outer edge    lv01                jfs        /1
 81-81    used    outer edge    lv01                jfs        /1
 82-88    stale   outer edge    lv01                jfs        /1
 89-91    used    outer edge    lv01                jfs        /1
 92-110   stale   outer edge    lv01                jfs        /1
111-111   used    outer middle  loglv00             jfslog     N/A
112-122   stale   outer middle  lv01                jfs        /1
123-123   used    outer middle  lv01                jfs        /1
124-163   stale   outer middle  lv01                jfs        /1
164-165   used    outer middle  lv01                jfs        /1
166-168   stale   outer middle  lv01                jfs        /1
169-170   used    outer middle  lv01                jfs        /1
171-190   stale   outer middle  lv01                jfs        /1
191-191   used    outer middle  lv01                jfs        /1
192-195   stale   outer middle  lv01                jfs        /1
196-196   used    outer middle  lv01                jfs        /1
197-197   stale   outer middle  lv01                jfs        /1
198-198   used    outer middle  lv01                jfs        /1
199-199   stale   outer middle  lv01                jfs        /1
200-200   used    outer middle  lv01                jfs        /1
201-207   stale   outer middle  lv01                jfs        /1
208-208   used    outer middle  lv01                jfs        /1
209-219   stale   outer middle  lv01                jfs        /1
220-222   stale   center        lv01                jfs        /1
223-223   used    center        lv01                jfs        /1
224-229   stale   center        lv01                jfs        /1
230-230   used    center        lv01                jfs        /1
231-245   stale   center        lv01                jfs        /1
246-246   used    center        lv01                jfs        /1
247-250   stale   center        lv01                jfs        /1
251-251   used    center        lv01                jfs        /1
252-260   stale   center        lv01                jfs        /1
261-261   used    center        lv01                jfs        /1
262-265   stale   center        lv01                jfs        /1
266-266   used    center        lv01                jfs        /1
267-271   stale   center        lv01                jfs        /1
272-272   used    center        lv01                jfs        /1
273-274   stale   center        lv01                jfs        /1
275-277   used    center        lv01                jfs        /1
278-281   stale   center        lv01                jfs        /1
282-283   used    center        lv01                jfs        /1
284-284   stale   center        lv01                jfs        /1
285-285   used    center        lv01                jfs        /1
286-299   stale   center        lv01                jfs        /1
300-300   used    center        lv01                jfs        /1
301-328   stale   center        lv01                jfs        /1
329-329   stale   inner middle  lv01                jfs        /1
330-330   used    inner middle  lv01                jfs        /1
331-339   stale   inner middle  lv01                jfs        /1
340-340   used    inner middle  lv01                jfs        /1
341-346   stale   inner middle  lv01                jfs        /1
347-348   used    inner middle  lv01                jfs        /1
349-355   stale   inner middle  lv01                jfs        /1
356-357   used    inner middle  lv01                jfs        /1
358-378   stale   inner middle  lv01                jfs        /1
379-379   used    inner middle  lv01                jfs        /1
380-381   stale   inner middle  lv01                jfs        /1
382-383   used    inner middle  lv01                jfs        /1
384-392   stale   inner middle  lv01                jfs        /1
393-394   used    inner middle  lv01                jfs        /1
395-396   stale   inner middle  lv01                jfs        /1
397-397   used    inner middle  lv01                jfs        /1
398-420   stale   inner middle  lv01                jfs        /1
421-423   used    inner middle  lv01                jfs        /1
424-425   stale   inner middle  lv01                jfs        /1
426-428   used    inner middle  lv01                jfs        /1
429-429   stale   inner middle  lv01                jfs        /1
430-430   used    inner middle  lv01                jfs        /1
431-433   stale   inner middle  lv01                jfs        /1
434-437   used    inner middle  lv01                jfs        /1
438-440   used    inner edge    lv01                jfs        /1
441-449   stale   inner edge    lv01                jfs        /1
450-536   used    inner edge    lv01                jfs        /1
537-546   free    inner edge

I need to break this mirror in order to replace the disk.

Thanks.

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Most Valuable Expert 2013
Top Expert 2013

Commented:
Hi,

try a syncvg -l loglv00 first, and check if the stale partition would vanish.

If anything fails we can still remove the log LV and recreate it, don't worry!
Of course we must umount lv01 to accomplish this!


wmp

Author

Commented:
Unfortunately it's not syncing:

:/# syncvg -l loglv00
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume loglv00.

I can unmount in aprox. 1hour from now.
Most Valuable Expert 2013
Top Expert 2013
Commented:
OK,

I think we'll have to remove/recreate loglv00.

In the meantime you could already remove the hdisk3-based mirror of lv01:

rmlvcopy lv01 1 hdisk3

Once lv01 is umounted issue

rmlv -f loglv00

mklv -t jfslog -y loglv00 secondvg 1 hdisk2

logform /dev/loglv00

mount /dev/lv01

hdisk3 should now be free (check with "lspv hdisk3"). If so, remove it from the VG:

reducevg secondvg hdisk3
rmdev -dl hdisk3

You can now safely replace the hdisk.

wmp
Build an E-Commerce Site with Angular 5

Learn how to build an E-Commerce site with Angular 5, a JavaScript framework used by developers to build web, desktop, and mobile applications.

Author

Commented:
OK

I removed lv01 from hdisk3:

:/# rmlvcopy lv01 1 hdisk3
:/# lslv -m lv01
lv01:/1
LP    PP1  PV1               PP2  PV2               PP3  PV3
0001  0112 hdisk2            
0002  0113 hdisk2            
0003  0114 hdisk2            
0004  0115 hdisk2            
0005  0116 hdisk2            
0006  0117 hdisk2            
0007  0118 hdisk2            
0008  0119 hdisk2
...
...
...

My guess is I won't be able to unmount as there are lots of processes running on lv01.
Users are prompted to exit in 5 mins. I've changed /etc/filesystems not to mount /1 so I will reboot the box first.

I'll issue the rest of the commands when the box is back up again.

Thanks!
Most Valuable Expert 2013
Top Expert 2013

Commented:
Yes, sorry,

normally I'm not giving up so fast in rescuing an LV, but I had astonishingly nearly the same case here at EE only a few days ago, where we tried to and fro saving the log LV, but finally all efforts failed and we had to format the log.

Good luck!

wmp

Author

Commented:
Thanks! :)

just to be sure, by the looks of the commands it will not harm the data on /1?
Most Valuable Expert 2013
Top Expert 2013

Commented:
No,

we're going to work just on loglv00, not on any other LV. lv01 is safe.

Under some circumstances it could be possible that we'll have to make the new log LV known to the FS, but that's no problem.

Author

Commented:
OK i rebooted with lv01 /1 unmounted but can't seem to remove logvl00 now:

:/# rmlv -f loglv00
0516-010 lqueryvg: Volume group must be varied on; use varyonvg command.
0516-912 rmlv: Unable to remove logical volume loglv00.
:/# lsvg
rootvg
secondvg
:/# varyonvg secondvg
PV Status:      hdisk2  0058a00f1eac2217        PVACTIVE
                hdisk3  0058a00f1eb13c34        PVMISSING
0516-056 varyonvg: The volume group is not varied on because a
        physical volume is marked missing. Run diagnostics.        
Most Valuable Expert 2013
Top Expert 2013

Commented:
Why did you leave quorum checking turned on? That's very bad for a mirrored VG.

Anyway, issue

varyonvg -f secondvg

and continue as suggested.

Author

Commented:
Sorry, varyonvg -f secondvg

I removed loglv00, made a new one on hdisk2 and formatted it.

However /1 won't mount. It replayes the log for /1 but cannot mount.

PHYSICAL VOLUME:    hdisk2                   VOLUME GROUP:     secondvg
PV IDENTIFIER:      0058a00f1eac2217 VG IDENTIFIER     0058a00f00004c00000000fa1eac36b5
PV STATE:           active
STALE PARTITIONS:   0                        ALLOCATABLE:      yes
PP SIZE:            128 megabyte(s)          LOGICAL VOLUMES:  2
TOTAL PPs:          546 (69888 megabytes)    VG DESCRIPTORS:   2
FREE PPs:           10 (1280 megabytes)      HOT SPARE:        no
USED PPs:           536 (68608 megabytes)
FREE DISTRIBUTION:  00..00..00..00..10
USED DISTRIBUTION:  110..109..109..109..99

hdisk2:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
lv01                  535   535   110..108..109..109..99 /1
loglv00               1     1     00..01..00..00..00    N/A

hdisk2:
PP RANGE  STATE   REGION        LV NAME             TYPE       MOUNT POINT
  1-110   used    outer edge    lv01                jfs        /1
111-111   used    outer middle  loglv00             jfslog     N/A
112-219   used    outer middle  lv01                jfs        /1
220-328   used    center        lv01                jfs        /1
329-437   used    inner middle  lv01                jfs        /1
438-536   used    inner edge    lv01                jfs        /1
537-546   free    inner edge

Replaying log for /dev/lv01.
mount: 0506-324 Cannot mount /dev/lv01 on /1: The media is not formatted or the format is not correct.
0506-342 The superblock on /dev/lv01 is dirty.  Run a full fsck to fix.

Most Valuable Expert 2013
Top Expert 2013

Commented:
Yes, this can happen with that many hardware problems you have. It's not related to the new log LV.

Just run

fsck -y /dev/lv01

Author

Commented:
Sweet it mounts again. That took the heat off :)

first I'll let the system backup overnight just to be sure.  Users can get to their data again.

However hdisk3 is hdisk4 now :)

:/# lsvg -p secondvg
secondvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            546         10          00..00..00..00..10
hdisk3            missing           546         546         110..109..109..109..109
:/# lspv
hdisk0          0058a00fa8d8d5f1                    rootvg
hdisk1          0058a00f0a04fd03                    rootvg
hdisk2          0058a00f1eac2217                    secondvg
hdisk4          none                                None                          

Probably reducevg doesn't work but i'll look into that tomorrow.

Thanks wmp, lot's of respect!
Most Valuable Expert 2013
Top Expert 2013

Commented:
Issue

reducevg secondvg hdisk3

(it will work!),

then remove the defective disk physically from the machine if you didn't already do so,

then

rmdev -dl hdisk3
rmdev -dl hdisk4

then

cfgmgr

You will get a new hdisk3 (formerly known as hdisk4) and the old hdisk3 will be gone.
 
Thx for the "point's"!   ;-)

wmp

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial