0516-076 lreducelv: Cannot remove last good copy of stale partition.
    Resynchronize the partitions with syncvg and try again.
0516-922 rmlvcopy: Unable to remove logical partition copies from
    logical volume loglv00.
0516-1135 unmirrorvg: The unmirror of the volume group failed.
    The volume group is still partially or fully mirrored.
I never came across this message before.
hdisk2 and hdisk3 are in a software mirror. And from the looks of it it's still mirrored:
:/# lspv -l hdisk2 Â Â Â Â Â Â
hdisk2:
LV NAME        LPs  PPs  DISTRIBUTION      MOUNT POINT
lv01 Â Â Â Â Â Â Â Â Â 535 Â 535 Â 110..108..109..109..99 /1
loglv00 Â Â Â Â Â Â Â 1 Â Â 1 Â Â 00..01..00..00..00 Â Â N/A
:/# lspv -l hdisk3
hdisk3:
LV NAME        LPs  PPs  DISTRIBUTION      MOUNT POINT
lv01 Â Â Â Â Â Â Â Â Â 535 Â 535 Â 110..108..109..109..99 /1
loglv00 Â Â Â Â Â Â Â 1 Â Â 1 Â Â 00..01..00..00..00 Â Â N/A
/# lsvg -l secondvg
secondvg:
LV NAME       TYPE    LPs  PPs  PVs  LV STATE    MOUNT POINT
loglv00       jfslog   1   2   2   open/stale   N/A
lv01         jfs     535  1070  2   open/stale   /1
It says I need to synchronize the partitions. How do I sync?
syncvg -p hdisk2 hdisk3
or
syncvg -l lv01
Full layout of the disks as followed:
------ PHYSICAL DISKS ------
  Â
hdisk2      0058a00f1eac2217           secondvg   Â
hdisk3      0058a00f1eb13c34           secondvg  Â
------ VOLUME GROUP INFO (secondvg) ------
VOLUME GROUP:  secondvg         VG IDENTIFIER:  0058a00f00004c00000000fa1eac36b5
VG STATE:    active          PP SIZE:     128 megabyte(s)
VG PERMISSION:  read/write        TOTAL PPs:    1092 (139776 megabytes)
MAX LVs: Â Â Â Â 256 Â Â Â Â Â Â Â Â Â Â Â FREE PPs: Â Â Â 20 (2560 megabytes)
LVs: Â Â Â Â Â Â 2 Â Â Â Â Â Â Â Â Â Â Â Â USED PPs: Â Â Â 1072 (137216 megabytes)
OPEN LVs: Â Â Â 2 Â Â Â Â Â Â Â Â Â Â Â Â QUORUM: Â Â Â Â 1
TOTAL PVs: Â Â Â 2 Â Â Â Â Â Â Â Â Â Â Â Â VG DESCRIPTORS: 3
STALE PVs: Â Â Â 2 Â Â Â Â Â Â Â Â Â Â Â Â STALE PPs: Â Â Â 388
ACTIVE PVs: Â Â 1 Â Â Â Â Â Â Â Â Â Â Â Â AUTO ON: Â Â Â Â yes
MAX PPs per PV: 1016 Â Â Â Â Â Â Â Â Â Â MAX PVs: Â Â Â Â 32
LTG size: Â Â Â 128 kilobyte(s) Â Â Â Â Â AUTO SYNC: Â Â Â no
HOT SPARE:    no                   Â
------ LOGICAL VOLUME NAME (secondvg) ------
secondvg:
LV NAME       TYPE    LPs  PPs  PVs  LV STATE    MOUNT POINT
loglv00       jfslog   1   2   2   open/stale   N/A
lv01         jfs     535  1070  2   open/stale   /1
------ DISK STATISTICS (hdisk2) ------
PHYSICAL VOLUME: Â Â hdisk2 Â Â Â Â Â Â Â Â Â VOLUME GROUP: Â Â secondvg
PV IDENTIFIER: Â Â Â 0058a00f1eac2217 VG IDENTIFIER Â Â 0058a00f00004c00000000fa1eac36b5
PV STATE:      active                  Â
STALE PARTITIONS: Â 1 Â Â Â Â Â Â Â Â Â Â Â Â ALLOCATABLE: Â Â Â yes
PP SIZE: Â Â Â Â Â Â 128 megabyte(s) Â Â Â Â Â LOGICAL VOLUMES: Â 2
TOTAL PPs: Â Â Â Â Â 546 (69888 megabytes) Â Â VG DESCRIPTORS: Â 2
FREE PPs: Â Â Â Â Â 10 (1280 megabytes) Â Â Â HOT SPARE: Â Â Â Â no
USED PPs: Â Â Â Â Â 536 (68608 megabytes) Â Â Â Â Â Â Â Â Â Â Â
FREE DISTRIBUTION: Â 00..00..00..00..10 Â Â Â Â Â Â Â Â Â Â Â Â
USED DISTRIBUTION: Â 110..109..109..109..99 Â Â Â Â Â Â Â Â Â Â
------ FILESYSTEMS (hdisk2) ------
hdisk2:
LV NAME        LPs  PPs  DISTRIBUTION      MOUNT POINT
lv01 Â Â Â Â Â Â Â Â Â 535 Â 535 Â 110..108..109..109..99 /1
loglv00 Â Â Â Â Â Â Â 1 Â Â 1 Â Â 00..01..00..00..00 Â Â N/A
------ FREE SPACE (hdisk2) ------
hdisk2:
PP RANGE Â STATE Â REGION Â Â Â Â LV NAME Â Â Â Â Â Â TYPE Â Â Â MOUNT POINT
 1-110  used   outer edge   lv01         jfs     /1
111-111  stale  outer middle  loglv00       jfslog   N/A
112-219  used   outer middle  lv01         jfs     /1
220-328  used   center     lv01         jfs     /1
329-437  used   inner middle  lv01         jfs     /1
438-536  used   inner edge   lv01         jfs     /1
537-546  free   inner edge                 Â
------ DISK STATISTICS (hdisk3) ------
PHYSICAL VOLUME: Â Â hdisk3 Â Â Â Â Â Â Â Â Â VOLUME GROUP: Â Â secondvg
PV IDENTIFIER: Â Â Â 0058a00f1eb13c34 VG IDENTIFIER Â Â 0058a00f00004c00000000fa1eac36b5
PV STATE:      missing                  Â
STALE PARTITIONS: Â 387 Â Â Â Â Â Â Â Â Â Â Â ALLOCATABLE: Â Â Â yes
PP SIZE: Â Â Â Â Â Â 128 megabyte(s) Â Â Â Â Â LOGICAL VOLUMES: Â 2
TOTAL PPs: Â Â Â Â Â 546 (69888 megabytes) Â Â VG DESCRIPTORS: Â 1
FREE PPs: Â Â Â Â Â 10 (1280 megabytes) Â Â Â HOT SPARE: Â Â Â Â no
USED PPs: Â Â Â Â Â 536 (68608 megabytes) Â Â Â Â Â Â Â Â Â Â Â
FREE DISTRIBUTION: Â 00..00..00..00..10 Â Â Â Â Â Â Â Â Â Â Â Â
USED DISTRIBUTION: Â 110..109..109..109..99 Â Â Â Â Â Â Â Â Â Â
------ FILESYSTEMS (hdisk3) ------
hdisk3:
LV NAME        LPs  PPs  DISTRIBUTION      MOUNT POINT
lv01 Â Â Â Â Â Â Â Â Â 535 Â 535 Â 110..108..109..109..99 /1
loglv00 Â Â Â Â Â Â Â 1 Â Â 1 Â Â 00..01..00..00..00 Â Â N/A
------ FREE SPACE (hdisk3) ------
hdisk3:
PP RANGE Â STATE Â REGION Â Â Â Â LV NAME Â Â Â Â Â Â TYPE Â Â Â MOUNT POINT
 1-4   stale  outer edge   lv01         jfs     /1
 5-5   used   outer edge   lv01         jfs     /1
 6-6   stale  outer edge   lv01         jfs     /1
 7-7   used   outer edge   lv01         jfs     /1
 8-8   stale  outer edge   lv01         jfs     /1
 9-9   used   outer edge   lv01         jfs     /1
 10-18   stale  outer edge   lv01         jfs     /1
 19-19   used   outer edge   lv01         jfs     /1
 20-24   stale  outer edge   lv01         jfs     /1
 25-25   used   outer edge   lv01         jfs     /1
 26-26   stale  outer edge   lv01         jfs     /1
 27-27   used   outer edge   lv01         jfs     /1
 28-61   stale  outer edge   lv01         jfs     /1
 62-62   used   outer edge   lv01         jfs     /1
 63-80   stale  outer edge   lv01         jfs     /1
 81-81   used   outer edge   lv01         jfs     /1
 82-88   stale  outer edge   lv01         jfs     /1
 89-91   used   outer edge   lv01         jfs     /1
 92-110  stale  outer edge   lv01         jfs     /1
111-111  used   outer middle  loglv00       jfslog   N/A
112-122  stale  outer middle  lv01         jfs     /1
123-123  used   outer middle  lv01         jfs     /1
124-163  stale  outer middle  lv01         jfs     /1
164-165  used   outer middle  lv01         jfs     /1
166-168  stale  outer middle  lv01         jfs     /1
169-170  used   outer middle  lv01         jfs     /1
171-190  stale  outer middle  lv01         jfs     /1
191-191  used   outer middle  lv01         jfs     /1
192-195  stale  outer middle  lv01         jfs     /1
196-196  used   outer middle  lv01         jfs     /1
197-197  stale  outer middle  lv01         jfs     /1
198-198  used   outer middle  lv01         jfs     /1
199-199  stale  outer middle  lv01         jfs     /1
200-200  used   outer middle  lv01         jfs     /1
201-207  stale  outer middle  lv01         jfs     /1
208-208  used   outer middle  lv01         jfs     /1
209-219  stale  outer middle  lv01         jfs     /1
220-222  stale  center     lv01         jfs     /1
223-223  used   center     lv01         jfs     /1
224-229  stale  center     lv01         jfs     /1
230-230  used   center     lv01         jfs     /1
231-245  stale  center     lv01         jfs     /1
246-246  used   center     lv01         jfs     /1
247-250  stale  center     lv01         jfs     /1
251-251  used   center     lv01         jfs     /1
252-260  stale  center     lv01         jfs     /1
261-261  used   center     lv01         jfs     /1
262-265  stale  center     lv01         jfs     /1
266-266  used   center     lv01         jfs     /1
267-271  stale  center     lv01         jfs     /1
272-272  used   center     lv01         jfs     /1
273-274  stale  center     lv01         jfs     /1
275-277  used   center     lv01         jfs     /1
278-281  stale  center     lv01         jfs     /1
282-283  used   center     lv01         jfs     /1
284-284  stale  center     lv01         jfs     /1
285-285  used   center     lv01         jfs     /1
286-299  stale  center     lv01         jfs     /1
300-300  used   center     lv01         jfs     /1
301-328  stale  center     lv01         jfs     /1
329-329  stale  inner middle  lv01         jfs     /1
330-330  used   inner middle  lv01         jfs     /1
331-339  stale  inner middle  lv01         jfs     /1
340-340  used   inner middle  lv01         jfs     /1
341-346  stale  inner middle  lv01         jfs     /1
347-348  used   inner middle  lv01         jfs     /1
349-355  stale  inner middle  lv01         jfs     /1
356-357  used   inner middle  lv01         jfs     /1
358-378  stale  inner middle  lv01         jfs     /1
379-379  used   inner middle  lv01         jfs     /1
380-381  stale  inner middle  lv01         jfs     /1
382-383  used   inner middle  lv01         jfs     /1
384-392  stale  inner middle  lv01         jfs     /1
393-394  used   inner middle  lv01         jfs     /1
395-396  stale  inner middle  lv01         jfs     /1
397-397  used   inner middle  lv01         jfs     /1
398-420  stale  inner middle  lv01         jfs     /1
421-423  used   inner middle  lv01         jfs     /1
424-425  stale  inner middle  lv01         jfs     /1
426-428  used   inner middle  lv01         jfs     /1
429-429  stale  inner middle  lv01         jfs     /1
430-430  used   inner middle  lv01         jfs     /1
431-433  stale  inner middle  lv01         jfs     /1
434-437  used   inner middle  lv01         jfs     /1
438-440  used   inner edge   lv01         jfs     /1
441-449  stale  inner edge   lv01         jfs     /1
450-536  used   inner edge   lv01         jfs     /1
537-546  free   inner edge
I need to break this mirror in order to replace the disk.
Thanks.
Unix OS
Last Comment
woolmilkporc
8/22/2022 - Mon
woolmilkporc
Hi,
try a syncvg -l loglv00 first, and check if the stale partition would vanish.
If anything fails we can still remove the log LV and recreate it, don't worry!
Of course we must umount lv01 to accomplish this!
:/# rmlvcopy lv01 1 hdisk3
:/# lslv -m lv01
lv01:/1
LP Â Â PP1 Â PV1 Â Â Â Â Â Â Â PP2 Â PV2 Â Â Â Â Â Â Â PP3 Â PV3
0001 Â 0112 hdisk2 Â Â Â Â Â Â
0002 Â 0113 hdisk2 Â Â Â Â Â Â
0003 Â 0114 hdisk2 Â Â Â Â Â Â
0004 Â 0115 hdisk2 Â Â Â Â Â Â
0005 Â 0116 hdisk2 Â Â Â Â Â Â
0006 Â 0117 hdisk2 Â Â Â Â Â Â
0007 Â 0118 hdisk2 Â Â Â Â Â Â
0008 Â 0119 hdisk2
...
...
...
My guess is I won't be able to unmount as there are lots of processes running on lv01.
Users are prompted to exit in 5 mins. I've changed /etc/filesystems not to mount /1 so I will reboot the box first.
I'll issue the rest of the commands when the box is back up again.
normally I'm not giving up so fast in rescuing an LV, but I had astonishingly nearly the same case here at EE only a few days ago, where we tried to and fro saving the log LV, but finally all efforts failed and we had to format the log.
Good luck!
wmp
dfke
ASKER
Thanks! :)
just to be sure, by the looks of the commands it will not harm the data on /1?
woolmilkporc
No,
we're going to work just on loglv00, not on any other LV. lv01 is safe.
Under some circumstances it could be possible that we'll have to make the new log LV known to the FS, but that's no problem.
OK i rebooted with lv01 /1 unmounted but can't seem to remove logvl00 now:
:/# rmlv -f loglv00
0516-010 lqueryvg: Volume group must be varied on; use varyonvg command.
0516-912 rmlv: Unable to remove logical volume loglv00.
:/# lsvg
rootvg
secondvg
:/# varyonvg secondvg
PV Status: Â Â Â hdisk2 Â 0058a00f1eac2217 Â Â Â Â PVACTIVE
        hdisk3  0058a00f1eb13c34     PVMISSING
0516-056 varyonvg: The volume group is not varied on because a
    physical volume is marked missing. Run diagnostics.    Â
woolmilkporc
Why did you leave quorum checking turned on? That's very bad for a mirrored VG.
Anyway, issue
varyonvg -f secondvg
and continue as suggested.
dfke
ASKER
Sorry, varyonvg -f secondvg
I removed loglv00, made a new one on hdisk2 and formatted it.
However /1 won't mount. It replayes the log for /1 but cannot mount.
PHYSICAL VOLUME: Â Â hdisk2 Â Â Â Â Â Â Â Â Â VOLUME GROUP: Â Â secondvg
PV IDENTIFIER: Â Â Â 0058a00f1eac2217 VG IDENTIFIER Â Â 0058a00f00004c00000000fa1eac36b5
PV STATE: Â Â Â Â Â active
STALE PARTITIONS: Â 0 Â Â Â Â Â Â Â Â Â Â Â Â ALLOCATABLE: Â Â Â yes
PP SIZE: Â Â Â Â Â Â 128 megabyte(s) Â Â Â Â Â LOGICAL VOLUMES: Â 2
TOTAL PPs: Â Â Â Â Â 546 (69888 megabytes) Â Â VG DESCRIPTORS: Â 2
FREE PPs: Â Â Â Â Â 10 (1280 megabytes) Â Â Â HOT SPARE: Â Â Â Â no
USED PPs: Â Â Â Â Â 536 (68608 megabytes)
FREE DISTRIBUTION: Â 00..00..00..00..10
USED DISTRIBUTION: Â 110..109..109..109..99
hdisk2:
LV NAME        LPs  PPs  DISTRIBUTION      MOUNT POINT
lv01 Â Â Â Â Â Â Â Â Â 535 Â 535 Â 110..108..109..109..99 /1
loglv00 Â Â Â Â Â Â Â 1 Â Â 1 Â Â 00..01..00..00..00 Â Â N/A
hdisk2:
PP RANGE Â STATE Â REGION Â Â Â Â LV NAME Â Â Â Â Â Â TYPE Â Â Â MOUNT POINT
 1-110  used   outer edge   lv01         jfs     /1
111-111  used   outer middle  loglv00       jfslog   N/A
112-219  used   outer middle  lv01         jfs     /1
220-328  used   center     lv01         jfs     /1
329-437  used   inner middle  lv01         jfs     /1
438-536  used   inner edge   lv01         jfs     /1
537-546  free   inner edge
Replaying log for /dev/lv01.
mount: 0506-324 Cannot mount /dev/lv01 on /1: The media is not formatted or the format is not correct.
0506-342 The superblock on /dev/lv01 is dirty. Â Run a full fsck to fix.
Yes, this can happen with that many hardware problems you have. It's not related to the new log LV.
Just run
fsck -y /dev/lv01
dfke
ASKER
Sweet it mounts again. That took the heat off :)
first I'll let the system backup overnight just to be sure. Â Users can get to their data again.
However hdisk3 is hdisk4 now :)
:/# lsvg -p secondvg
secondvg:
PV_NAME      PV STATE      TOTAL PPs  FREE PPs   FREE DISTRIBUTION
hdisk2       active       546     10      00..00..00..00..10
hdisk3       missing      546     546     110..109..109..109..109
:/# lspv
hdisk0 Â Â Â Â Â 0058a00fa8d8d5f1 Â Â Â Â Â Â Â Â Â Â rootvg
hdisk1 Â Â Â Â Â 0058a00f0a04fd03 Â Â Â Â Â Â Â Â Â Â rootvg
hdisk2 Â Â Â Â Â 0058a00f1eac2217 Â Â Â Â Â Â Â Â Â Â secondvg
hdisk4      none                 None             Â
Probably reducevg doesn't work but i'll look into that tomorrow.
Thanks wmp, lot's of respect!
woolmilkporc
Issue
reducevg secondvg hdisk3
(it will work!),
then remove the defective disk physically from the machine if you didn't already do so,
then
rmdev -dl hdisk3
rmdev -dl hdisk4
then
cfgmgr
You will get a new hdisk3 (formerly known as hdisk4) and the old hdisk3 will be gone.
Â
Thx for the "point's"! Â ;-)
try a syncvg -l loglv00 first, and check if the stale partition would vanish.
If anything fails we can still remove the log LV and recreate it, don't worry!
Of course we must umount lv01 to accomplish this!
wmp