Replacing Faulty Disk in ROOTVG
Analyzing Disk Fault
The first signs that a hard disk is going faulty are temporary error log messages in Error Reporter. If you see random temporary errors, then you don't have an immediate problem but if you start to see a bundle of temporary errors then the disk will need replacing. The worse case scenario is permanent error against a hard disk and stale partitions.
Check to see how many errors have been logged and whether they are permanent of temporary by:
errpt |more
1581762B 0727203502 T H hdisk0 DISK OPERATION ERROR
1581762B 0727203502 P H hdisk0 DISK OPERATION ERROR
The first error log message shows that there is a temporary disk problem on hdisk0, whilst the second error log message shows a permanent error also on hdisk0. The procedures for replacing hdisk0 & hdisk1 <part of rootvg> are slightly different. See the steps below.
To check for stale partitons, run the command: lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 64 128 2 open/syncd N/A
hd8 jfslog 1 2 2 open/stale N/A
hd4 jfs 4 8 2 open/stale /
Steps for replacing faulty disks in other volume groups are much simpler than replacing disks in rootvg. I have written a procedure for this below also.
For procedures on replacing faulty SSA disk, refer to the link
Replacing hdisk0 in rootvg
Change bootlist
bosboot -a -d hdisk1 Make sure hdisk1 has a boot image
bootlist -m normal hdisk1 hdisk0 Change the bootlist so the system will use hdisk1 before hdisk0
Removing Primary Dump Device
sysdumpdev -l The primary dump device will always be on hdisk0, this will need to be changed
primary /dev/pdumplv
secondary /dev/sdumplv
copy directory /var/adm/dump
forced copy flag FALSE
always allow dump TRUE
dump compression ON
sysdumpdev -Pp /dev/hd6 Changes primary dump device
primary /dev/hd6
secondary /dev/sdumplv
copy directory /var/adm/dump
forced copy flag FALSE
always allow dump TRUE
dump compression ON
rmlv pdumplv Remove the logical volume pdumplv, the primary dump device
Un-Mirroring Hard Disk from VG
Now you need to un-mirror the volume group so the disk can be removed. There are two ways you can do this, one is whereby you run it at a disk level and the other is at a logical partition level. The outcome will be the same with both commands but with the second you have more control.
Method One
unmirrorvg rootvg hdisk0 Unmirrors the disk.
NB: Sometimes this is unstable, especially if you have stale partitions. I have also noticed that if pdumplv is mirrored <shouldn't be by default>, this command will fail. In this instance, unmirror the logical volume and then run the unmirrorvg command, alternatively follow the method below.
Method Two
lsvg -l rootvg Lists all logical volumes in rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 64 128 2 open/syncd N/A
hd8 jfslog 1 2 2 open/syncd N/A
hd4 jfs 4 8 2 open/syncd /
rmlvcopy LVNAME 1 hdisk0 Run this command for each logical volume
e.g: rmlvcopy hd5 1 hdisk0
Check the disk has been umirrored by: lsvg -l rootvg. For each LV, the PVs column will have 1
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 1 closed/syncd N/A
hd6 paging 64 128 1 open/syncd N/A
hd8 jfslog 1 2 1 open/syncd N/A
hd4 jfs 4 8 1 open/syncd /
Make a note of the SCSI id and serial number which will make the CE's life easier when he has to remove the disk. I have highlighted the SCSI id <8> and serial number <4DFJY156> from the example below. The command you need to run is. lscfg -vl hdisk0
DEVICE LOCATION DESCRIPTION
hdisk0 10-88-00-8,0 16 Bit LVD SCSI Disk Drive <9100 MB>
Manufacturer............................IBM
Machine Type and Model......DDYS-T09170M
FRU Number...........................00P1517
ROS Level and ID...................53394841
Serial Number.........................4DFJY156
EC Level...................................F79924
Part Number............................07N3852
Device Specific.<Z0>...............000003029F00013A
Device Specific.<Z1>...............07N4925
Device Specific.<Z2>...............0933
Device Specific.<Z3>...............00315
Device Specific.<Z4>...............0001
Device Specific.<Z5>...............22
Device Specific.<Z6>...............F79924
Remove the Disk from VG
reducevg rootvg hdisk0 Remove hdisk0 from the volume group
rmdev -l hdisk0 -d Remove the definition of hdisk0 from the system
lsvg rootvg Ensure disk is removed
lspv hdisk0 Ensure disk is removed
Now Remove the Disk physically and add the New Disk.
Add the New Disk to the System
cfgmgr Now run configuration Manager to add the new disk to the system
diag Then go into diagnostics to update the system log so the system is aware that hdisk0 has been replaced
Task Selection ->
Log Repair Action ->
hdisk0
Esc 0 To exit diagnostics after Log Repair Action has completed.
errpt | more Check Log Repair Action has taken place. You should see an entry like :-
2F3E09A4 0819110902 I H hdisk2 REPAIR ACTION
diag Go back into diagnostics and certify this disk. This will indicate whether the new disk is ok
Task Selection ->
Certify the disk ->
hdisk0 Commit the changes and exit by pressing F3
Esc 0 To exit diagnostics after Certifying the new disk
Add disk into the Volume Group
extendvg rootvg hdisk0 Add disk into the volume group rootvg
Now you need to re-mirror the disk. Again you can mirror at a disk level or at a logical level.
Re-Mirroring Hard Disk
Method One
mirrorvg rootvg hdisk0 Mirrors the disk
syncvg -v rootvg Synchronizes the volume group and the data contained within it
NB: This method will mirror the logical volume pdumplv. Unmirror the logical volume by:
rmlvcopy pdumplv 1 hdisk1
Method Two
lsvg -l rootvg Lists all the logical volumes to re-mirror
mklvcopy -k LVNAME 2 hdisk0 Run this command for each logical volume. This will also synchronize the data <-k>
e.g: mklvcopy hd5 hdisk0
NB: Do not mirror the logical volume pdumplv
syncvg -v rootvg Synchronizes the volume group and the data contained within it
lsvg -l rootvg Check datavg has been mirrored and status is open/syncd
Check the volume group has been completely re-mirrored by: lsvg -l rootvg. The PV column should have 2 for each LVNAME apart from pdumplv & sdumplv
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 64 128 2 open/syncd N/A
hd8 jfslog 1 2 2 open/syncd N/A
hd4 jfs 4 8 2 open/syncd /
mklv -y 'pdumplv' rootvg 40 hdisk0 Re-create the logical volume for your primary dump device
sysdumpdev -Pp /dev/pdumplv Re-alocate your primary dump device.
primary /dev/pdumplv
secondary /dev/sdumplv
copy directory /var/adm/dump
forced copy flag FALSE
always allow dump TRUE
dump compression ON
bosboot -a -d hdisk0 Update the boot image on hdisk0
bootlist -m normal hdisk0 hdisk1 Change your boot list back.
