Chapter 6. LVM Troubleshooting
This chapter provide instructions for troubleshooting a variety of LVM issues.
6.1. Troubleshooting Diagnostics
If a command is not working as expected, you can gather diagnostics in the following ways:
Use the -v
, -vv
, -vvv
, or -vvvv
argument of any command for increasingly verbose levels of output.
If the problem is related to the logical volume activation, set 'activation = 1' in the 'log' section of the configuration file and run the command with the -vvvv
argument. After you have finished examining this output be sure to reset this parameter to 0, to avoid possible problems with the machine locking during low memory situations.
Run the lvmdump
command, which provides an information dump for diagnostic purposes. For information, see the lvmdump
(8) man page.
Execute the lvs -v
, pvs -a
or dmsetup info -c
command for additional system information.
Examine the last backup of the metadata in the /etc/lvm/backup
file and archived versions in the /etc/lvm/archive
file.
Check the current configuration information by running the lvm dumpconfig
command.
Check the .cache
file in the /etc/lvm
directory for a record of which devices have physical volumes on them.
6.2. Displaying Information on Failed Devices
You can use the -P
argument of the lvs
or vgs
command to display information about a failed volume that would otherwise not appear in the output. This argument permits some operations even though the metadata is not completely consistent internally. For example, if one of the devices that made up the volume group vg
failed, the vgs
command might show the following output.
# vgs -o +devices
Volume group "vg" not found
If you specify the -P
argument of the vgs
command, the volume group is still unusable but you can see more information about the failed device.
# vgs -P -o +devices
Partial mode. Incomplete volume groups will be activated read-only. VG #PV #LV #SN Attr VSize VFree Devices vg 9 2 0 rz-pn- 2.11T 2.07T unknown device(0) vg 9 2 0 rz-pn- 2.11T 2.07T unknown device(5120),/dev/sda1(0)
In this example, the failed device caused both a linear and a striped logical volume in the volume group to fail. The lvs
command without the -P
argument shows the following output.
# lvs -a -o +devices
Volume group "vg" not found
Using the -P
argument shows the logical volumes that have failed.
# lvs -P -a -o +devices
Partial mode. Incomplete volume groups will be activated read-only. LV VG Attr LSize Origin Snap% Move Log Copy% Devices linear vg -wi-a- 20.00G unknown device(0) stripe vg -wi-a- 20.00G unknown device(5120),/dev/sda1(0)
The following examples show the output of the pvs
and lvs
commands with the -P
argument specified when a leg of a mirrored logical volume has failed.
# vgs -a -o +devices -P
Partial mode. Incomplete volume groups will be activated read-only. VG #PV #LV #SN Attr VSize VFree Devices corey 4 4 0 rz-pnc 1.58T 1.34T my_mirror_mimage_0(0),my_mirror_mimage_1(0) corey 4 4 0 rz-pnc 1.58T 1.34T /dev/sdd1(0) corey 4 4 0 rz-pnc 1.58T 1.34T unknown device(0) corey 4 4 0 rz-pnc 1.58T 1.34T /dev/sdb1(0)
# lvs -a -o +devices -P
Partial mode. Incomplete volume groups will be activated read-only. LV VG Attr LSize Origin Snap% Move Log Copy% Devices my_mirror corey mwi-a- 120.00G my_mirror_mlog 1.95 my_mirror_mimage_0(0),my_mirror_mimage_1(0) [my_mirror_mimage_0] corey iwi-ao 120.00G unknown device(0) [my_mirror_mimage_1] corey iwi-ao 120.00G /dev/sdb1(0) [my_mirror_mlog] corey lwi-ao 4.00M /dev/sdd1(0)
6.3. Recovering from LVM Mirror Failure
This section provides an example of recovering from a situation where one leg of an LVM mirrored volume fails because the underlying device for a physical volume goes down and the
mirror_log_fault_policy
parameter is set to
remove
, requiring that you manually rebuild the mirror. For information on setting the
mirror_log_fault_policy
parameter, refer to
Section 6.3, "Recovering from LVM Mirror Failure".
When a mirror leg fails, LVM converts the mirrored volume into a linear volume, which continues to operate as before but without the mirrored redundancy. At that point, you can add a new disk device to the system to use as a replacement physical device and rebuild the mirror.
The following command creates the physical volumes which will be used for the mirror.
# pvcreate /dev/sd[abcdefgh][12]
Physical volume "/dev/sda1" successfully created Physical volume "/dev/sda2" successfully created Physical volume "/dev/sdb1" successfully created Physical volume "/dev/sdb2" successfully created Physical volume "/dev/sdc1" successfully created Physical volume "/dev/sdc2" successfully created Physical volume "/dev/sdd1" successfully created Physical volume "/dev/sdd2" successfully created Physical volume "/dev/sde1" successfully created Physical volume "/dev/sde2" successfully created Physical volume "/dev/sdf1" successfully created Physical volume "/dev/sdf2" successfully created Physical volume "/dev/sdg1" successfully created Physical volume "/dev/sdg2" successfully created Physical volume "/dev/sdh1" successfully created Physical volume "/dev/sdh2" successfully created
The following commands creates the volume group vg
and the mirrored volume groupfs
.
# vgcreate vg /dev/sd[abcdefgh][12]
Volume group "vg" successfully created[root@link-08 ~]# lvcreate -L 750M -n groupfs -m 1 vg /dev/sda1 /dev/sdb1 /dev/sdc1
Rounding up size to full physical extent 752.00 MB Logical volume "groupfs" created
You can use the lvs
command to verify the layout of the mirrored volume and the underlying devices for the mirror leg and the mirror log. Note that in the first example the mirror is not yet completely synced; you should wait until the Copy%
field displays 100.00 before continuing.
# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices groupfs vg mwi-a- 752.00M groupfs_mlog 21.28 groupfs_mimage_0(0),groupfs_mimage_1(0) [groupfs_mimage_0] vg iwi-ao 752.00M /dev/sda1(0) [groupfs_mimage_1] vg iwi-ao 752.00M /dev/sdb1(0) [groupfs_mlog] vg lwi-ao 4.00M /dev/sdc1(0)[root@link-08 ~]# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices groupfs vg mwi-a- 752.00M groupfs_mlog 100.00 groupfs_mimage_0(0),groupfs_mimage_1(0) [groupfs_mimage_0] vg iwi-ao 752.00M /dev/sda1(0) [groupfs_mimage_1] vg iwi-ao 752.00M /dev/sdb1(0) [groupfs_mlog] vg lwi-ao 4.00M i /dev/sdc1(0)
In this example, the primary leg of the mirror /dev/sda1
fails. Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a dd
command
# dd if=/dev/zero of=/dev/vg/groupfs count=10
10+0 records in10+0 records out
You can use the lvs
command to verify that the device is now a linear device. Because of the failed disk, I/O errors occur.
# lvs -a -o +devices
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error /dev/sda2: read failed after 0 of 2048 at 0: Input/output error LV VG Attr LSize Origin Snap% Move Log Copy% Devices groupfs vg -wi-a- 752.00M /dev/sdb1(0)
At this point you should still be able to use the logical volume, but there will be no mirror redundancy.
To rebuild the mirrored volume, you replace the broken drive and recreate the physical volume. If you use the same disk rather than replacing it with a new one, you will see "inconsistent" warnings when you run the pvcreate
command. You can prevent that warning from appearing by executing the vgreduce --removemissing
command.
# pvcreate /dev/sdi[12]
Physical volume "/dev/sdi1" successfully created Physical volume "/dev/sdi2" successfully created[root@link-08 ~]# pvscan
PV /dev/sdb1 VG vg lvm2 [67.83 GB / 67.10 GB free] PV /dev/sdb2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdc1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdc2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdd1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdd2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sde1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sde2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdf1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdf2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdg1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdg2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdh1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdh2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdi1 lvm2 [603.94 GB] PV /dev/sdi2 lvm2 [603.94 GB] Total: 16 [2.11 TB] / in use: 14 [949.65 GB] / in no VG: 2 [1.18 TB]
Next you extend the original volume group with the new physical volume.
# vgextend vg /dev/sdi[12]
Volume group "vg" successfully extended# pvscan
PV /dev/sdb1 VG vg lvm2 [67.83 GB / 67.10 GB free] PV /dev/sdb2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdc1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdc2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdd1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdd2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sde1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sde2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdf1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdf2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdg1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdg2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdh1 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdh2 VG vg lvm2 [67.83 GB / 67.83 GB free] PV /dev/sdi1 VG vg lvm2 [603.93 GB / 603.93 GB free] PV /dev/sdi2 VG vg lvm2 [603.93 GB / 603.93 GB free] Total: 16 [2.11 TB] / in use: 16 [2.11 TB] / in no VG: 0 [0 ]
Convert the linear volume back to its original mirrored state.
# lvconvert -m 1 /dev/vg/groupfs /dev/sdi1 /dev/sdb1 /dev/sdc1
Logical volume mirror converted.
You can use the lvs
command to verify that the mirror is restored.
# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices groupfs vg mwi-a- 752.00M groupfs_mlog 68.62 groupfs_mimage_0(0),groupfs_mimage_1(0) [groupfs_mimage_0] vg iwi-ao 752.00M /dev/sdb1(0) [groupfs_mimage_1] vg iwi-ao 752.00M /dev/sdi1(0) [groupfs_mlog] vg lwi-ao 4.00M /dev/sdc1(0)
6.4. Recovering Physical Volume Metadata
If the volume group metadata area of a physical volume is accidentally overwritten or otherwise destroyed, you will get an error message indicating that the metadata area is incorrect, or that the system was unable to find a physical volume with a particular UUID. You may be able to recover the data the physical volume by writing a new metadata area on the physical volume specifying the same UUID as the lost metadata.
You should not attempt this procedure with a working LVM logical volume. You will lose your data if you specify the incorrect UUID.
The following example shows the sort of output you may see if the metadata area is missing or corrupted.
# lvs -a -o +devices
Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'. Couldn't find all physical volumes for volume group VG. Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'. Couldn't find all physical volumes for volume group VG. ...
You may be able to find the UUID for the physical volume that was overwritten by looking in the /etc/lvm/archive
directory. Look in the file VolumeGroupName_xxxx
.vg
for the last known valid archived LVM metadata for that volume group.
Alternately, you may find that deactivating the volume and setting the partial
(-P
) argument will enable you to find the UUID of the missing corrupted physical volume.
# vgchange -an --partial
Partial mode. Incomplete volume groups will be activated read-only. Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'. Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'. ...
Use the --uuid
and --restorefile
arguments of the pvcreate
command to restore the physical volume. The following example labels the /dev/sdh1
device as a physical volume with the UUID indicated above, FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk
. This command restores the physical volume label with the metadata information contained in VG_00050.vg
, the most recent good archived metadata for the volume group. The restorefile
argument instructs the pvcreate
command to make the new physical volume compatible with the old one on the volume group, ensuring that the new metadata will not be placed where the old physical volume contained data (which could happen, for example, if the original pvcreate
command had used the command line arguments that control metadata placement, or if the physical volume was originally created using a different version of the software that used different defaults). The pvcreate
command overwrites only the LVM metadata areas and does not affect the existing data areas.
# pvcreate --uuid "FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk" --restorefile /etc/lvm/archive/VG_00050.vg /dev/sdh1
Physical volume "/dev/sdh1" successfully created
You can then use the vgcfgrestore
command to restore the volume group's metadata.
# vgcfgrestore VG
Restored volume group VG
You can now display the logical volumes.
# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices stripe VG -wi--- 300.00G /dev/sdh1 (0),/dev/sda1(0) stripe VG -wi--- 300.00G /dev/sdh1 (34728),/dev/sdb1(0)
The following commands activate the volumes and display the active volumes.
# lvchange -ay /dev/VG/stripe
[root@link-07 backup]# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices stripe VG -wi-a- 300.00G /dev/sdh1 (0),/dev/sda1(0) stripe VG -wi-a- 300.00G /dev/sdh1 (34728),/dev/sdb1(0)
If the on-disk LVM metadata takes as least as much space as what overrode it, this command can recover the physical volume. If what overrode the metadata went past the metadata area, the data on the volume may have been affected. You might be able to use the fsck
command to recover that data.
6.5. Replacing a Missing Physical Volume
If a physical volume fails or otherwise needs to be replaced, you can label a new physical volume to replace the one that has been lost in the existing volume group by following the same procedure as you would for recovering physical volume metadata, described in
Section 6.4, "Recovering Physical Volume Metadata". You can use the
--partial
and
--verbose
arguments of the
vgdisplay
command to display the UUIDs and sizes of any physical volumes that are no longer present. If you wish to substitute another physical volume of the same size, you can use the
pvcreate
command with the
--restorefile
and
--uuid
arguments to initialize a new device with the same UUID as the missing physical volume. You can then use the
vgcfgrestore
command to restore the volume group's metadata.
6.6. Removing Lost Physical Volumes from a Volume Group
If you lose a physical volume, you can activate the remaining physical volumes in the volume group with the --partial
argument of the vgchange
command. You can remove all the logical volumes that used that physical volume from the volume group with the --removemissing
argument of the vgreduce
command.
It is recommended that you run the vgreduce
command with the --test
argument to verify what you will be destroying.
Like most LVM operations, the vgreduce
command is reversible in a sense if you immediately use the vgcfgrestore
command to restore the volume group metadata to its previous state. For example, if you used the --removemissing
argument of the vgreduce
command without the --test
argument and find you have removed logical volumes you wanted to keep, you can still replace the physical volume and use another vgcfgrestore
command to return the volume group to its previous state.
6.7. Insufficient Free Extents for a Logical Volume
You may get the error message "Insufficient free extents" when creating a logical volume when you think you have enough extents based on the output of the vgdisplay
or vgs
commands. This is because these commands round figures to 2 decimal places to provide human-readable output. To specify exact size, use free physical extent count instead of some multiple of bytes to determine the size of the logical volume.
The vgdisplay
command, by default, includes this line of output that indicates the free physical extents.
# vgdisplay
--- Volume group --- ... Free PE / Size 8780 / 34.30 GB
Alternately, you can use the vg_free_count
and vg_extent_count
arguments of the vgs
command to display the free extents and the total number of extents.
# vgs -o +vg_free_count,vg_extent_count
VG #PV #LV #SN Attr VSize VFree Free #Ext testvg 2 0 0 wz--n- 34.30G 34.30G 8780 8780
With 8780 free physical extents, you can run the following command, using the lower-case l argument to use extents instead of bytes:
# lvcreate -l8780 -n testlv testvg
This uses all the free extents in the volume group.
# vgs -o +vg_free_count,vg_extent_count
VG #PV #LV #SN Attr VSize VFree Free #Ext testvg 2 1 0 wz--n- 34.30G 0 0 8780