It’s quite a hefty long title, and it still doesn’t quite convey what I want to write about in the post.
This post is about a specific situation that can occur whereby you may have accidentally deleted a data disk or recovered a VM that had “selective disk backup” enabled and you’re missing a data disk, of a Linux VM that had the data disk as part of a Logical Volume Manager (LVM) managed file system.
In this post I show how to recover the unbootable VM using a rescue VM, then repair the volume by adding a new data disk and eventually repairing the LVM volume group and the XFS file system.
In our setup, we have a SLES 12 VM (the victim) with the following disk architecture:
I actually have 3 data disks, but we will only be working with 2 of them.
The 2 data disk LUNs map to Linux physical disks /dev/sdd and /dev/sde and are part of volume group volTMP, which contains a logical volume lvTMP1 striped over the two disks and on lvTMP1 is an XFS file system mounted as “/BIGSTRIPEDDISK”.
I actually created this setup as part of this post here, so you can follow the instructions on that post to get to the same state if you wish.
I also have, ready to start up, a Ubuntu VM created using a basic Azure VM type (it’s a B1s) and an Azure Ubuntu Server 18 LTS image.
This will be my rescue VM. It’s small, light and fast to boot up.
You don’t have to use a Ubuntu VM, but you will need another VM that is running Linux and able to mount the file systems that you use for your root file systems (mine is ext4).
We Do the Damage
In this scenario we are deleting one of the data disks of the SLES 12 VM, from inside the Azure Portal.
The same situation could occur if you restored a VM from backup, but the VM had “selective disk backup” enabled, and restored with missing data disks.
The first thing we do, with the VM already shutdown, is remove the data disk (LUN2) from the Portal:
NOTE: We are not actually deleting the disk here in our test setup. It just detaches it from the VM. But imagine that we did detach and delete it completely.
Save the change:
We then start the VM:
The VM May Not Boot
Depending on your file system mount options and your O/S (I’m using SLES 12), by default the Linux VM will refuse to boot fully.
It will actually get stuck trying to mount the file system /BIGSTRIPEDDISK because the data disk is now missing (we deleted it!).
NOTE: If you have “nofail” in the fstab mount options, then your VM may boot normally, with the file system missing. You’re lucky. Skip though to the section on adding a new data disk (after section “Swap O/S Disk”).
The Linux O/S will go into recovery mode. If you have Boot Diagnostics enabled, you can verify this in the “Serial Console” within Azure Portal on the VM resource details screen.
In recovery mode, you are prompted to enter the root password to give you access to a basic shell. However, when deploying from Azure images, you don’t get a root user password, so you won’t know it!
If you don’t have Boot Diagnostics enabled, then you will be waiting a some minutes until the VM boot hits a timeout and Azure Portal informs you it failed to start:
In either of the above cases, you may end up at this same point. The VM will not boot due to the failed disk.
What we need to do to recover from this situation and allow our SLES 12 VM to boot, is to comment out the failed file system from the /etc/fstab file on the SLES 12 VM’s O/S disk.
This will involve the use of the handy “swap O/S disk” button in the Azure Portal.
Create an Image of the O/S Disk
We have to create a snapshot image of the existing SLES 12 VM O/S disk, because we cannot detach the O/S disk from the existing VM.
Locate the SLES 12 VM in the Portal and click it’s O/S disk:
Click the “Create Snapshot” button, then give the snapshot a useful name:
I used standard HDD (cheaper), but you can choose SSD if you wish:
Click to go to the snapshot once it has been created:
We now have an image of the O/S disk, which we can use to create a new O/S disk.
Create New Disk from Image
We will create a new managed disk from the image of the O/S disk.
This will allow us to mount it on our Ubuntu VM (the rescue VM).
From the Azure Portal create a brand new disk same size and specification as the original O/S disk.
NOTE: The Ubuntu VM is limited and may not support higher performing disk types like Ultra Disk. In which case you may need to create the new disk as a lower performance disk.
Select the image you created as the source and give the new disk a recognisable name:
Attach New Disk to Rescue VM
We now attach the new disk to the rescue VM (my Ubuntu VM) from the “disks” section of the Ubuntu VM resource:
It’s the first data disk, so is going on LUN 0:
Mounting the Disk on Rescue VM
Start the rescue VM (Ubuntu) if it is not already started, log onto the VM and either as root or using “sudo” check the disk devices present by running “lsblk”:
In my example the new disk is visible as /dev/sdc.
Because the disk is an O/S disk, it has partitions (it’s not a whole disk). For this reason, we have to mount the specific partition that the root (“/”) file system was mounted from.
In my case I can easily see that partition 4 (sdc4) because it is the largest partition on the /dev/sdc disk at 28.8G in size.
We have to create a location to mount the partition (“mkdir /mnt/suse_os_disk”) then mount partition 4 from sdc using the “mount” command:
The mount command is intelligent enough to know what file system is on the disk.
Adjust Fstab File
With the new disk mounted on the rescue VM, we can use our favourite text editor to adjust the fstab file and comment out the affected file system, to prevent it from being mounted.
We comment out /BIGSTRIPEDDISK :
Save the file changes.
We can now safely unmount the disk and then disconnect it from the rescue VM:
From the Azure Portal, we delete the new data disk from the rescue VM:
Swap O/S Disk
In Azure Portal, go to the SLES 12 VM and in the disks view of the VM, click the “Swap OS disk” button:
Select the new disk that we have just unmounted from the rescue VM:
Start the SLES VM and it will boot off the new disk:
The VM will boot up successfully.
Great stuff. All that effort and so far we have a booting VM.
We still have the initial problem, we deleted one of our data disks. We need to create a new data disk.
Add New Data Disk
In Azure Portal on the SLES VM, create a new data disk to the same specification as it existed originally.
You can guess if you are not sure, but you have to remember that it should be the same tier and size as other disks in a striped LVM logical volume.
Save the change:
Repair Volume Group
With the new data disk added, we can now start the process of repairing the volume group.
We execute a pvscan to list physical volumes on the VM:
In the above we can see that LVM is reporting a missing physical volume. This is the one we deleted.
Using “lsblk” we can see the new device right at the end, it’s /dev/sde:
We can create the new physical volume and apply the previous UUID to the disk, to make LVM think this is the same disk, then we get LVM to write the configuration backup to the new disk.
First, let’s check what LVM configuration backups we have for our volume group:
ls -ltr /etc/lvm/archive/volTMP*
We choose the latest one available before we lost the disk:
We can now re-create the physical volume, applying the previously used UUID and LVM configuration (metadata):
pvcreate --uuid '<previous missing uuid from the pvscan output>' /
--restorefile /etc/lvm/archive/volTMP_<latest>.vg /dev/sde
Now we tell LVM to restore itself into a working order using the configurations available on the disks:
Let’s check the status of our logical volume that exists in the volTMP volume group:
In the above we notice that the “a” (active) flag is not set, the logical volume is therefore not yet active.
Let’s activate it:
lvchange -ay /dev/volTMP/lvTMP1
You can see that it is now active. Great!
We have repaired LVM. We no longer get any warnings about missing disks when executing the LVM related commands like “pvs, lvs, vgs”.
Repair File System
If we were to try and mount the file system /BIGSTRIPEDDISK, it would show an XFS error, because our new disk does not yet have a file system on it.
The file system is in a strange status, because 50% of the blocks are on the disk that was not deleted, and 50% are non-existent, because they were on the disk that was deleted.
So we actually have to repair the file system.
Instead of repairing, we could have chosen to just apply a new file system with mkfs.xfs, but let’s do a repair and see what the process is.
xfs_repair -L /dev/mapper/volTMP-lvTMP1
We can now edit the fstab and uncomment our file system /BIGSTRIPEDDISK:
Finally, we try and mount the file system:
It worked, and it was a clean mount. Nice.
Where Are My Files
With our repaired file system mounted, we dive in and look for files:
Ah yes! It’s clean!
No files will exist because we lost the disk. The LVM striping that we use is for performance, not redundancy, which means when you have to re-create the disk and repair the file system, all files will be lost.
- Actually deleting data disks is not simple in the Azure Portal. Microsoft have done a good job to try and prevent you from doing it by mistake, but it is still possible to do it by accident and also through code.
- Turn on boot diagnostics on your VMs, it helps to see what is going on during boot.
- Add “nofail” to the mount options for the data disks on Debian based systems. This will allow them to boot even with missing data disks.
- When a data disk goes missing, that is actively mounted at Linux boot, the VM may not boot at all.
You could reset the all your root account passwords and securely store them, which would allow you to enter recovery mode, but this is not something that most companies do.
Be prepared and have a rescue VM ready to start up. This is the best option and could help in a number of scenarios.
- Once booting again, we can use LVM to help simply restore the state of the volumes and file systems. We don’t need to re-create the LVM setup.
- In a striped logical volume, we stripe for performance, not redundancy, you will lose data if you lose one of the data disks of a striped logical volume.
- Using the “selective disk backup” feature saves backup vault space, but it means you will need to use this process to restore the volume groups for missing disks! Be wary and plan ahead!
- Test backup & restore processes!
In another blog post, I will show how to automate the root disk snapshot and disk creation followed by attaching to another VM. We will have a single script that can be run to automate the whole process. This is useful to help fix other issues such as when you have enabled Linux HugePages with more memory than the VM has!
You may also be interested in: