Converting a non-LVM Linux system to LVM + RAID1



A few weeks ago I thought about upgrading my home server to use LVM and RAID1, but before I had time to get it done, my server suddenly refused to boot (I'd shut it down to add some RAM). Instead it would just reset after POST. I discovered that it would boot from a different SATA hard drive, but only if I disconnected its usual boot drive. It would also boot from an IDE CD drive, no matter which SATA drives were connected, and the data on all drives was intact and readable.

I'd just bought a couple of 2TB drives to increase the storage on my MythTV box, so to get the server back up as quickly as possible, I created a couple of partitions on one of the drives (one swap, one data), copied everything from the old drive to the new one, installed GRUB and then booted the system again. I also installed the second drive, ready for the LVM + RAID1 conversion.

The server was originally installed with Ubuntu Dapper and has undergone several dist-upgrades since then. When I first installed it, I had only a single drive so RAID wasn't an option, and I'd never used LVM and didn't want to experiment with it at the time, so I opted for a number of fixed size partitions.


  1. The system is running off /dev/sda and doesn't use LVM or RAID.
  2. You wish to setup /dev/sdb to be your new LVM + RAID1 drive and you plan to use two identically sized drives in your RAID1 array.
  3. You're using GRUB legacy as your boot loader.
  4. You already have lvm2 and mdadm installed.
  5. The system is running a Debian or a Debian-based distribution such as Ubuntu. Most of these instructions will be suitable for other distros, but the GRUB configuration will probably differ.
  6. You want to minimise your down time. Depending upon how much data you have to copy, it can take several hours to populate the new drive, so these instructions don't assume that you've shutdown to single user mode and that your old drive's contents will be unchanged while you build the new one.

Step 0 - Backup

Before you start, make sure you backup your system, or at least, anything that's important. If you make a mistake, you could destroy all of your data. You probably won't, but it could happen. You should always have an up-to-date backup anyway, but even if you do, check it and make sure it's OK before you go any further.

Step 1 - Partition the disc

/dev/sda was my existing 2TB drive and /dev/sdb was the unused 2TB drive that I was going to use to setup LVM + RAID.

The first step is to decide how you want to partition your new drive. You can't have /boot on an LVM volume, so it needs to be on a separate partition (I have heard that GRUB2 may support booting from LVM volumes, but GRUB legacy and LILO don't). You can then choose whether to have swap on an LVM volume or on a separate partition. I chose the latter. The rest of the drive can be one partition that will be managed by LVM.

So, I created three primary partitions. /dev/sdb1 (/boot) is 256MB, /dev/sdb2 (swap) is 6GB and /dev/sdb3 (lvm) is the rest of the drive. The partition type of all three is 0xfd (Linux raid autodetect).

Step 2 - Create your RAID1 arrays

Next you need to create degraded RAID1 arrays using the new drive:

mdadm -C -l1 -n2 /dev/md0 /dev/sdb1 missing mdadm -C -l1 -n2 /dev/md1 /dev/sdb2 missing mdadm -C -l1 -n2 /dev/md2 /dev/sdb3 missing

If you chose to put swap on your LVM partition then you'll only have two partitions and hence only two RAID1 arrays.

Step 3 - Create your filesystems

Format the non-LVM partitions:

mke2fs -j /dev/md0 mkswap /dev/md1

Step 4 - Create your LVM volumes

You now need to decide how many partitions you want to setup. I chose to have separate volumes for /, /home, /tmp, /usr, /usr/local and /var, and a large /storage volume for all my data, because that works for me, but if you just want one big partition then that'll work too. Just setup the logical volumes according to your own preferred paritioning scheme.

You'll first need to initialise the LVM partition and create a volume group. You can call your volume group whatever you want, e.g. the hostname, but I chose simply to use "vg0":

pvcreate /dev/md2 vgcreate vg0 /dev/md2

Now you're ready to create the various logical volumes you want:

lvcreate -L16G -nroot vg0 lvcreate -L300G -nhome vg0 lvcreate -L16G -nusr vg0 lvcreate -L8G -ntmp vg0 lvcreate -L32G -nvar vg0 lvcreate -L4G -nusrlocal vg0 lvcreate -L1024G -nstorage vg0

You'll now be able to access all of these logical volume via their device names /dev/mapper/<vg_name>-<lv_name>.

Step 5 - Format the LVM volumes

I chose to use ext3 for all of my volumes, so formatting required only a simple loop:

for i in /dev/mapper/vg0-*; do echo $i; mke2fs -j $i; done

Step 6 - Copy data from your running system

Now you need to mount the new partitions and logical volumes in preparation for copying over your data. I chose to mount them all under /media/lvm:

mkdir /media/lvm mount /dev/mapper/vg0-root /media/lvm/ mkdir /media/lvm/{boot,home,proc,storage,sys,tmp,usr,var} mount /dev/md0 /media/lvm/boot/ mount /dev/mapper/vg0-home /media/lvm/home/ mkdir /media/lvm/storage/ mount /dev/mapper/vg0-storage /media/lvm/storage/ mount /dev/mapper/vg0-tmp /media/lvm/tmp/ mount /dev/mapper/vg0-usr /media/lvm/usr/ mkdir /media/lvm/usr/local mount /dev/mapper/vg0-usrlocal /media/lvm/usr/local/ mount /dev/mapper/vg0-var /media/lvm/var/

There are a few different ways to copy the data, but I chose to use rsync because I'm familiar with it. There are a number of exclusions; each is explained below:

rsync -av --exclude=lost+found \ --exclude=/media/lvm \ --exclude='/dev/.[isu]*' \ --exclude='/lib/init/rw/*' \ --exclude='/lib/modules/*/volatile/*' \ --exclude='/proc/*' \ --exclude='/sys/*' \ --exclude='/home/*/.gvfs' \ --exclude='/var/lock/*' \ --exclude='/var/run/*' \ / /media/lvm/

lost+found is a directory created at the root of every ext2/ext3 partition, and there's no need to copy these directories across, especially if the new drive doesn't have the same partition layout as the original. This directory is where fsck puts files or directories it recovers from a corrupted partition and is normally empty.

/media/lvm is where I've mounted the new drive and I don't want to recursively copy itself :-)

These exclusions in /dev are files or directories that are created during boot, or are tmpfs mounts (RAM discs) added during system initialisation.

/lib/init/rw/ is another tmpfs mount, and only the mount point is needed on your new drive.

/lib/modules/*/volatile are also tmpfs mounts, and you must not have the file .mounted present in these directories on your disc. This file is used during system initialisation to decide whether or not to mount this filesystem. At that point, the root filesystem is still mounted read-only so you'll get errors when the system tries to load files into this directory.

/proc and /sys are virtual filesystems and only the mount points are needed.

/home/*/.gvfs are virtual filesystems and are only readable by the appropriate users, and not even root can access them. Excluding them from the copy avoids the errors that result from trying to copy them.

/var/lock and /var/run are two more tmpfs filesystems and only the mount points are needed.

Later, you'll need to do another final sync and this time you'll need to add a few more exlusions to avoid overwriting files which you need to modify to get the new drive to boot.

Step 7 - Setup the new drive to boot

Create an mdadm config file for the new system:

mdadm -Es >> /media/lvm/etc/mdadm/mdadm.conf

Check it once it's been created to be sure that it contains the correct RAID arrays. On my system, the scan gave me two entries for /dev/md2 (but with the UUID of /dev/md1 in one of them). That could have been due to me experimenting with md devices because I can't reproduce it now.

To create a new initramfs, you need two files from the virtual filesystem /proc. You'll be chrooting into the new drive so you'll need to copy these over temporarily:

cp /proc/cmdline /media/lvm/proc/ cp /proc/modules /media/lvm/proc/

Now edit /media/lvm/proc/cmdline and replace the root drive (it'll either be a partition name or a UUID) with /dev/mapper/vg0-root (change as appropriate to match your volume group and logical volume naming scheme):

root=/dev/mapper/vg0-root ro

Edit /media/lvm/etc/fstab so that the new LVM volumes are mounted in all the right places. Here's what mine now contains (note the non-LVM /boot and swap:

# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/mapper/vg0-root / ext3 defaults,errors=remount-ro,relatime 0 1 /dev/md0 /boot ext3 defaults,relatime 0 2 /dev/mapper/vg0-home /home ext3 defaults,relatime 0 2 /dev/mapper/vg0-tmp /tmp ext3 defaults,relatime 0 2 /dev/mapper/vg0-usr /usr ext3 defaults,relatime 0 2 /dev/mapper/vg0-var /var ext3 defaults,relatime 0 2 /dev/mapper/vg0-usrlocal /usr/local ext3 defaults,relatime 0 2 /dev/mapper/vg0-storage /storage/ ext3 defaults,relatime 0 2 /dev/md1 none swap sw 0 0 /dev/cdrom /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0

Edit your new GRUB menu in /media/lvm/boot/grub/menu.lst. I'm using GRUB legacy; if you're using GRUB2 you'll have to figure this bit out for yourself.

Comments start with "#", except that those after the "BEGIN AUTOMAGIC KERNELS LIST" line are special. In this block, lines starting with "##" are comments, but lines starting with just a single "#" are used by update-grub to add new boot stanzas when installing new kernels. You will need to change some of these lines.

The first of thse is the name of the root device passed to the kernel at boot time. This is one of those special comments, and contains the keyword kopt. Here's the relevant part of mine:

## default kernel options ## default kernel options for automagic boot options ## If you want special options for specific kernels use kopt_x_y_z ## where x.y.z is kernel version. Minor versions can be omitted. ## e.g. kopt=root=/dev/hda1 ro ## kopt_2_6_8=root=/dev/hdc1 ro ## kopt_2_6_8_2_686=root=/dev/hdc2 ro # kopt=root=UUID=3c3d9cda-031e-4903-93e1-1ddc34d1c9ec ro

Change the kopt line to:

# kopt=root=/dev/mapper/vg0-root ro

Next you need to change grub's root device. This is the partition which contains /boot and is defined in this part of menu.lst:

## default grub root device ## e.g. groot=(hd0,0) # groot=(hd0,0)

I changed mine to use the UUID of the boot partiton (/dev/md0). blkid /dev/md0 will give you the UUID. Note that you don't include UUID= in this line:

# groot=6f13e634-8f52-4aae-8df2-ebe0eb72bb91

Next you need to change all of the existing boot stanzas in the new menu.lst. Here's mine before I started:

title Ubuntu 8.10, kernel 2.6.27-17-generic root (hd0,0) kernel /boot/vmlinuz-2.6.27-17-generic root=UUID=3c3d9cda-031e-4903-93e1-1ddc34d1c9ec ro initrd /boot/initrd.img-2.6.27-17-generic

And here it is again after the changes:

title Ubuntu 8.10, kernel 2.6.27-17-generic uuid 6f13e634-8f52-4aae-8df2-ebe0eb72bb91 kernel /vmlinuz-2.6.27-17-generic root=/dev/mapper/vg0-root ro initrd /initrd.img-2.6.27-17-generic

The changes are:

You'll need to make similar changes to all boot stanzas in the file.

You may also want to increase the boot timeout to give you time to select single user mode the first time you boot from the new drive. This is the timeout option and comes before the "BEGIN AUTOMAGIC KERNELS LIST" line. I set mine to 10 seconds. That's the default but I'd reduced mine, so I bumped it up again.

Now you're ready to create a new initramfs (previously known as an initrd):

chroot /media/lvm update-initramfs -c -k `uname -r`

If you have multiple kernels installed, you'll need to run update-initramfs for each kernel. Put the kernel version number in place of `uname -r` in the above example. Once you've finished, delete the two files you copied into the new /proc.

Now you can install GRUB in the MBR of the new drive. Exit from the chroot shell and run:

grub-install --root-directory=/media/lvm /dev/sdb

The previous version of this page had grub-install being run from within the chroot shell and without the --root-directory option, but I discovered when replacing the drive in an older machine that grub-install uses /etc/mtab to find the partition containing /boot when verifying that the stage 1, 1.5 and 2 files are copied correctly. If grub-install gives you an error message like The file /boot/grub/stage2 not read correctly, you know it's comparing the files on the old drive with those it tried to install on the new one, either because you tried to install GRUB from within the chroot, or you didn't pass the right command line options to grub-install.

Step 8 - Prepare to migrate to the new drive

If you don't care too much about minimising downtime, you can skip this step and just do one resync in single user mode. This intermediate sync just means that almost all of the data changes are sync'ed before dropping into single user mode, making the final sync quicker and hence minimising the time the system is offline.

Do another data sync to ensure that the old and new drives contain (almost) the same data:

rsync -av --delete --exclude=lost+found \ --exclude=/media/lvm \ --exclude=/boot \ --exclude=/dev \ --exclude=/etc/fstab \ --exclude=/etc/mdadm/mdadm.conf \ --exclude=/lib/init/rw \ --exclude='/lib/modules/*/volatile/*' \ --exclude=/proc \ --exclude=/sys \ --exclude='/home/*/.gvfs' \ --exclude=/var/lib/initramfs-tools/ \ --exclude=/var/lock \ --exclude=/var/run \ / /media/lvm/

Note that this time I've added --delete and a few more exclusions. The former is because we want to make sure that any files deleted on the old drive while we've been working on the migration are deleted from the new drive too.

/boot is now excluded so that we don't overwrite the new GRUB menu and the new initramfs. I also found that some of the other files in /boot were changed by grub-install, and I didn't want to overwrite them. These extra changes might have been because the version of grub on the system is more recent that the one that was first installed a few years ago. Unless you've installed or removed a kernel since you started this conversion, it's highly unlikely that any files in this directory have been changed on the original drive, so it's easiest to simply exclude this directory from the sync.

/etc/fstab and etc/mdadm/mdadm.conf are files that you've changed on the new drive and you do not want to lose those changes.

/var/lib/initramfs-tools/ contains one directory for each installed kernel, and each directory contains a file which has the SHA1 checksum of that kernel's initramfs. You've rebuilt the initramfs for each kernel on the new drive, so these files will have been changed and you want to keep those changes.

Some of the earlier exclusions have had the trailing /* removed. This isn't entirely necessary but doesn't hurt. The /* was needed on the first copy so that the directories were created without their contents, but now that the directories are on the new drive they can be excluded entirely from the resync.

Step 9 - Final sync and reboot

Change to single user mode by running telinit 1 as root, then you'll need to do a final data sync before rebooting using the same rsync command as the sync in the previous section.

Once this sync has been done you can shutdown and reboot. You can choose to swap the old and new drives over so that the new drive becomes /dev/sda and the old one /dev/sdb, or you can change the boot order in the BIOS to make sure you boot from the new drive. I did the former. If you're replacing your old drive with new drives for your RAID array, then you can remove the old drive and make sure that both of your new drives are installed.

When you reboot, I suggest you select single user mode first, just in case something goes wrong. Watch the boot messages carefully, looking for anything unusual. Drop into a root shell and check that everything is OK, and if it is, exit the root shell & let the system continue with a normal boot. If you've done everything right, your system should come up normally on the new drive.

Step 10 - Add the second drive to your RAID array

Once you're happy that everything is OK and you don't need what's on your old drive, you can repartition it and add it into your new RAID1 array. I waited until my daily backup had run a couple of times & didn't find many changed files, just to be sure. If you're keeping your old drive, you don't need to wait, you can partition the other new drive immediately.

First, run fdisk -l on the running drive so that you can make the partitions on the two drives be the same sizes. Then create your partitions on the new drive, just as you did in step 1 of this guide.

Add the new partitions into your RAID1 array. This assumes that you created three partitions and have the running drive as /dev/sda and the new spare as /dev/sdb:

mdadm /dev/md0 -a /dev/sdb1 mdadm /dev/md1 -a /dev/sdb2 mdadm /dev/md2 -a /dev/sdb3

If you run cat /proc/mdstat, you'll see that the RAID arrays are now sync'ing. The first two small arrays will sync very quickly (a few seconds) but larger arrays can take 1.8TB LVM RAID1 array took about five and a half hours to sync.

This is what it looks like while it's sync'ing:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md2 : active raid1 sdb3[2] sda3[0] 1946684288 blocks [2/1] [U_] [=====>...............] recovery = 27.0% (526898624/1946684288) finish=284.8min speed=83062K/sec md1 : active raid1 sdb2[1] sda2[0] 6297408 blocks [2/2] [UU] md0 : active raid1 sdb1[1] sda1[0] 530048 blocks [2/2] [UU] unused devices:

and this is what it looks like while it's finished:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md2 : active raid1 sdb3[1] sda3[0] 1946684288 blocks [2/2] [UU] md1 : active raid1 sdb2[1] sda2[0] 6297408 blocks [2/2] [UU] md0 : active raid1 sdb1[1] sda1[0] 530048 blocks [2/2] [UU] unused devices:

Finally, install GRUB in the MBR of the second drive:

grub-install /dev/sdb

That's it, you should now have a system with LVM and RAID1 that will boot from either drive. Don't try it until all the RAID arrays have completed sync'ing though, or you may end up with a corrupted RAID array and lose your data.