A few weeks ago I thought about upgrading my home server to use LVM and RAID1, but before I had time to get it done, my server suddenly refused to boot (I'd shut it down to add some RAM). Instead it would just reset after POST. I discovered that it would boot from a different SATA hard drive, but only if I disconnected its usual boot drive. It would also boot from an IDE CD drive, no matter which SATA drives were connected, and the data on all drives was intact and readable.
I'd just bought a couple of 2TB drives to increase the storage on my MythTV box, so to get the server back up as quickly as possible, I created a couple of partitions on one of the drives (one swap, one data), copied everything from the old drive to the new one, installed GRUB and then booted the system again. I also installed the second drive, ready for the LVM + RAID1 conversion.
The server was originally installed with Ubuntu Dapper and has undergone several dist-upgrades since then. When I first installed it, I had only a single drive so RAID wasn't an option, and I'd never used LVM and didn't want to experiment with it at the time, so I opted for a number of fixed size partitions.
/dev/sda and
doesn't use LVM or RAID.
/dev/sdb to be your
new LVM + RAID1 drive and you plan to use two
identically sized drives in your RAID1 array.
Before you start, make sure you backup your system, or at least, anything that's important. If you make a mistake, you could destroy all of your data. You probably won't, but it could happen. You should always have an up-to-date backup anyway, but even if you do, check it and make sure it's OK before you go any further.
/dev/sda was my existing 2TB drive and
/dev/sdb was the unused 2TB drive that I was
going to use to setup LVM + RAID.
The first step is to decide how you want to partition your new drive. You can't have /boot on an LVM volume, so it needs to be on a separate partition (I have heard that GRUB2 may support booting from LVM volumes, but GRUB legacy and LILO don't). You can then choose whether to have swap on an LVM volume or on a separate partition. I chose the latter. The rest of the drive can be one partition that will be managed by LVM.
So, I created three primary partitions. /dev/sdb1
(/boot) is 256MB, /dev/sdb2 (swap) is
6GB and /dev/sdb3 (lvm) is the rest of the
drive. The partition type of all three is 0xfd
(Linux raid autodetect).
Next you need to create degraded RAID1 arrays using the new drive:
If you chose to put swap on your LVM partition then you'll only have two partitions and hence only two RAID1 arrays.
Format the non-LVM partitions:
You now need to decide how many partitions you want to
setup. I chose to have separate volumes for /,
/home, /tmp, /usr,
/usr/local and /var, and a large
/storage volume for all my data, because that
works for me, but if you just want one big partition then
that'll work too. Just setup the logical volumes according
to your own preferred paritioning scheme.
You'll first need to initialise the LVM partition and create a volume group. You can call your volume group whatever you want, e.g. the hostname, but I chose simply to use "vg0":
Now you're ready to create the various logical volumes you want:
You'll now be able to access all of these logical volume via
their device names
/dev/mapper/<vg_name>-<lv_name>.
I chose to use ext3 for all of my volumes, so formatting required only a simple loop:
Now you need to mount the new partitions and logical
volumes in preparation for copying over your data. I
chose to mount them all under /media/lvm:
There are a few different ways to copy the data, but I chose
to use rsync because I'm familiar with it.
There are a number of exclusions; each is explained below:
lost+found is a directory created at the
root of every ext2/ext3 partition, and there's no need
to copy these directories across, especially if the
new drive doesn't have the same partition layout as
the original. This directory is where fsck puts files
or directories it recovers from a corrupted partition
and is normally empty.
/media/lvm is where I've mounted the new
drive and I don't want to recursively copy itself :-)
These exclusions in /dev are files or directories
that are created during boot, or are tmpfs mounts (RAM discs)
added during system initialisation.
/lib/init/rw/ is another tmpfs mount, and only the
mount point is needed on your new drive.
/lib/modules/*/volatile are also tmpfs mounts,
and you must not have the file
.mounted present in these directories on your
disc. This file is used during system initialisation to
decide whether or not to mount this filesystem. At that
point, the root filesystem is still mounted read-only so
you'll get errors when the system tries to load files into
this directory.
/proc and /sys are virtual
filesystems and only the mount points are needed.
/home/*/.gvfs are virtual filesystems and are
only readable by the appropriate users, and not even root
can access them. Excluding them from the copy avoids the
errors that result from trying to copy them.
/var/lock and /var/run are two
more tmpfs filesystems and only the mount points are needed.
Later, you'll need to do another final sync and this time you'll need to add a few more exlusions to avoid overwriting files which you need to modify to get the new drive to boot.
Create an mdadm config file for the new system:
Check it once it's been created to be sure that it contains
the correct RAID arrays. On my system, the scan gave me two
entries for /dev/md2 (but with the UUID of
/dev/md1 in one of them). That could have been
due to me experimenting with md devices because I can't
reproduce it now.
To create a new initramfs, you need two files from the
virtual filesystem /proc. You'll be chrooting
into the new drive so you'll need to copy these over
temporarily:
Now edit /media/lvm/proc/cmdline and replace
the root drive (it'll either be a partition name or a UUID)
with /dev/mapper/vg0-root (change as
appropriate to match your volume group and logical volume
naming scheme):
Edit /media/lvm/etc/fstab so that the new LVM
volumes are mounted in all the right places. Here's what
mine now contains (note the non-LVM /boot and
swap:
Edit your new GRUB menu in
/media/lvm/boot/grub/menu.lst. I'm using GRUB
legacy; if you're using GRUB2 you'll have to figure this bit
out for yourself.
Comments start with "#", except that those after the
"BEGIN AUTOMAGIC KERNELS LIST"
line are special. In this block, lines starting with "##"
are comments, but lines starting with just a single "#" are
used by update-grub to add new boot stanzas
when installing new kernels. You will need to change some
of these lines.
The first of thse is the name of the root device passed to
the kernel at boot time. This is one of those special comments,
and contains the keyword kopt. Here's the relevant
part of mine:
Change the kopt line to:
Next you need to change grub's root device. This is
the partition which contains /boot and is
defined in this part of menu.lst:
I changed mine to use the UUID of the boot partiton
(/dev/md0). blkid /dev/md0
will give you the UUID. Note that you don't include
UUID= in this line:
Next you need to change all of the existing boot stanzas
in the new menu.lst. Here's mine before I
started:
And here it is again after the changes:
The changes are:
root line with uuid
to match the groot change made earlier.
root= on the kernel
command line to match the new root device name.
/boot.
You'll need to make similar changes to all boot stanzas in the file.
You may also want to increase the boot timeout to give you
time to select single user mode the first time you boot from
the new drive. This is the timeout option and
comes before the "BEGIN AUTOMAGIC KERNELS LIST"
line. I set mine to 10 seconds. That's the default but I'd
reduced mine, so I bumped it up again.
Now you're ready to create a new initramfs (previously known as an initrd):
If you have multiple kernels installed, you'll need
to run update-initramfs for each kernel.
Put the kernel version number in place of `uname
-r` in the above example. Once you've finished,
delete the two files you copied into the new
Now you can install GRUB in the MBR of the new drive. Exit from the chroot shell and run:
The previous version of this page had
grub-install being run from within the chroot
shell and without the --root-directory option,
but I discovered when replacing the drive in an older
machine that grub-install uses
/etc/mtab to find the partition containing
/boot when verifying that the stage 1, 1.5 and
2 files are copied correctly. If grub-install
gives you an error message like The file
/boot/grub/stage2 not read correctly, you
know it's comparing the files on the old drive with those it
tried to install on the new one, either because you tried to
install GRUB from within the chroot, or you didn't pass the
right command line options to grub-install.
If you don't care too much about minimising downtime, you can skip this step and just do one resync in single user mode. This intermediate sync just means that almost all of the data changes are sync'ed before dropping into single user mode, making the final sync quicker and hence minimising the time the system is offline.
Do another data sync to ensure that the old and new drives contain (almost) the same data:
Note that this time I've added --delete and
a few more exclusions. The former is because we want to
make sure that any files deleted on the old drive while
we've been working on the migration are deleted from the
new drive too.
/boot is now excluded so that we don't
overwrite the new GRUB menu and the new initramfs. I also
found that some of the other files in /boot
were changed by grub-install, and I didn't want
to overwrite them. These extra changes might have been
because the version of grub on the system is more recent
that the one that was first installed a few years ago.
Unless you've installed or removed a kernel since you
started this conversion, it's highly unlikely that any files
in this directory have been changed on the original drive,
so it's easiest to simply exclude this directory from the
sync.
/etc/fstab and etc/mdadm/mdadm.conf
are files that you've changed on the new drive and you do
not want to lose those changes.
/var/lib/initramfs-tools/ contains one directory
for each installed kernel, and each directory contains a file
which has the SHA1 checksum of that kernel's initramfs. You've
rebuilt the initramfs for each kernel on the new drive, so these
files will have been changed and you want to keep those changes.
Some of the earlier exclusions have had the trailing
/* removed. This isn't entirely necessary but
doesn't hurt. The /* was needed on the first
copy so that the directories were created without their
contents, but now that the directories are on the new
drive they can be excluded entirely from the resync.
Change to single user mode by running telinit 1
as root, then you'll need to do a final data sync before
rebooting using the same rsync command as the
sync in the previous section.
Once this sync has been done you can shutdown and reboot.
You can choose to swap the old and new drives over so that
the new drive becomes /dev/sda and the old one
/dev/sdb, or you can change the boot order in
the BIOS to make sure you boot from the new drive. I did
the former. If you're replacing your old drive with new
drives for your RAID array, then you can remove the old
drive and make sure that both of your new drives are
installed.
When you reboot, I suggest you select single user mode first, just in case something goes wrong. Watch the boot messages carefully, looking for anything unusual. Drop into a root shell and check that everything is OK, and if it is, exit the root shell & let the system continue with a normal boot. If you've done everything right, your system should come up normally on the new drive.
Once you're happy that everything is OK and you don't need what's on your old drive, you can repartition it and add it into your new RAID1 array. I waited until my daily backup had run a couple of times & didn't find many changed files, just to be sure. If you're keeping your old drive, you don't need to wait, you can partition the other new drive immediately.
First, run fdisk -l on the running drive so
that you can make the partitions on the two drives be the
same sizes. Then create your partitions on the new drive,
just as you did in step 1 of this guide.
Add the new partitions into your RAID1 array. This assumes
that you created three partitions and have the running drive
as /dev/sda and the new spare as
/dev/sdb:
If you run cat /proc/mdstat, you'll see that
the RAID arrays are now sync'ing. The first two small
arrays will sync very quickly (a few seconds) but larger
arrays can take hours.my 1.8TB LVM RAID1 array took about
five and a half hours to sync.
This is what it looks like while it's sync'ing:
and this is what it looks like while it's finished:
Finally, install GRUB in the MBR of the second drive:
That's it, you should now have a system with LVM and RAID1 that will boot from either drive. Don't try it until all the RAID arrays have completed sync'ing though, or you may end up with a corrupted RAID array and lose your data.