Gentoo RAID-1, LUKS, LVM2 (obsolete)

This Document Is Deprecated

2009-07-12: This document is now deprecated: it's too old, Gentoo has changed, and (more importantly) I no longer like the idea of running a server fully on LUKS: it makes it impossible to reboot without human assistance. Good for a laptop, bad for a server. My new instructions are here.

This document describes how to set up one particular configuration of encrypted disks on Gentoo. The configuration looks something like this:

          hda                  hdc
     +------------+   	  +------------+
     |   /boot    |<==R==>|  /boot     |
     +------------+   A   +------------+
     |            |   I   |            |
     |    swap    |<==D==>|   swap     |
     |            |       |            |            LVM2
     +------------+   1   +------------+        +------------+
     |/  /  /  /  |       |/  /  /  /  | > L >  |   /        |
     |  /  /  /  /|<=====>|  /  /  /  /| > U >  |            |
     | /  /  /  / |       | /  /  /  / | > K >  +------------+
     :/  /  /  /  :       :/  /  /  /  : > S >  |   /home    |
     |  /  /  /  /|       |  /  /  /  /|        |            |
     +------------+       +------------+        +------------+

That is:

  • Two 250G hard disks:
    • Three partitions on each: /boot, swap, and a LUKS-encrypted one);
    • Each of those partitions tied together in software RAID-1.
  • The encrypted partition, when decrypted, is managed by LVM2

Questions

  • Why LVM2 on top of LUKS? (e.g. as opposed to LUKS on top of other partitions) Because that requires just one password, and because it lets me grow / and /home and whatever else I need.
  • Why RAID-1 on swap? (as opposed to RAID-0 for twice the space) Because if a drive goes bad with swap on it, system behavior will be undefined.
  • Why Gentoo? Because it comes with RAID, LVM2, and LUKS already on the install CD. And because I have two Gentoo systems and like how they work.

How

Boot the Gentoo 2006.1 LiveCD. If you like, you can bring up an xterm (Applications -> Terminal) and start ssh. This lets you log in from the network:

livecd# passwd root
livecd# /etc/init.d/sshd start

Partition

Partition your first drive. Here's my layout; yours may differ:

livecd# fdisk -l /dev/hda

Disk /dev/hda: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1          17      136521   fd  Linux raid autodetect
/dev/hda2              18         280     2112547+  fd  Linux raid autodetect
/dev/hda3             281       30515   242862637+  fd  Linux raid autodetect

Copy the partition table to the other one:

livecd# sfdisk -d /dev/hda | sfdisk /dev/hdc

Start RAID

RAID is part of LiveCD, but you need to create the device nodes and start things up:

livecd# for i in 1 2 3;do mknod /dev/md$i b 9 $i;done
livecd# for i in 1 2 3;do mdadm --create --verbose /dev/md$i --level=1 --raid-devices=2 /dev/hda$i /dev/hdc$i;done

Note that I take pains to match the md numbers with the hd ones. That is absolutely not necessary--Linux doesn't care--but it makes your life much easier when all the numbers are the same.

You may want to wait for the RAID volumes to sync up. That took about 2 hours on my system.

livecd# watch -t -n .5 cat /proc/mdstat

Create Filesystems

When the RAID is done, initialize your /boot and swap partitions:

livecd# mkfs.ext3 -j -O dir_index /dev/md1
livecd# mkswap /dev/md2
livecd# swapon /dev/md2

Initialize LUKS, then LVM, on the big partition:

livecd# cryptsetup luksFormat /dev/md3
WARNING!
========
This will overwrite data on /dev/md3 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: your passphrase
Verify passphrase: your passphrase
Command successful.
# cryptsetup luksOpen /dev/md3 pvcrypt
Enter LUKS passphrase: your passphrase
key slot 0 unlocked.
Command successful.


livecd# pvcreate /dev/mapper/pvcrypt
  Physical volume "/dev/mapper/pvcrypt" successfully created
livecd# vgcreate vgcrypt /dev/mapper/pvcrypt
  Volume group "vgcrypt" successfully created
livecd# lvcreate -n lvroot -L 8G vgcrypt
  /dev/cdrom: open failed: Read-only file system
  Attempt to close device '/dev/cdrom' which is not open.
  Logical volume "lvroot" created
livecd# lvcreate -n lvhome -L 50G vgcrypt
  /dev/cdrom: open failed: Read-only file system
  Attempt to close device '/dev/cdrom' which is not open.
  Logical volume "lvhome" created

livecd# mkfs.ext3 -j -O dir_index /dev/mapper/vgcrypt-lvroot
livecd# mkfs.ext3 -j -O dir_index /dev/mapper/vgcrypt-lvhome

Ignore the /dev/cdrom errors. Feel free to set your own sizes.

Note also that the pv name doesn't seem to matter. Gentoo will call it something else (root) when it boots. It's OK.


Mount Filesystems

The Gentoo install expects them in /mnt/gentoo :

livecd# mount /dev/mapper/vgcrypt-lvroot /mnt/gentoo
livecd# mkdir /mnt/gentoo/boot
livecd# mount /dev/md1 /mnt/gentoo/boot
livecd# mkdir /mnt/gentoo/home
livecd# mount /dev/mapper/vgcrypt-lvhome /mnt/gentoo/home

Install Gentoo

You can't use the fancy menu-driven installer. It will fail in the grub step.

Follow the Gentoo Handbook instructions to install your gentoo. Skip step 4 (Partitioning).

In the grub step, follow these instructions to install on both drives.


genkernel.conf

(NOTE: we're in the chroot now)

You will need to use genkernel. Even if you like building your kernels by hand, you need to use genkernel. That's because of all the initrd magic.

Here are the relevant portions of /etc/genkernel.conf.

LUKS=1
CMD_LVM2=1

The CMD_ in CMD_LVM2 is critical! genkernel-3.4.4 has a bug where simply setting LVM2=1 will be ignored. You need to set CMD_LVM2=1. The bug maybe fixed by the time you read this. Search for CMD_LVM2 in /usr/share/genkernel/gen_determineargs.sh; if there's an 'else LVM2=0' right after it, the bug is still there.

Here are other genkernel.conf options I use:

MENUCONFIG="yes"
CLEAN="no"
MRPROPER="no"
BOOTLOADER="grub"
GENSPLASH=1
GENSPLASH_THEME="emergence"
GENSPLASH_RES="1600x1200"

Now run genkernel all. Don't panic if you don't see the 'Including LUKS support' message in the genkernel output: that's another bug in genkernel.

Be sure to menuconfig! In particular, you will need to include these options in your kernel:

Cryptographic options --->
  SHA256 digest algorithm
  AES cipher algorithms
  AES cipher algorithms (i586)
  (You may need more.  I just enabled them all.)

Device Drivers --->
  Multi-device support (RAID and LVM) --->
    RAID support --->
      RAID-1 (mirroring) mode

Enable static (*), not M. This is critical because genkernel doesn't include the crypto modules in its initrd. So you try to luksOpen on boot and can't.


grub.conf

Your grub config line needs a lot of extra work. Here's mine. I've put the parameters one per line for clarity: in the real world, you need to slop them all onto one huge long line. Important stuff is shown in bold.

title Gentoo 2.6.18-g3
  root (hd0,0)
  kernel /boot/kernel-genkernel-x86-2.6.18-gentoo-r3
         root=/dev/ram0                             boilerplate
         init=/linuxrc                               "  "  " " 
         ramdisk=8192                               "  "  " " 
         video=vesafb-tng:mtrr:3,1600x1200-24@60   for splash
         splash=verbose,theme:emergence             "  "  " "
         CONSOLE=/dev/tty1                          "  "  " "
         dolvm2                                    include LVM2 in initrd
         lvmraid=/dev/md1,/dev/md2,/dev/md3        start these RAIDs
         crypt_root=/dev/md3                       this is what gets luksOpen'ed
         real_root=/dev/mapper/vgcrypt-lvroot      this gets mounted as root, after the luksopen
         udev
  initrd /boot/initramfs-genkernel-x86-2.6.18-gentoo-r3

Reboot

Take out the CD and reboot. Cross your fingers! Grub will load, then your boot will start. The boot gets surprisingly far: there's a lot of crap in that genkernel initrd. But then at some point it asks you for the LUKS passphrase.

If it fails: boot back into the LiveCD. Reassemble the RAIDs:

livecd# for i in 1 2 3;do mknod /dev/md$i b 9 $i;done
livecd# for i in 1 2 3;do mdadm --assemble /dev/md$i /dev/hda$i /dev/hdc$i;done

(Note the simpler --assemble version: that preserves your data.)

Mount them and go exploring. Good luck.


Resources

My efforts would not have been possible without the following web pages:


Created: 2006 November 28