Gentoo RAID-1, LUKS, LVM2

June 2009: This is an update of my 2006 article.

This document describes how to set up one particular configuration of encrypted disks on Gentoo. The configuration looks something like this:

          sda                  sdb
     +------------+       +------------+
     |   /boot    |<= R =>|   /boot    |
     +------------+       +------------+
     |    swap    |<= A =>|    swap    |
     +------------+       +------------+
     |   (LVM)    |<= I =>|   (LVM)    |
     |+----------+|       |+----------+|
     ||   root   ||<= D =>||   root   ||
     |+----------+|       |+----------+|        +------------+
     ||/ / / / / ||       ||/ / / / / || > L >  |   /home    |
     || / / / / /||       || / / / / /|| > U >  |            |
     ||/ / / / / ||<= 1 =>||/ / / / / || > K >  |            |
     || / / / / /||       || / / / / /|| > S >  |            |
     ||/ / / / / ||       ||/ / / / / || >   >  |            |
     |+----------+|       |+----------+|        +------------+
     +------------+       +------------+

That is:

  • Two 500G hard disks:
    • Three partitions on each: /boot, swap, and one for data. The data one is subpartitioned, managed by LVM
    • Each of those partitions tied together in software RAID-1.
  • The root partition is cleartext, but /home/esm (my home directory) is encrypted using LUKS
  • .

Actually I have three 500G hard disks, but one is a hot spare.

Questions

  • Why only encrypt my home directory? (as opposed to all of /root): because this system is both a workstation and a server. For 3 years I ran LVM over LUKS, fully encrypted root and everything. About half the time when I was out of town, the power would die for longer than my UPSes could handle. Rebooting would then require the assistance of a neighbor. Now at least critical services (web, mail, ssh) will come up.
  • Why RAID-1 on swap? (as opposed to RAID-0 for twice the space) Because if a drive goes bad with swap on it, on RAID-0, system behavior will be undefined. With RAID-1, no big deal.
  • Why Gentoo? Because it comes with RAID, LVM2, and LUKS already on the install CD. And because I've run Gentoo for over 5 years and am really, really happy with it.

How

Boot the Gentoo Minimal Install CD and start ssh:

livecd# passwd root
New UNIX password: blahblah
Retype new UNIX password: blahblah
passwd: password updated successfully
livecd# /etc/init.d/sshd start

Partition

Partition your first drive. Here's my layout; yours may differ:

livecd# fdisk -l /dev/sda

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xf8a5a02d

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          14      112423+  fd  Linux raid autodetect
/dev/sda2              15        2104    16787925   fd  Linux raid autodetect
/dev/sda3            2105       60801   471483652+  fd  Linux raid autodetect

Make sure you set the type of each to FD.

Copy the partition table to the other ones:

livecd# sfdisk -d /dev/sda | sfdisk /dev/sdb
livecd# sfdisk -d /dev/sda | sfdisk /dev/sdc

(If you don't have a hot-spare disk, ignore the sdc step)


Start RAID

livecd# for i in 1 2 3;do \
    mdadm --create --verbose /dev/md$i \
    --level=1 \
    --raid-devices=2  /dev/sda$i /dev/sdb$i \
    --spare-devices=1 /dev/sdc$i; \
done
mdadm: size set to 112320K
mdadm: array /dev/md1 started.
mdadm: size set to 16787840K
mdadm: array /dev/md2 started.
mdadm: size set to 471483584K
mdadm: array /dev/md3 started.

Note that I take pains to match the md numbers with the sd ones. That is absolutely not necessary--Linux doesn't care--but it makes your life much easier when all the numbers are the same.

You don't need to wait for the RAID volumes to sync up. If you want to, you can while away the 2 hours via:

livecd# watch -t -n .5 cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdc3[2](S) sdb3[1] sda3[0]
      471483584 blocks [2/2] [UU]
      [>....................]  resync =  0.5% (2804480/471483584) finish=89.1min speed=87640K/sec

md2 : active raid1 sdc2[2](S) sdb2[1] sda2[0]
      16787840 blocks [2/2] [UU]
        resync=DELAYED

md1 : active raid1 sdc1[2](S) sdb1[1] sda1[0]
      112320 blocks [2/2] [UU]

unused devices: 

Create Filesystems

Initialize your /boot and swap partitions:

livecd# mkfs.ext3 -j -O dir_index /dev/md1
mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
28112 inodes, 112320 blocks
5616 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
14 block groups
8192 blocks per group, 8192 fragments per group
2008 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 21 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

livecd# mkswap /dev/md2
Setting up swapspace version 1, size = 16787836 KiB
no label, UUID=ef24d4dc-da76-49f3-9313-1e62d0dcd4ab
livecd# swapon /dev/md2

Initialize LVM on the big partition:

livecd ~ # pvcreate /dev/md3
  Physical volume "/dev/md3" successfully created

livecd ~ # vgcreate vg0 /dev/md3
  Volume group "vg0" successfully created

(vg0 can be any name you like)

Create some actual mountable partitions:

livecd ~ # lvcreate --name root --size 10G vg0
  Logical volume "root" created
livecd ~ # lvcreate --name esm --size 100G vg0
  Logical volume "esm" created
livecd ~ # lvcreate --name work --size 100G vg0
  Logical volume "work" created

Create root filesystem. I chose JFS because it seems reliable.

livecd ~ # mkfs.jfs /dev/vg0/root
mkfs.jfs version 1.1.13, 17-Jul-2008
Warning!  All data on device /dev/vg0/root will be lost!

Continue? (Y/N) y
   \

Format completed successfully.

10485760 kilobytes total disk space.

Initialize encrypted filesystems, e.g. your home directory

livecd ~ # cryptsetup luksFormat /dev/vg0/esm

WARNING!
========
This will overwrite data on /dev/vg0/esm irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: your passphrase
Verify passphrase: your passphrase
Command successful.

FIXME: I've stalled out, and hope to finish this later...

Mount, and install Gentoo.

livecd ~ # mount /dev/vg0/root /mnt/gentoo
livecd ~ # mkdir /mnt/gentoo/boot
livecd ~ # mount /dev/md1 /mnt/gentoo/boot

Proceed with the Gentoo installation.

FIXME FIXME FIXME: LUKS the subpartitions


Mount Filesystems

The Gentoo install expects them in /mnt/gentoo :

livecd# mount /dev/mapper/vgcrypt-lvroot /mnt/gentoo
livecd# mkdir /mnt/gentoo/boot
livecd# mount /dev/md1 /mnt/gentoo/boot
livecd# mkdir /mnt/gentoo/home
livecd# mount /dev/mapper/vgcrypt-lvhome /mnt/gentoo/home

Install Gentoo

Follow the Gentoo Handbook instructions to install your gentoo. Skip step 4 (Partitioning).

In the grub step, follow these instructions to install on both drives.

grub> find /boot/grub/stage1
 (hd0,0)
 (hd1,0)

grub> device (hd0) /dev/sda

grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  20 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+20 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
Done.

grub> device (hd0) /dev/sdb

grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  20 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+20 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
Done.

genkernel.conf

(NOTE: we're in the chroot now)

You will need to use genkernel. Even if you like building your kernels by hand, you need to use genkernel. That's because of all the initrd magic.

Run genkernel --lvm all. FIXME


grub.conf

Your grub config line needs a lot of extra work. Here's mine. I've put the parameters one per line for clarity: in the real world, you need to slop them all onto one huge long line. Important stuff is shown in bold.

title Gentoo 2.6.28-r9
  root (hd0,0)
  kernel /boot/kernel-genkernel-x86_64-2.6.28-hardened-r9
         root=/dev/ram0                             boilerplate
         init=/linuxrc                               "  "  " " 
         ramdisk=8192                                "  "  " " 
         dolvm                                      LVM required before booting
         lvmraid=/dev/md3                           start this RAID before booting
         real_root=/dev/vg0/root                    this is what gets mounted as root, after LVM+RAID
  initrd /boot/initramfs-genkernel-x86_64-2.6.28-hardened-r9

Reboot

Take out the CD and reboot. You should now boot into your new system. If it fails: boot back into the LiveCD. Reassemble the RAIDs:

livecd# for i in 1 2 3;do mknod /dev/md$i b 9 $i;done
livecd# for i in 1 2 3;do mdadm --assemble /dev/md$i /dev/sda$i /dev/sdb$i

(Note the simpler --assemble version, instead of --create: that preserves your data.)

Mount them and go exploring. Good luck.


Resources

My efforts would not have been possible without the following web pages:


Created: 2007 July 7