A bit of an odd topic after a period of silence, but it’s something that annoyed me greatly recently and so I wanted to spare others that experience. Meanwhile, what has been keeping the updates? framebunker and StaticSky have! In case you missed it, I started (slowly) brushing off and open-sourcing some of our work there (yes, I’ll do what I can to move through my “to open source” list faster): https://github.com/framebunker/Utilities

Update January 2020: Found & updated the link to the blog post which mentioned the appropriate sectors to clear on SSDs for md. Also one of the people who assisted in the Keybase team, David Hauken, wrote a post on this topic.

Challenge

So anyway, I set up a new computer at home - the age of my 2013 laptop showing. On that I wanted to run Ubuntu off a full-disk encrypted drive. However a core point of the build is speed and full-disk encryption (via LUKS) does not necessarily go hand in hand with that.

To offset that, I decided to go for a configuration with two PCIe4 NVMe sticks running in RAID0 (well cooled). Simple enough in principle, but then you hit the community coverage on the topic of “RAID0 is dangerous! Remember to back up your things and don’t pour water on your hardware”. It would appear that an Ubuntu (or event Linux) configuration like this is not super common.

Luckily there were enough bits and pieces out there to, in combination with invaluable assistance from the people of the sysadmins_unite Keybase team, string together a solution.

Process

If, like me, you are migrating an existing install to this setup then first make a backup of that install. I just used rsync - making sure that all file flags etc. were left intact in the copy. In creating the RAID0 across the two NVMe sticks any data already present on either will of-course be lost.

Next, burn the Ubuntu installer onto a bootable medium. This approach probably work on a ton of other configurations, but given that I have only used it with Ubuntu 18.04, I am going to describe them in that known-to-work context.

Boot into your installer and fire up a terminal. In these steps it is assumed that the two NVMe sticks mount at /dev/nvme0n1 and /dev/nvme1n1. If that is not the case in your setup, replace in the correct name where used. On my setup I was able to verify the mount points both in this installer terminal and in the Asus BIOS.

The first step is to install the mdadm tool - for creation and management of md RAID configurations. Note that critical to this particular usecase is the clearing of superblocks and overwriting of first one gigabyte of data. Most guides I came across would instruct clearing data at the end of the target drives, but one (I unfortunately lost the link - sorry about that never mind - found it in an old window of research tabs) pointed me to the first gigabyte. As I understand it, this is specific to how md operates on SSD or at least NVMe.

Configuring your RAID:

sudo apt-get install mdadm
sudo mdadm --zero-superblock /dev/nvme0n1
sudo mdadm --zero-superblock /dev/nvme1n1
sudo dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1000
sudo dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=1000
sudo mdadm --create --verbose --level=0 --raid-devices=2 /dev/md0 /dev/nvme0n1 /dev/nvme1n1

With your raid up and running in your installer session, I really hope you did not forget to copy data you really wanted off those sticks. Anyway, the next step is to run the Ubuntu installer, opting to do “something else” when typical vs. not-quite-typical install configuration is chosen. This takes you to a screen where you get to specify which drives and partitions are get used for what in your fancy new install. Necessary config:

• Format your new /dev/md0 drive. For me that was formatting to LUKS and then adding an ext4 partition to that. You could also format directly to ext4 if you do not want the encryption, but in that case you might want to do that outside of the installer - for optimum performance. See the later section for specifics on that.
• Specify that the installer use your final ext4 partition as root /.
• Use a different drive and partition for EFI and /boot. This is critical as your bios will not know how to assemble the md RAID and boot from it - you need /boot to handle that. There were some very creative suggestions in sysadmins_unite on where one might put /boot, but I simply have a SATA drive in my rig used for more rough work and first tier backup. I just set up /boot and EFI partitions there.

That’s it - configure the rest of the installer as you like and run it to completion. When it does complete and prompt you to restart, don’t do that. The RAID still only really exists within this installer session so if you reboot now, you might as well start over.

The Ubuntu installer will have your new install partition mounted at /target. You need to make some edits to it in order to have it properly mount and run off your RAID from boot. If you closed the terminal from before, open up a new one and run this:

sudo mount /dev/[partition used for /boot] /target/boot
sudo mount --bind /dev /target/dev
sudo mount --bind /sys /target/sys
sudo mount --bind /proc /target/proc
sudo cp /etc/resolv.conf /target/etc/
sudo gedit /target/etc/default/grub

In gedit you need to update the value of GRUB_CMDLINE_LINUX to "domdadm", save, and close the editor.

Finally you need to add mdadm to your new install, which will also apply your grub change along with its own to the boot setup, after which we are good to reboot and test the install:

sudo chroot /target
apt-get install mdadm
exit
sudo reboot now

If you just wanted a fresh install, then this is the “profit!” step. Otherwise, restore your previous install onto your new root partition and update it to support your new boot setup by rebooting into the installer again and running this in a terminal:

sudo apt-get install mdadm
sudo mdadm -A -s
sudo mkdir /target
sudo mount /dev/md0 /target
sudo mount --bind /dev /target/dev
sudo mount --bind /sys /target/sys
sudo mount --bind /proc /target/proc
sudo cp /etc/resolv.conf /target/etc/
sudo chroot /target
apt-get install mdadm
exit
sudo reboot now

And that is your “profit!” step.

ext4

If you just want to run ext4 as fast as possible directly on your RAID drive, you might want to manually work out the configuration of that partition to optimally work with the underlying RAID configuration. Key here is specifying the correct stride and stripe-width:

• Chunk = 512 (output by or set via mdadm)
• Block = 4 (ext4 dictated)
• Devices = 2
• Stride = 512 / 4 = 128
• Stripe width = 2 * 128 = 256
• Resulting in the formatting command sudo mkfs.ext4 -v -L nvmeRAID -m 1 -b 4096 -E stride=128,stripe-width=256 /dev/md0.

Wrapup

I hope this works out for you. It definitely took me more research, trial, and error than I would have liked to arrive at a functional approach. At this point I have been running the resulting setup for a month or two without the hint of an issue.