Ubuntu with UEFI iSCSI root on x86_64

If I'm not wrong, the old ubuntu server can be installed directly on iSCSI disk for MBR type of PC. But I like to convert a Core 2 Due MacBook Pro to Ubuntu server, which only has UEFI and can not boot up into MBR.

Note: This is just my observation. Maybe I'm wrong.

Preparation

Ubuntu 20.04 installation USB drive
16GB USB drive for OS installation

Steps

Install OS

Partition the USB drive into the same format as Fedora root on iSCSI which I had done before.

EFI partition, 512MB, vfat, on USB drive
/boot, btrfs, 1GB, on USB drive
/, btrfs, on iSCSI LUN

Note: I separated /boot and /, because I need to move / into iSCSI LUN, and I'm also not sure UEFI can configure iSCSI to detect iSCSI LUN before grub find out /boot partition. I tried UEFI in raspberry pi, which can configure iSCSI, but I didn't see such menu in MacBook Pro.

OS structure

After installed, the system structured as below

UEFI - /boot/efi/EFI

# find /boot/efi -ls
        1      4 drwxr-xr-x   3 root     root         4096 Jan  1  1970 /boot/efi
        4      4 drwxr-xr-x   4 root     root         4096 Nov  9 23:31 /boot/efi/EFI
        7      4 drwxr-xr-x   2 root     root         4096 Nov  9 23:31 /boot/efi/EFI/BOOT
       36    936 -rwxr-xr-x   1 root     root       955656 Nov 10 10:46 /boot/efi/EFI/BOOT/BOOTX64.EFI
       37     84 -rwxr-xr-x   1 root     root        85672 Nov 10 10:46 /boot/efi/EFI/BOOT/fbx64.efi
       38    840 -rwxr-xr-x   1 root     root       856232 Nov 10 10:46 /boot/efi/EFI/BOOT/mmx64.efi
       11      4 drwxr-xr-x   2 root     root         4096 Nov 10 08:56 /boot/efi/EFI/ubuntu
       44      4 -rwxr-xr-x   1 root     root          108 Nov 10 10:46 /boot/efi/EFI/ubuntu/BOOTX64.CSV
       45      4 -rwxr-xr-x   1 root     root          121 Nov 10 10:46 /boot/efi/EFI/ubuntu/grub.cfg
       46   1696 -rwxr-xr-x   1 root     root      1734528 Nov 10 10:46 /boot/efi/EFI/ubuntu/grubx64.efi
       47    840 -rwxr-xr-x   1 root     root       856232 Nov 10 10:46 /boot/efi/EFI/ubuntu/mmx64.efi
       48    936 -rwxr-xr-x   1 root     root       955656 Nov 10 10:46 /boot/efi/EFI/ubuntu/shimx64.efi

In above list, there following two files are important

The file /boot/efi/EFI/ubuntu/BOOTX64.CSV has following content

shimx64.efi,ubuntu,,This is the boot entry for ubuntu

The EFI grub configuration file, /boot/efi/EFI/ubuntu/grub.cfg has following content, which contains uuid of boot partition and location info, and it is named as root hd3,gpt2

search.fs_uuid 812cce04-3b56-4e17-8e38-b325304293f2 root hd3,gpt2
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

Note: Although the USB device location number is changing depending on the sequence of device detection, but the uuid will never be changed. Here, names it as hd3,gpt2, is because boot partition was the gpt partition 2 on 3rd device /dev/sdd2. The device name hd3 doesn't need to be the real device location, but it is only the reference to be used in ubuntu grab configuration later.

Ubuntu boot directory - /boot

This directory includes kernel files and grub configuration file.

-rw------- 1 root root  4755119 Oct 15 17:56 System.map-5.4.0-90-generic
-rw-r--r-- 1 root root   237884 Oct 15 17:56 config-5.4.0-90-generic
drwxr-xr-x 3 root root     4096 Jan  1  1970 efi
drwxr-xr-x 1 root root       82 Nov 10 08:54 grub
lrwxrwxrwx 1 root root       27 Nov  9 23:29 initrd.img -> initrd.img-5.4.0-90-generic
-rw-r--r-- 1 root root 84224544 Nov 10 02:51 initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root       27 Nov  9 23:29 initrd.img.old -> initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root       24 Nov  9 23:29 vmlinuz -> vmlinuz-5.4.0-90-generic
-rw------- 1 root root 11780352 Oct 15 19:36 vmlinuz-5.4.0-90-generic
lrwxrwxrwx 1 root root       24 Nov  9 23:29 vmlinuz.old -> vmlinuz-5.4.0-90-generic

Ubuntu grub - /boot/grub/grub.cfg

This is the grub configuration for ubuntu boot, the importent parts are, ip configuration, iscsi configuration, and turn off screen configuration.

linux /vmlinuz-5.4.0-90-generic root=UUID=<YOUR_DEV_UUID> ro ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30

Note: This file generated using /etc/default/grub, I'm not sure how to change root to label based, and there is option ro, which conflicts with rw that I was given in /etc/default/grub.

Ubuntu grub parameter file - /etc/default/grub

This file is used to build actual /boot/grub/grub.cfg.

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL=console

By default, nothing disappers on screen when booting up, uncomment the GRUB_TERMINAL=console to fix the issue.

Note: I prefer grub menu, it can be used to edit kernel parameter when anything went wrong. Troubleshooting can be easier during kernel upgrade.

Two kinds of commands use this file to update grub configuration

update-grub or update-grub2

Note: update-grub2 is a soft link of update-grub

This is to update /boot/grub/grub.cfg using /etc/default/grub file.

Some people reported issue that update-grub or update-grub2 which used this file to build grub configuration, doesn't update /boot/efi/EFI/ubuntu/grub.cfg file.

dpkg-reconfigure grub-efi-amd64

This command will update both grub configuration file /boot/grub/grub.cfg and EFI grub configuration file /boot/efi/EFI/ubuntu/grub.cfg.

Configure iSCSI service

The service was installed by default in ubuntu server 20.04.

Enable iscsid service

systemctl enable iscsid

Configure /etc/iscsi/initiatorname.iscsi

InitiatorName=<YOUR_INITIATOR_NAME>

Configure /etc/iscsi/iscsid.conf

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>

Start iscsid service

After iscsid configuration changed, restart iscsid service is required.

systemctl restart iscsid

Discover and Login

# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

Identify block device

Use lsblk command to identify device file, it should be something like /dev/sdX.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

Network boot
UEFI iSCSI boot
USB device backup

Partition /dev/sdd1: vfat, 1GB, for /boot filesystem
Partition /dev/sdd2: for root filesystem

Note: The iSCSI LUN appears as /dev/sdd

Format iSCSI LUN

Format /dev/sdd1 as vfat and /dev/sdd2 as btrfs

mkfs.vfat /dev/sdd1
mkfs.btrfs /dev/sdd2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Note: Verifying iscsi module in updating list is important

Update grub

Ubuntu grub parameter file /etc/default/grub as listed in previous section. Beware of iSCSI parameters.

Update both /boot/grub/grub.cfg and /boot/efi/EFI/ubuntu/grub.cfg

dpkg-reconfigure grub-efi-amd64

Test Reboot

This is the first time test reboot, the outcome should be

No hanging issue
New iSCSI disks can be found after reboot without running iscsiadm command manually. Verify using lsblk command.

The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.

Duplicate files

Duplicate files to iSCSI LUN

mount /dev/sdd2 /mnt
mkdir /mnt/boot
mount /dev/sdd1 /mnt/boot
rsync -avhP --exclude /boot/efi --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/efi,mnt}

Change root partition in grub

Use UUID

Identify UUID for new root filesystem

blkid /dev/sdd2

Replace root device definitions as root=UUID=<UUID> in /boot/grub/grub.cfg using block id found

Use LABEL

Assign LABEL to new root filesystem

btrfs fi label /mnt ROOT

Replace root device definitions as root=LABEL=ROOT in /boot/grub/grub.cfg.

Update `/etc/fstab`

Replace root filesystem (/) uuid using the found in previous section (in iSCSI LUN), or if LABEL is assigned, then following line can be used.

LABEL=ROOT / btrfs defaults 0 1

Test Reboot

The reboot is to test root partition switchs to iSCSI LUN. Verify using df command.

/dev/sdc2       15727596 5091296  10317712  34% /
...
/dev/sda2        1048576  111400    819256  12% /boot
/dev/sda1         523248    5356    517892   2% /boot/efi

Now, the root (/) is in different device as /boot and /boot/efi.

Update grub again

Run dpkg-reconfigure grub-efi-amd64 again, then reboot the system. This is to verify all configuration are correct.

Test Reboot

After reboot, the system should have expected setup, which archives

The root (/) partition is in iSCSI LUN
Filesystems are structured according to /boot/efi/EFI/ubuntu/grub.cfg and /etc/fstab.

Clone to smaller USB drive

Due to both EFI and /boot partitions are all small partition, and they are only needed during boot up, a smaller and slower USB drive can be used.

The outcome is also testing the root filesystem fully moved.

Create partition

EFI partition, 512MB, vfat, on USB drive
/boot, btrfs, 1GB, on USB drive

Create filesystem

/dev/sdc1 is EFI partition, and /dev/sdc2 is /boot partition

mkfs.vfat /dev/sdc1
mkfs.btrfs /dev/sdc2

Deplicate files

mount /dev/sdc2 /mnt
mkdir /mnt/efi
mount /dev/sdc1 /mnt/efi
rsync -avhP /boot/ /mnt/

Umount filesystems

umount /mnt/efi
umount /mnt
umount /boot/efi
umount /boot

Update `/etc/fstab`

Edit /etc/fstab and update UUID for both /boot and /boot/efi as below

/dev/disk/by-uuid/812cce04-3b56-4e17-8e38-b325304293f2 /boot btrfs defaults 0 1
/dev/disk/by-uuid/6B77-6F14 /boot/efi vfat defaults 0 1

Mount filesystem

This is also to confirm /etc/fstab is correct.

mount -a

Update grub

Run following command, and verify grub settings, include /boot/grub/grub.cfg, /boot/efi/EFI/ubuntu/grub.cfg.

dpkg-reconfigure grub-efi-amd64

Final test boot

Shutdown system and remove original USB, after that power on device. Make sure everything are expected after system boot up.

Other considerations

Disable iSCSI logout

The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (/) filesystem. Disable iSCSI logout during service stop.

systemctl edit --full open-iscsi.service

Comment out following line

#ExecStop=/lib/open-iscsi/logout-all.sh

Use fix IP address for iSCSI

Following IP configuration can be used to configure fix IP (192.168.1.51) in kernel

ip=192.168.1.51::192.168.1.254:255.255.255.0:fish:enp0s10::192.168.1.250::

If the IP address is different than OS, and they are using same interface, then there will be two IP entries for same interface, for example,

2: enp0s10:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:26:4a:18:82:c6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.51/24 brd 192.168.1.255 scope global enp0s10
       valid_lft forever preferred_lft forever
    inet 192.168.1.9/24 brd 192.168.1.255 scope global secondary dynamic enp0s10
       valid_lft 43172sec preferred_lft 43172sec
    inet6 fe80::226:4aff:fe18:82c6/64 scope link 
       valid_lft forever preferred_lft forever

With two IP address, the dedicated iSCSI IP range can be setup if required. Furthermore, dedicated network interface/LAN can be used for iSCSI network.

Future considerations

Missing iSCSI module

A recent issue I encountered for ubuntu `do-release-upgrade`, is missing `iscsi_tcp.ko` module, and the package `linux-modules-extra` is not in apt list. The apt dependency doesn't include linux-modules-extra, but the package can be downloaded from apt repository, manual installation is required.

Update:

The latest missing package can be installed using following command.

apt install linux-modules-extra-raspi

Backup USB device

Because USB devices are not mirror, backing up is required, and restoration is essential. The first partition in LUN can be used for backup.

Boot partition on iSCSI

If UEFI supports iSCSI, the boot device should able to be in iSCSI LUN as well. The advantage is, boot device also can be part of LUN snapshot for backup. But three stages involve iSCSI connection maybe having issues.

UEFI iSCSI connection
Grub iSCSI connection
OS iSCSI connection

In fact, the OS doesn't need iSCSI connection if no additional iSCSI targets required except the one connected by Grub, because the LUNs connected in Grub are represented as local disks.

Multiboot

If move Boot partition to iSCSI LUN, then left one small configuration in EFI partition, such as UUID for boot device, etc. If setting up multiboot in EFI is possible, then the same USB device can be used for different boot devices.

Grub in Fedora way

In Fedora, the format of kernel parameters are different, and IP address and it's bridge configuration can be done in following way. If can be done in same way as Fedora, the IP address of iSCSI can be fixed, no need to be dhcp, and iSCSI definition can be shorter.

GRUB_CMDLINE_LINUX="netroot=iscsi::@::3260:: rd.iscsi.initiator=YOUR_INITIATOR_NAME rhgb quiet ip=192.168.1.9::192.168.1.254:255.255.255.0::br0:off nameserver=192.168.1.250 ifname=enp0s10:00:26:4a:18:82:c6 bridge=br0:enp0s10"

Prior to start

I had tried a few times to install ubuntu server 20.04 on MacBook Pro on iSCSI, but failed. I also tried Fedora 34, and it was successfully installed root partition on iSCSI LUN.

For Fedora 34, there are 3 partitions,

EFI partition, 512MB, vfat, on USB drive
/boot, btrfs, 2GB, on USB drive
/, btrfs, on iSCSI LUN

It is using GRUB2.

After observation, I think I'm able to do the same for ubuntu.

In fact, I prefer ubuntu, because do-release-upgrade is doing well for ubuntu. For Fedora, I had done version upgrade many years ago too, but very manual and it is not official supported. Meaning one day, the upgrade method can never work again.

Troubleshooting

grub configuration error

Such as iSCSI configuration was given wrongly, due to no grub menu, the USB drive needs to be connected to another linux system to modify. An ubuntu VM is handy in this case.

References

Convert Raspberry Pi Ubuntu to iSCSI btrfs root
How to Configure the GRUB2 Boot Loader’s Settings
The kernel’s command-line parameters
update-grub does not update /boot/efi/EFI/ubuntu/grub.cfg

Day: November 10, 2021

Ubuntu with UEFI iSCSI root on x86_64

Preparation

Steps

Install OS

OS structure

Configure iSCSI service

Identify block device

Partitioning

Format iSCSI LUN

Update initramfs

Update grub

Test Reboot

Duplicate files

Change root partition in grub

Use UUID

Use LABEL

Update /etc/fstab

Test Reboot

Update grub again

Test Reboot

Clone to smaller USB drive

Create partition

Create filesystem

Deplicate files

Umount filesystems

Update /etc/fstab

Mount filesystem

Update grub

Final test boot

Other considerations

Disable iSCSI logout

Use fix IP address for iSCSI

Future considerations

Missing iSCSI module

Backup USB device

Boot partition on iSCSI

Multiboot

Grub in Fedora way

Prior to start

Troubleshooting

grub configuration error

References

Update `/etc/fstab`

Update `/etc/fstab`