Ubuntu with UEFI iSCSI root on x86_64
If I'm not wrong, the old ubuntu server can be installed directly on iSCSI disk for MBR type of PC. But I like to convert a Core 2 Due MacBook Pro to Ubuntu server, which only has UEFI and can not boot up into MBR.
Note: This is just my observation. Maybe I'm wrong.
Preparation
- Ubuntu 20.04 installation USB drive
- 16GB USB drive for OS installation
Steps
Install OS
Partition the USB drive into the same format as Fedora root on iSCSI which I had done before.
- EFI partition, 512MB, vfat, on USB drive
- /boot, btrfs, 1GB, on USB drive
- /, btrfs, on iSCSI LUN
Note: I separated /boot and /, because I need to move / into iSCSI LUN, and I'm also not sure UEFI can configure iSCSI to detect iSCSI LUN before grub find out /boot partition. I tried UEFI in raspberry pi, which can configure iSCSI, but I didn't see such menu in MacBook Pro.
OS structure
After installed, the system structured as below
# find /boot/efi -ls
1 4 drwxr-xr-x 3 root root 4096 Jan 1 1970 /boot/efi
4 4 drwxr-xr-x 4 root root 4096 Nov 9 23:31 /boot/efi/EFI
7 4 drwxr-xr-x 2 root root 4096 Nov 9 23:31 /boot/efi/EFI/BOOT
36 936 -rwxr-xr-x 1 root root 955656 Nov 10 10:46 /boot/efi/EFI/BOOT/BOOTX64.EFI
37 84 -rwxr-xr-x 1 root root 85672 Nov 10 10:46 /boot/efi/EFI/BOOT/fbx64.efi
38 840 -rwxr-xr-x 1 root root 856232 Nov 10 10:46 /boot/efi/EFI/BOOT/mmx64.efi
11 4 drwxr-xr-x 2 root root 4096 Nov 10 08:56 /boot/efi/EFI/ubuntu
44 4 -rwxr-xr-x 1 root root 108 Nov 10 10:46 /boot/efi/EFI/ubuntu/BOOTX64.CSV
45 4 -rwxr-xr-x 1 root root 121 Nov 10 10:46 /boot/efi/EFI/ubuntu/grub.cfg
46 1696 -rwxr-xr-x 1 root root 1734528 Nov 10 10:46 /boot/efi/EFI/ubuntu/grubx64.efi
47 840 -rwxr-xr-x 1 root root 856232 Nov 10 10:46 /boot/efi/EFI/ubuntu/mmx64.efi
48 936 -rwxr-xr-x 1 root root 955656 Nov 10 10:46 /boot/efi/EFI/ubuntu/shimx64.efi
In above list, there following two files are important
The file /boot/efi/EFI/ubuntu/BOOTX64.CSV has following content
shimx64.efi,ubuntu,,This is the boot entry for ubuntu
The EFI grub configuration file, /boot/efi/EFI/ubuntu/grub.cfg has following content, which contains uuid of boot partition and location info, and it is named as root hd3,gpt2
search.fs_uuid 812cce04-3b56-4e17-8e38-b325304293f2 root hd3,gpt2
set prefix=($root)'/grub'
configfile $prefix/grub.cfg
Note: Although the USB device location number is changing depending on the sequence of device detection, but the uuid will never be changed. Here, names it as hd3,gpt2, is because boot partition was the gpt partition 2 on 3rd device /dev/sdd2. The device name hd3 doesn't need to be the real device location, but it is only the reference to be used in ubuntu grab configuration later.
- Ubuntu boot directory -
/boot
This directory includes kernel files and grub configuration file.
-rw------- 1 root root 4755119 Oct 15 17:56 System.map-5.4.0-90-generic
-rw-r--r-- 1 root root 237884 Oct 15 17:56 config-5.4.0-90-generic
drwxr-xr-x 3 root root 4096 Jan 1 1970 efi
drwxr-xr-x 1 root root 82 Nov 10 08:54 grub
lrwxrwxrwx 1 root root 27 Nov 9 23:29 initrd.img -> initrd.img-5.4.0-90-generic
-rw-r--r-- 1 root root 84224544 Nov 10 02:51 initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root 27 Nov 9 23:29 initrd.img.old -> initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root 24 Nov 9 23:29 vmlinuz -> vmlinuz-5.4.0-90-generic
-rw------- 1 root root 11780352 Oct 15 19:36 vmlinuz-5.4.0-90-generic
lrwxrwxrwx 1 root root 24 Nov 9 23:29 vmlinuz.old -> vmlinuz-5.4.0-90-generic
- Ubuntu grub -
/boot/grub/grub.cfg
This is the grub configuration for ubuntu boot, the importent parts are, ip configuration, iscsi configuration, and turn off screen configuration.
linux /vmlinuz-5.4.0-90-generic root=UUID=<YOUR_DEV_UUID> ro ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30
Note: This file generated using /etc/default/grub, I'm not sure how to change root to label based, and there is option ro, which conflicts with rw that I was given in /etc/default/grub.
- Ubuntu grub parameter file -
/etc/default/grub
This file is used to build actual /boot/grub/grub.cfg.
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL=console
By default, nothing disappers on screen when booting up, uncomment the GRUB_TERMINAL=console to fix the issue.
Note: I prefer grub menu, it can be used to edit kernel parameter when anything went wrong. Troubleshooting can be easier during kernel upgrade.
Two kinds of commands use this file to update grub configuration
update-grub or update-grub2
Note: update-grub2 is a soft link of update-grub
This is to update /boot/grub/grub.cfg using /etc/default/grub file.
Some people reported issue that update-grub or update-grub2 which used this file to build grub configuration, doesn't update /boot/efi/EFI/ubuntu/grub.cfg file.
dpkg-reconfigure grub-efi-amd64
This command will update both grub configuration file /boot/grub/grub.cfg and EFI grub configuration file /boot/efi/EFI/ubuntu/grub.cfg.
Configure iSCSI service
The service was installed by default in ubuntu server 20.04.
systemctl enable iscsid
- Configure
/etc/iscsi/initiatorname.iscsi
InitiatorName=<YOUR_INITIATOR_NAME>
- Configure
/etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
After iscsid configuration changed, restart iscsid service is required.
systemctl restart iscsid
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login
Note: If can not login, restart iscsid and try again.
Identify block device
Use lsblk command to identify device file, it should be something like /dev/sdX.
Partitioning
Creating two partitions using fdisk, the first partition is to prepare following for future used, such as
- Network boot
- UEFI iSCSI boot
- USB device backup
Partition /dev/sdd1: vfat, 1GB, for /boot filesystem
Partition /dev/sdd2: for root filesystem
Note: The iSCSI LUN appears as /dev/sdd
Format iSCSI LUN
Format /dev/sdd1 as vfat and /dev/sdd2 as btrfs
mkfs.vfat /dev/sdd1
mkfs.btrfs /dev/sdd2
Update initramfs
This is to enable ubuntu load iscsi driver during boot
touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c
Note: Verifying iscsi module in updating list is important
Update grub
Ubuntu grub parameter file /etc/default/grub as listed in previous section. Beware of iSCSI parameters.
Update both /boot/grub/grub.cfg and /boot/efi/EFI/ubuntu/grub.cfg
dpkg-reconfigure grub-efi-amd64
Test Reboot
This is the first time test reboot, the outcome should be
- No hanging issue
- New iSCSI disks can be found after reboot without running
iscsiadm command manually. Verify using lsblk command.
The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.
Duplicate files
Duplicate files to iSCSI LUN
mount /dev/sdd2 /mnt
mkdir /mnt/boot
mount /dev/sdd1 /mnt/boot
rsync -avhP --exclude /boot/efi --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/efi,mnt}
Change root partition in grub
Use UUID
Identify UUID for new root filesystem
blkid /dev/sdd2
Replace root device definitions as root=UUID=<UUID> in /boot/grub/grub.cfg using block id found
Use LABEL
Assign LABEL to new root filesystem
btrfs fi label /mnt ROOT
Replace root device definitions as root=LABEL=ROOT in /boot/grub/grub.cfg.
Update /etc/fstab
Replace root filesystem (/) uuid using the found in previous section (in iSCSI LUN), or if LABEL is assigned, then following line can be used.
LABEL=ROOT / btrfs defaults 0 1
Test Reboot
The reboot is to test root partition switchs to iSCSI LUN. Verify using df command.
/dev/sdc2 15727596 5091296 10317712 34% /
...
/dev/sda2 1048576 111400 819256 12% /boot
/dev/sda1 523248 5356 517892 2% /boot/efi
Now, the root (/) is in different device as /boot and /boot/efi.
Update grub again
Run dpkg-reconfigure grub-efi-amd64 again, then reboot the system. This is to verify all configuration are correct.
Test Reboot
After reboot, the system should have expected setup, which archives
- The root (/) partition is in iSCSI LUN
- Filesystems are structured according to
/boot/efi/EFI/ubuntu/grub.cfg and /etc/fstab.
Clone to smaller USB drive
Due to both EFI and /boot partitions are all small partition, and they are only needed during boot up, a smaller and slower USB drive can be used.
The outcome is also testing the root filesystem fully moved.
Create partition
- EFI partition, 512MB, vfat, on USB drive
- /boot, btrfs, 1GB, on USB drive
Create filesystem
/dev/sdc1 is EFI partition, and /dev/sdc2 is /boot partition
mkfs.vfat /dev/sdc1
mkfs.btrfs /dev/sdc2
Deplicate files
mount /dev/sdc2 /mnt
mkdir /mnt/efi
mount /dev/sdc1 /mnt/efi
rsync -avhP /boot/ /mnt/
Umount filesystems
umount /mnt/efi
umount /mnt
umount /boot/efi
umount /boot
Update /etc/fstab
Edit /etc/fstab and update UUID for both /boot and /boot/efi as below
/dev/disk/by-uuid/812cce04-3b56-4e17-8e38-b325304293f2 /boot btrfs defaults 0 1
/dev/disk/by-uuid/6B77-6F14 /boot/efi vfat defaults 0 1
Mount filesystem
This is also to confirm /etc/fstab is correct.
mount -a
Update grub
Run following command, and verify grub settings, include /boot/grub/grub.cfg, /boot/efi/EFI/ubuntu/grub.cfg.
dpkg-reconfigure grub-efi-amd64
Final test boot
Shutdown system and remove original USB, after that power on device. Make sure everything are expected after system boot up.
Other considerations
Disable iSCSI logout
The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (/) filesystem. Disable iSCSI logout during service stop.
systemctl edit --full open-iscsi.service
Comment out following line
#ExecStop=/lib/open-iscsi/logout-all.sh
Use fix IP address for iSCSI
Following IP configuration can be used to configure fix IP (192.168.1.51) in kernel
ip=192.168.1.51::192.168.1.254:255.255.255.0:fish:enp0s10::192.168.1.250::
If the IP address is different than OS, and they are using same interface, then there will be two IP entries for same interface, for example,
2: enp0s10: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:26:4a:18:82:c6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.51/24 brd 192.168.1.255 scope global enp0s10
valid_lft forever preferred_lft forever
inet 192.168.1.9/24 brd 192.168.1.255 scope global secondary dynamic enp0s10
valid_lft 43172sec preferred_lft 43172sec
inet6 fe80::226:4aff:fe18:82c6/64 scope link
valid_lft forever preferred_lft forever
With two IP address, the dedicated iSCSI IP range can be setup if required. Furthermore, dedicated network interface/LAN can be used for iSCSI network.
Future considerations
Missing iSCSI module
A recent issue I encountered for ubuntu `do-release-upgrade`, is missing `iscsi_tcp.ko` module, and the package `linux-modules-extra` is not in apt list. The apt dependency doesn't include linux-modules-extra, but the package can be downloaded from apt repository, manual installation is required.
The latest missing package can be installed using following command.
apt install linux-modules-extra-raspi
Backup USB device
Because USB devices are not mirror, backing up is required, and restoration is essential. The first partition in LUN can be used for backup.
Boot partition on iSCSI
If UEFI supports iSCSI, the boot device should able to be in iSCSI LUN as well. The advantage is, boot device also can be part of LUN snapshot for backup. But three stages involve iSCSI connection maybe having issues.
- UEFI iSCSI connection
- Grub iSCSI connection
- OS iSCSI connection
In fact, the OS doesn't need iSCSI connection if no additional iSCSI targets required except the one connected by Grub, because the LUNs connected in Grub are represented as local disks.
Multiboot
If move Boot partition to iSCSI LUN, then left one small configuration in EFI partition, such as UUID for boot device, etc. If setting up multiboot in EFI is possible, then the same USB device can be used for different boot devices.
Grub in Fedora way
In Fedora, the format of kernel parameters are different, and IP address and it's bridge configuration can be done in following way. If can be done in same way as Fedora, the IP address of iSCSI can be fixed, no need to be dhcp, and iSCSI definition can be shorter.
GRUB_CMDLINE_LINUX="netroot=iscsi::@::3260:: rd.iscsi.initiator=YOUR_INITIATOR_NAME rhgb quiet ip=192.168.1.9::192.168.1.254:255.255.255.0::br0:off nameserver=192.168.1.250 ifname=enp0s10:00:26:4a:18:82:c6 bridge=br0:enp0s10"
Prior to start
I had tried a few times to install ubuntu server 20.04 on MacBook Pro on iSCSI, but failed. I also tried Fedora 34, and it was successfully installed root partition on iSCSI LUN.
For Fedora 34, there are 3 partitions,
- EFI partition, 512MB, vfat, on USB drive
- /boot, btrfs, 2GB, on USB drive
- /, btrfs, on iSCSI LUN
It is using GRUB2.
After observation, I think I'm able to do the same for ubuntu.
In fact, I prefer ubuntu, because do-release-upgrade is doing well for ubuntu. For Fedora, I had done version upgrade many years ago too, but very manual and it is not official supported. Meaning one day, the upgrade method can never work again.
Troubleshooting
grub configuration error
Such as iSCSI configuration was given wrongly, due to no grub menu, the USB drive needs to be connected to another linux system to modify. An ubuntu VM is handy in this case.
References
Convert Raspberry Pi Ubuntu to iSCSI btrfs root
How to Configure the GRUB2 Boot Loader’s Settings
The kernel’s command-line parameters
update-grub does not update /boot/efi/EFI/ubuntu/grub.cfg