Proxmox with UEFI iSCSI root

Note: I only record down the steps as draft, some steps are not required in this document, will update next time if I need to do same task.

I planed to convert TrueNAS to TrueNAS on Proxmox as many people implemented. The first step is install Proxmox.

As Proxmox uses local drive very frequently, slow USB disk is not suggested as mentioned in Internet, so I tried to convert Proxmox to iSCSI root.

Reasons

Some advantages

• Proxmox is a proper virtual environment with container support (LXC)
• Proxmox is on a customized debian environment, which uses apt to update. But TrueNAS, at least now, apt upgrade breaks installation.
• Proxmox can perform passthru controller or disk, give VM better storage management
• Boot partition only used during booting and Kernel update, and it is very small
• The root partition on iSCSI can take iSCSI advantages, such as snapshot, etc.

Other reasons

• TrueNAS VM can be converted to Proxmox as others do
• TrueNAS can not be iSCSI boot using normal method, because it is not a normal Linux

Another reason is, 10 days ago, my TrueNAS self rebooted even night, I could not find reason. And I also want to compare the performance between TrueNAS VM and Proxmox VM.

Preparation

I installed Proxmox on TrueNAS VM first, then move VM EFI boot partition to physical USB drive.

• Create VM on TrueNAS
• 16GB USB drive for EFI boot partition (Only requires 513M USB drive)
• 10GB local disk for OS installation
• 16GB iSCSI LUN on NAS (10GB is enough)

Steps

Install OS

Download proxmox image from Proxmox VE 7.1 ISO Installer proxmox-ve_7.1-2.iso from website https://www.proxmox.com/en/downloads

Install proxmox in the VM with selection of zfs as filesystem.

It has 3 partitions

• BIOS boot partition (23-2047 sector)
• EFI partition, 512MB, vfat
• /, zfs

The target is to move / to iSCSI LUN, and others to USB drive.

OS structure

After OS installed, the some system structured info as below

• BIOS boot partition is not mountable
• EFI partition contains kernel, this is to avoid /boot partition
• EFI partition is not mounted after boot up, this is to avoid corruption
• Proxmox kernel parameter file /etc/kernel/cmdline
• Proxmox boot loader files loader/entries/entry.conf and loader/loader.conf
• The zfs in proxmox uses partition device as disk, not partition id, better use partition id because it will not be changed.

Other files

• (Not required) Ubuntu grub parameter file - /etc/default/grub

This file is used to build actual /boot/grub/grub.cfg, it is not required unless you like to boot into grub to verify configuration, which gives more error info.

Note: I used grub to detected error in cmdline. Installation of grub on boot disk is required, and reinstall proxmox boot is needed after troubleshooting.

Configure iSCSI service

The service was installed by default in proxmox.

• Configure /etc/iscsi/initiatorname.iscsi

InitiatorName=<YOUR_INITIATOR_NAME>

• Configure /etc/iscsi/iscsid.conf

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>

• Start iscsid service

After iscsid configuration changed, restart iscsid service is required.

systemctl restart iscsid

• Discover and Login

# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

Identify block device

Use lsblk command to identify device file, it should be something like /dev/sdX.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

• Network boot
• UEFI iSCSI boot
• USB device backup

Partition /dev/sda2: vfat, 512MB (EFI)
Partition /dev/sda3: for root filesystem (Label as )

Note: The iSCSI LUN appears as /dev/sda

FYI, the first partition can not be created using fdisk command.

Format iSCSI LUN

Format /dev/sda1 as vfat

mkfs.vfat /dev/sdd1

Copy EFI data

Note: The disk in VM appears as /dev/vda

mkdir /mnt/1 /mnt/2
mount /dev/vda2 /mnt/1
mount /dev/sda2 /mnt/2
cd /mnt/1
cp -a . /mnt/2/
umount /mnt/1 /mnt/2
rmdir /mnt/1 /mnt/2

Note: Do not use dd command at this stage, because there will be two partitions have same partition id, proxmox-boot-tool will not update correctly

Duplicate /root data

Find uuid

blkid

Attach iSCSI LUN to local disk

zpool attach rpool vda3 <partition_id of iSCSI LUN>

Update initramfs

This is to enable ubuntu load iscsi driver during boot

echo "ISCSI_AUTO=true" > /etc/iscsi/iscsi.initramfs
update-initramfs -u
update-initramfs -v -k $(uname -r) -c

Note: I ran both update-initramfs command, one suggested by proxmox, another one I used before in other systems' migrations

Update /etc/kernel/cmdline

Append following code into first line

ip=192.168.1.51::192.168.1.254:255.255.255.0:<hostname>:[interface]::192.168.1.250:: ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD>

Note: interface can be empty if only has one network card.

Update Proxmox boot script

proxmox-boot-tool refresh

Verify Proxmox Loader files

Verify Proxmox boot loader files loader/entries/entry.conf and loader/loader.conf whether up to date.

Note: If two partitions has same partition id, the proxmox-boot-tool might updated other partition.

Disable iSCSI logout

The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (/) filesystem. Disable iSCSI logout during service stop.

systemctl edit --full open-iscsi.service

Comment out following line

#ExecStop=/lib/open-iscsi/logout-all.sh

Test Reboot

This is the first time test reboot, the outcome should be

• No hanging issue
• New iSCSI disks can be found after reboot without running iscsiadm command manually. Verify using lsblk command.
• The command zpool status shows both local and iSCSI LUN are listed correctly.

The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.

Detach local disk

zpool detach rpool vda3

Reboot Test

• Verify the rpool only has iSCSI LUN

Copy boot partitions to USB drive

This is to copy both BIOS boot and EFI partition. I used dd command because the first partition could not be created manually by fdisk, I think it can be created using proxmox-boot-tool, but I didn't try that.

• Check size of partitions, and record down the first sector number of root partition

fdisk -l /dev/vda

• Use dd create image

dd if=/dev/vda of=/tmp/efi.dsk count=<the first sector number of root partition>

*Note: the count should be equal to the sectors cover both BIOS boot and EFI partition. Bigger is ok too, because we will remove the root partition.

Copy disk image to USB drive

dd if=/tmp/efi.dsk of=/dev/sdd

Note: /dev/sdd is the USB device. I used another VM to do this

Remove third partition on USB drive

# fdisk /dev/sdd
d
3
w

Boot from USB drive

Insert USB drive into a physical server, and boot from USB

Troubleshooting

Booting can not detect iSCSI server

If you have multiple network card, need to update the interface name in ip parameter in boot menu.

• After booting into ramdisk run ip a find correct interface name, then reboot
• Press e key to editor boot menu when showing boot menu
• Update network interface name
• Press Enter to boot

iSCSI login error

• Check multiple connection allowed option in iSCSI target configuration, especially if iSCSI logout disabled

Slow network caused iSCSI connection error

This is a strange issue, because it only happened before I successfully booted.

• After booting into ramdisk, run ping <iSCSI server>, verify network connection
• Reboot and press Control-s after iSCSI login, press Control-q to release after detected iSCSI LUN

Verify iSCSI configuration

Update grub

• Ubuntu grub parameter file /etc/default/grub as listed in previous section.

...
GRUB_CMDLINE_LINUX_DEFAULT="ip=192.168.1.51::192.168.1.254:255.255.255.0:<hostname>:[interface]::192.168.1.250:: ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD>"

• Mount EFI partition

mount /dev/vda2 /boot/efi

• Update both /boot/grub/grub.cfg and /boot/efi/EFI/ubuntu/grub.cfg

dpkg-reconfigure grub-efi-amd64

• Install grub on disk

grub-install.real /dev/vda

Note: grub-install is disabled by default

Return back to Proxmox Boot

proxmox-boot-tool init /dev/vda2
proxmox-boot-tool refresh

Follow up actions

bridge network

Change bridge network interface depending on the correct physical interface.

USB disk free space

Use following command to create zfs on USB free space

• Create partition to cover USB drive free space

• Create zpool

zpool create upoolb <new_free_partition>

• Add /upoolb directory to save ISO images or LXC templates

Both type of data are not updated fequently.

References

Host Bootloader
Installation
Proxmox ISCSI installation
Install Proxmox VE on Debian Buster