Table of Contents
Proxmox with UEFI iSCSI root
Note: I only record down the steps as draft, some steps are not required in this document, will update next time if I need to do same task.
I planed to convert TrueNAS to TrueNAS on Proxmox as many people implemented. The first step is install Proxmox.
As Proxmox uses local drive very frequently, slow USB disk is not suggested as mentioned in Internet, so I tried to convert Proxmox to iSCSI root.
- Proxmox is a proper virtual environment with container support (LXC)
- Proxmox is on a customized debian environment, which uses apt to update. But TrueNAS, at least now,
apt upgradebreaks installation.
- Proxmox can perform passthru controller or disk, give VM better storage management
- Boot partition only used during booting and Kernel update, and it is very small
- The root partition on iSCSI can take iSCSI advantages, such as snapshot, etc.
- TrueNAS VM can be converted to Proxmox as others do
- TrueNAS can not be iSCSI boot using normal method, because it is not a normal Linux
Another reason is, 10 days ago, my TrueNAS self rebooted even night, I could not find reason. And I also want to compare the performance between TrueNAS VM and Proxmox VM.
I installed Proxmox on TrueNAS VM first, then move VM EFI boot partition to physical USB drive.
- Create VM on TrueNAS
- 16GB USB drive for EFI boot partition (Only requires 513M USB drive)
- 10GB local disk for OS installation
- 16GB iSCSI LUN on NAS (10GB is enough)
Download proxmox image from Proxmox VE 7.1 ISO Installer
proxmox-ve_7.1-2.iso from website https://www.proxmox.com/en/downloads
Install proxmox in the VM with selection of zfs as filesystem.
It has 3 partitions
- BIOS boot partition (23-2047 sector)
- EFI partition, 512MB, vfat
The target is to move
/ to iSCSI LUN, and others to USB drive.
After OS installed, the some system structured info as below
- BIOS boot partition is not mountable
- EFI partition contains kernel, this is to avoid
- EFI partition is not mounted after boot up, this is to avoid corruption
- Proxmox kernel parameter file
- Proxmox boot loader files
- The zfs in proxmox uses partition device as disk, not partition id, better use partition id because it will not be changed.
- (Not required) Ubuntu grub parameter file -
This file is used to build actual
/boot/grub/grub.cfg, it is not required unless you like to boot into grub to verify configuration, which gives more error info.
Note: I used grub to detected error in cmdline. Installation of grub on boot disk is required, and reinstall proxmox boot is needed after troubleshooting.
Configure iSCSI service
The service was installed by default in proxmox.
node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
- Start iscsid service
After iscsid configuration changed, restart iscsid service is required.
systemctl restart iscsid
- Discover and Login
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login
Note: If can not login, restart
iscsid and try again.
Identify block device
lsblk command to identify device file, it should be something like
Creating two partitions using fdisk, the first partition is to prepare following for future used, such as
- Network boot
- UEFI iSCSI boot
- USB device backup
Partition /dev/sda2: vfat, 512MB (EFI)
Partition /dev/sda3: for root filesystem (Label as )
Note: The iSCSI LUN appears as
FYI, the first partition can not be created using
Format iSCSI LUN
/dev/sda1 as vfat
Copy EFI data
Note: The disk in VM appears as
mkdir /mnt/1 /mnt/2
mount /dev/vda2 /mnt/1
mount /dev/sda2 /mnt/2
cp -a . /mnt/2/
umount /mnt/1 /mnt/2
rmdir /mnt/1 /mnt/2
Note: Do not use
dd command at this stage, because there will be two partitions have same partition id,
proxmox-boot-tool will not update correctly
Attach iSCSI LUN to local disk
zpool attach rpool vda3 <partition_id of iSCSI LUN>
This is to enable ubuntu load iscsi driver during boot
echo "ISCSI_AUTO=true" > /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c
Note: I ran both update-initramfs command, one suggested by proxmox, another one I used before in other systems' migrations
Append following code into first line
ip=192.168.1.51::192.168.1.254:255.255.255.0:<hostname>:[interface]::192.168.1.250:: ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD>
Note: interface can be empty if only has one network card.
Update Proxmox boot script
Verify Proxmox Loader files
Verify Proxmox boot loader files
loader/loader.conf whether up to date.
Note: If two partitions has same partition id, the proxmox-boot-tool might updated other partition.
Disable iSCSI logout
The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (
/) filesystem. Disable iSCSI logout during service stop.
systemctl edit --full open-iscsi.service
Comment out following line
This is the first time test reboot, the outcome should be
- No hanging issue
- New iSCSI disks can be found after reboot without running
iscsiadmcommand manually. Verify using
- The command
zpool statusshows both local and iSCSI LUN are listed correctly.
The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.
Detach local disk
zpool detach rpool vda3
- Verify the rpool only has iSCSI LUN
Copy boot partitions to USB drive
This is to copy both BIOS boot and EFI partition. I used
dd command because the first partition could not be created manually by
fdisk, I think it can be created using
proxmox-boot-tool, but I didn't try that.
- Check size of partitions, and record down the first sector number of root partition
fdisk -l /dev/vda
dd if=/dev/vda of=/tmp/efi.dsk count=<the first sector number of root partition>
*Note: the count should be equal to the sectors cover both BIOS boot and EFI partition. Bigger is ok too, because we will remove the root partition.
Copy disk image to USB drive
dd if=/tmp/efi.dsk of=/dev/sdd
/dev/sdd is the USB device. I used another VM to do this
Remove third partition on USB drive
# fdisk /dev/sdd
Boot from USB drive
Insert USB drive into a physical server, and boot from USB
Booting can not detect iSCSI server
If you have multiple network card, need to update the interface name in ip parameter in boot menu.
- After booting into ramdisk run
ip afind correct interface name, then reboot
ekey to editor boot menu when showing boot menu
- Update network interface name
iSCSI login error
- Check multiple connection allowed option in iSCSI target configuration, especially if iSCSI logout disabled
Slow network caused iSCSI connection error
This is a strange issue, because it only happened before I successfully booted.
- After booting into ramdisk, run
ping <iSCSI server>, verify network connection
- Reboot and press
Control-safter iSCSI login, press
Control-qto release after detected iSCSI LUN
Verify iSCSI configuration
- Ubuntu grub parameter file
/etc/default/grubas listed in previous section.
GRUB_CMDLINE_LINUX_DEFAULT="ip=192.168.1.51::192.168.1.254:255.255.255.0:<hostname>:[interface]::192.168.1.250:: ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD>"
- Mount EFI partition
mount /dev/vda2 /boot/efi
- Update both
- Install grub on disk
grub-install is disabled by default
Return back to Proxmox Boot
proxmox-boot-tool init /dev/vda2
Follow up actions
Change bridge network interface depending on the correct physical interface.
USB disk free space
Use following command to create zfs on USB free space
Create partition to cover USB drive free space
zpool create upoolb <new_free_partition>
/upoolbdirectory to save ISO images or LXC templates
Both type of data are not updated fequently.