Tag: iscsi

Ubuntu iSCSI connect to multiple targets on boot

Posted on by Bian Xi Leave a Comment on Ubuntu iSCSI connect to multiple targets on boot

Ubuntu iSCSI connect to multiple targets on boot

Using iSCSI grub parameters can login to one iSCSI target at booting time, but for more than one targets, auto iSCSI session login might not be possible, at least I haven't found a way to do that.

In order to login to other multiple targets, modification need to be done in iSCSI discovered configuration

Login on boot

After discover the iSCSI targets, the targets will be in /etc/iscsi/nodes folder. Then found the correct target which in the format as below.

/etc/iscsi/nodes/<IQN>/<IP>,<Port>,1/default

Modify the above file and change following two parameters

...
node.startup = automatic
...
node.conn[0].startup = onboot
...

Ubuntu with UEFI iSCSI root on x86_64

Posted on by Bian Xi Leave a Comment on Ubuntu with UEFI iSCSI root on x86_64

Ubuntu with UEFI iSCSI root on x86_64

If I'm not wrong, the old ubuntu server can be installed directly on iSCSI disk for MBR type of PC. But I like to convert a Core 2 Due MacBook Pro to Ubuntu server, which only has UEFI and can not boot up into MBR.

Note: This is just my observation. Maybe I'm wrong.

Preparation

  • Ubuntu 20.04 installation USB drive
  • 16GB USB drive for OS installation

Steps

Install OS

Partition the USB drive into the same format as Fedora root on iSCSI which I had done before.

  • EFI partition, 512MB, vfat, on USB drive
  • /boot, btrfs, 1GB, on USB drive
  • /, btrfs, on iSCSI LUN

Note: I separated /boot and /, because I need to move / into iSCSI LUN, and I'm also not sure UEFI can configure iSCSI to detect iSCSI LUN before grub find out /boot partition. I tried UEFI in raspberry pi, which can configure iSCSI, but I didn't see such menu in MacBook Pro.

OS structure

After installed, the system structured as below

  • UEFI - /boot/efi/EFI
# find /boot/efi -ls
        1      4 drwxr-xr-x   3 root     root         4096 Jan  1  1970 /boot/efi
        4      4 drwxr-xr-x   4 root     root         4096 Nov  9 23:31 /boot/efi/EFI
        7      4 drwxr-xr-x   2 root     root         4096 Nov  9 23:31 /boot/efi/EFI/BOOT
       36    936 -rwxr-xr-x   1 root     root       955656 Nov 10 10:46 /boot/efi/EFI/BOOT/BOOTX64.EFI
       37     84 -rwxr-xr-x   1 root     root        85672 Nov 10 10:46 /boot/efi/EFI/BOOT/fbx64.efi
       38    840 -rwxr-xr-x   1 root     root       856232 Nov 10 10:46 /boot/efi/EFI/BOOT/mmx64.efi
       11      4 drwxr-xr-x   2 root     root         4096 Nov 10 08:56 /boot/efi/EFI/ubuntu
       44      4 -rwxr-xr-x   1 root     root          108 Nov 10 10:46 /boot/efi/EFI/ubuntu/BOOTX64.CSV
       45      4 -rwxr-xr-x   1 root     root          121 Nov 10 10:46 /boot/efi/EFI/ubuntu/grub.cfg
       46   1696 -rwxr-xr-x   1 root     root      1734528 Nov 10 10:46 /boot/efi/EFI/ubuntu/grubx64.efi
       47    840 -rwxr-xr-x   1 root     root       856232 Nov 10 10:46 /boot/efi/EFI/ubuntu/mmx64.efi
       48    936 -rwxr-xr-x   1 root     root       955656 Nov 10 10:46 /boot/efi/EFI/ubuntu/shimx64.efi

In above list, there following two files are important

The file /boot/efi/EFI/ubuntu/BOOTX64.CSV has following content

shimx64.efi,ubuntu,,This is the boot entry for ubuntu

The EFI grub configuration file, /boot/efi/EFI/ubuntu/grub.cfg has following content, which contains uuid of boot partition and location info, and it is named as root hd3,gpt2

search.fs_uuid 812cce04-3b56-4e17-8e38-b325304293f2 root hd3,gpt2
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

Note: Although the USB device location number is changing depending on the sequence of device detection, but the uuid will never be changed. Here, names it as hd3,gpt2, is because boot partition was the gpt partition 2 on 3rd device /dev/sdd2. The device name hd3 doesn't need to be the real device location, but it is only the reference to be used in ubuntu grab configuration later.

  • Ubuntu boot directory - /boot

This directory includes kernel files and grub configuration file.

-rw------- 1 root root  4755119 Oct 15 17:56 System.map-5.4.0-90-generic
-rw-r--r-- 1 root root   237884 Oct 15 17:56 config-5.4.0-90-generic
drwxr-xr-x 3 root root     4096 Jan  1  1970 efi
drwxr-xr-x 1 root root       82 Nov 10 08:54 grub
lrwxrwxrwx 1 root root       27 Nov  9 23:29 initrd.img -> initrd.img-5.4.0-90-generic
-rw-r--r-- 1 root root 84224544 Nov 10 02:51 initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root       27 Nov  9 23:29 initrd.img.old -> initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root       24 Nov  9 23:29 vmlinuz -> vmlinuz-5.4.0-90-generic
-rw------- 1 root root 11780352 Oct 15 19:36 vmlinuz-5.4.0-90-generic
lrwxrwxrwx 1 root root       24 Nov  9 23:29 vmlinuz.old -> vmlinuz-5.4.0-90-generic
  • Ubuntu grub - /boot/grub/grub.cfg

This is the grub configuration for ubuntu boot, the importent parts are, ip configuration, iscsi configuration, and turn off screen configuration.

linux /vmlinuz-5.4.0-90-generic root=UUID=<YOUR_DEV_UUID> ro ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30

Note: This file generated using /etc/default/grub, I'm not sure how to change root to label based, and there is option ro, which conflicts with rw that I was given in /etc/default/grub.

  • Ubuntu grub parameter file - /etc/default/grub

This file is used to build actual /boot/grub/grub.cfg.

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL=console

By default, nothing disappers on screen when booting up, uncomment the GRUB_TERMINAL=console to fix the issue.

Note: I prefer grub menu, it can be used to edit kernel parameter when anything went wrong. Troubleshooting can be easier during kernel upgrade.

Two kinds of commands use this file to update grub configuration

  • update-grub or update-grub2

Note: update-grub2 is a soft link of update-grub

This is to update /boot/grub/grub.cfg using /etc/default/grub file.

Some people reported issue that update-grub or update-grub2 which used this file to build grub configuration, doesn't update /boot/efi/EFI/ubuntu/grub.cfg file.

  • dpkg-reconfigure grub-efi-amd64

This command will update both grub configuration file /boot/grub/grub.cfg and EFI grub configuration file /boot/efi/EFI/ubuntu/grub.cfg.

Configure iSCSI service

The service was installed by default in ubuntu server 20.04.

  • Enable iscsid service
systemctl enable iscsid
  • Configure /etc/iscsi/initiatorname.iscsi
InitiatorName=<YOUR_INITIATOR_NAME>
  • Configure /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
  • Start iscsid service

After iscsid configuration changed, restart iscsid service is required.

systemctl restart iscsid
  • Discover and Login
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

Identify block device

Use lsblk command to identify device file, it should be something like /dev/sdX.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • USB device backup

Partition /dev/sdd1: vfat, 1GB, for /boot filesystem
Partition /dev/sdd2: for root filesystem

Note: The iSCSI LUN appears as /dev/sdd

Format iSCSI LUN

Format /dev/sdd1 as vfat and /dev/sdd2 as btrfs

mkfs.vfat /dev/sdd1
mkfs.btrfs /dev/sdd2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Note: Verifying iscsi module in updating list is important

Update grub

Ubuntu grub parameter file /etc/default/grub as listed in previous section. Beware of iSCSI parameters.

Update both /boot/grub/grub.cfg and /boot/efi/EFI/ubuntu/grub.cfg

dpkg-reconfigure grub-efi-amd64

Test Reboot

This is the first time test reboot, the outcome should be

  • No hanging issue
  • New iSCSI disks can be found after reboot without running iscsiadm command manually. Verify using lsblk command.

The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.

Duplicate files

Duplicate files to iSCSI LUN

mount /dev/sdd2 /mnt
mkdir /mnt/boot
mount /dev/sdd1 /mnt/boot
rsync -avhP --exclude /boot/efi --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/efi,mnt}

Change root partition in grub

Use UUID

Identify UUID for new root filesystem

blkid /dev/sdd2

Replace root device definitions as root=UUID=<UUID> in /boot/grub/grub.cfg using block id found

Use LABEL

Assign LABEL to new root filesystem

btrfs fi label /mnt ROOT

Replace root device definitions as root=LABEL=ROOT in /boot/grub/grub.cfg.

Update /etc/fstab

Replace root filesystem (/) uuid using the found in previous section (in iSCSI LUN), or if LABEL is assigned, then following line can be used.

LABEL=ROOT / btrfs defaults 0 1

Test Reboot

The reboot is to test root partition switchs to iSCSI LUN. Verify using df command.

/dev/sdc2       15727596 5091296  10317712  34% /
...
/dev/sda2        1048576  111400    819256  12% /boot
/dev/sda1         523248    5356    517892   2% /boot/efi

Now, the root (/) is in different device as /boot and /boot/efi.

Update grub again

Run dpkg-reconfigure grub-efi-amd64 again, then reboot the system. This is to verify all configuration are correct.

Test Reboot

After reboot, the system should have expected setup, which archives

  • The root (/) partition is in iSCSI LUN
  • Filesystems are structured according to /boot/efi/EFI/ubuntu/grub.cfg and /etc/fstab.

Clone to smaller USB drive

Due to both EFI and /boot partitions are all small partition, and they are only needed during boot up, a smaller and slower USB drive can be used.

The outcome is also testing the root filesystem fully moved.

Create partition

  • EFI partition, 512MB, vfat, on USB drive
  • /boot, btrfs, 1GB, on USB drive

Create filesystem

/dev/sdc1 is EFI partition, and /dev/sdc2 is /boot partition

mkfs.vfat /dev/sdc1
mkfs.btrfs /dev/sdc2

Deplicate files

mount /dev/sdc2 /mnt
mkdir /mnt/efi
mount /dev/sdc1 /mnt/efi
rsync -avhP /boot/ /mnt/

Umount filesystems

umount /mnt/efi
umount /mnt
umount /boot/efi
umount /boot

Update /etc/fstab

Edit /etc/fstab and update UUID for both /boot and /boot/efi as below

/dev/disk/by-uuid/812cce04-3b56-4e17-8e38-b325304293f2 /boot btrfs defaults 0 1
/dev/disk/by-uuid/6B77-6F14 /boot/efi vfat defaults 0 1

Mount filesystem

This is also to confirm /etc/fstab is correct.

mount -a

Update grub

Run following command, and verify grub settings, include /boot/grub/grub.cfg, /boot/efi/EFI/ubuntu/grub.cfg.

dpkg-reconfigure grub-efi-amd64

Final test boot

Shutdown system and remove original USB, after that power on device. Make sure everything are expected after system boot up.

Other considerations

Disable iSCSI logout

The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (/) filesystem. Disable iSCSI logout during service stop.

systemctl edit --full open-iscsi.service

Comment out following line

#ExecStop=/lib/open-iscsi/logout-all.sh

Use fix IP address for iSCSI

Following IP configuration can be used to configure fix IP (192.168.1.51) in kernel

ip=192.168.1.51::192.168.1.254:255.255.255.0:fish:enp0s10::192.168.1.250::

If the IP address is different than OS, and they are using same interface, then there will be two IP entries for same interface, for example,

2: enp0s10:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:26:4a:18:82:c6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.51/24 brd 192.168.1.255 scope global enp0s10
       valid_lft forever preferred_lft forever
    inet 192.168.1.9/24 brd 192.168.1.255 scope global secondary dynamic enp0s10
       valid_lft 43172sec preferred_lft 43172sec
    inet6 fe80::226:4aff:fe18:82c6/64 scope link 
       valid_lft forever preferred_lft forever

With two IP address, the dedicated iSCSI IP range can be setup if required. Furthermore, dedicated network interface/LAN can be used for iSCSI network.

Future considerations

Missing iSCSI module

A recent issue I encountered for ubuntu `do-release-upgrade`, is missing `iscsi_tcp.ko` module, and the package `linux-modules-extra` is not in apt list. The apt dependency doesn't include linux-modules-extra, but the package can be downloaded from apt repository, manual installation is required.

  • Update:

The latest missing package can be installed using following command.

apt install linux-modules-extra-raspi

Backup USB device

Because USB devices are not mirror, backing up is required, and restoration is essential. The first partition in LUN can be used for backup.

Boot partition on iSCSI

If UEFI supports iSCSI, the boot device should able to be in iSCSI LUN as well. The advantage is, boot device also can be part of LUN snapshot for backup. But three stages involve iSCSI connection maybe having issues.

  • UEFI iSCSI connection
  • Grub iSCSI connection
  • OS iSCSI connection

In fact, the OS doesn't need iSCSI connection if no additional iSCSI targets required except the one connected by Grub, because the LUNs connected in Grub are represented as local disks.

Multiboot

If move Boot partition to iSCSI LUN, then left one small configuration in EFI partition, such as UUID for boot device, etc. If setting up multiboot in EFI is possible, then the same USB device can be used for different boot devices.

Grub in Fedora way

In Fedora, the format of kernel parameters are different, and IP address and it's bridge configuration can be done in following way. If can be done in same way as Fedora, the IP address of iSCSI can be fixed, no need to be dhcp, and iSCSI definition can be shorter.

GRUB_CMDLINE_LINUX="netroot=iscsi::@::3260:: rd.iscsi.initiator=YOUR_INITIATOR_NAME rhgb quiet ip=192.168.1.9::192.168.1.254:255.255.255.0::br0:off nameserver=192.168.1.250 ifname=enp0s10:00:26:4a:18:82:c6 bridge=br0:enp0s10"

Prior to start

I had tried a few times to install ubuntu server 20.04 on MacBook Pro on iSCSI, but failed. I also tried Fedora 34, and it was successfully installed root partition on iSCSI LUN.

For Fedora 34, there are 3 partitions,

  • EFI partition, 512MB, vfat, on USB drive
  • /boot, btrfs, 2GB, on USB drive
  • /, btrfs, on iSCSI LUN

It is using GRUB2.

After observation, I think I'm able to do the same for ubuntu.

In fact, I prefer ubuntu, because do-release-upgrade is doing well for ubuntu. For Fedora, I had done version upgrade many years ago too, but very manual and it is not official supported. Meaning one day, the upgrade method can never work again.

Troubleshooting

grub configuration error

Such as iSCSI configuration was given wrongly, due to no grub menu, the USB drive needs to be connected to another linux system to modify. An ubuntu VM is handy in this case.

References

Convert Raspberry Pi Ubuntu to iSCSI btrfs root
How to Configure the GRUB2 Boot Loader’s Settings
The kernel’s command-line parameters
update-grub does not update /boot/efi/EFI/ubuntu/grub.cfg

Missing iSCSI module in Ubuntu 20.10

Posted on by Bian Xi Leave a Comment on Missing iSCSI module in Ubuntu 20.10

Missing iSCSI module in Ubuntu 20.10

Recently, I have upgraded Ubuntu 20.04 to 20.10, then iscsi_tcp module is missing, which caused iscsi LUN could not be loaded.

I checked /var/log/syslog, showed zram (reported by init-zram-swapping) and autofs (reported by containerd) module were missing too.

Error

Looking at error message shown during iscsi discovery, the command was looking for file iscsi_tcp.ko. Which can not be found in the system.

Update

The latest missing package can be installed using following command.

apt install linux-modules-extra-raspi

apt search

When performing apt search, shows three packages should include such file, but when listing all files in those packages, there is no iscsi_tcp.ko file at all.

# apt search iscsi_tcp.ko
Sorting... Done
Full Text Search... Done
libopeniscsiusr/impish,now 2.1.4-0ubuntu1 arm64 [installed]
  iSCSI userspace library

libopeniscsiusr-dev/impish 2.1.4-0ubuntu1 all
  iSCSI userspace library headers

open-iscsi/impish,now 2.1.4-0ubuntu1 arm64 [installed]
  iSCSI initiator tools

Reinstall OS

Then I decided to install a new fresh OS, but got the same result.

Search in ubuntu website

When searching packages content file iscsi_tcp.ko from ubuntu website, found there is no module package called linux-modules-extra for kernel 5.13.0-1009-raspi, the latest is 5.13.0-1008-raspi, which is for 20.04.

https://packages.ubuntu.com/search?suite=impish§ion=all&arch=any&keywords=iscsi_tcp.ko&searchon=contents

In other words, 20.10 doesn't have iscsi_tcp module, at all.

# apt install linux-modules-extra
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-modules-extra
#

Search in ubuntu repo

Then I go to ubuntu packages repo site below

http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/

...
/lib/modules/5.13.0-1008-oracle/kernel/drivers/scsi/libiscsi_tcp.ko     linux-modules-5.13.0-1008-oracle [arm64, amd64]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/iscsi_tcp.ko  linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/libiscsi_tcp.ko   linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/iscsi_tcp.ko     linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/libiscsi_tcp.ko  linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-19-generic-64k/kernel/drivers/scsi/iscsi_tcp.ko     linux-modules-5.13.0-19-generic-64k [arm64] 
...

found that there are linux-modules-extra package for both 1009 and 1010.

...
[ ] linux-modules-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb    2021-10-29 10:29    23M
[ ] linux-modules-extra-5.13.0-1008-raspi-nolpae_5.13.0-1008.9_armhf.deb    2021-09-29 20:44    21M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_arm64.deb   2021-09-29 20:43    23M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_armhf.deb   2021-09-29 20:44    21M
[ ] linux-modules-extra-5.13.0-1009-raspi-nolpae_5.13.0-1009.10_armhf.deb   2021-10-28 08:33    21M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb  2021-10-28 08:33    23M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_armhf.deb  2021-10-28 08:33    21M
[ ] linux-modules-extra-5.13.0-1010-raspi-nolpae_5.13.0-1010.11_armhf.deb   2021-10-29 10:29    21M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_arm64.deb  2021-10-29 10:29    23M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb  2021-10-29 10:29    21M
[ ] linux-raspi-headers-5.4.0-1008_5.4.0-1008.8_arm64.deb   2020-04-10 11:18    11M
...

Install linux-modules-extra

Then I decided to do download the package and check the contents

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb
dpkg -c linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb | grep iscsi

found the iscsi_tcp.ko is in the package

Install package via dpkg

dpkg -i linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb

Testing

Tested using iscsiadm, shows the problem fixed

iscsiadm --mode discovery --op update --type sendtargets --portal 192.168.1.17

In fact, this also fixed some other similar issues, such as, zram model missing, etc.

Reconfigure btrfs filesystem

In this system, the btrfs filesystem which was mirrored to iscsi LUN using bcache, the iscsi LUN was removed using following command during troubleshooting

btrfs balance start -f -sconvert=single -mconvert=single -dconvert=single /app
btrfs device remove /dev/bcache0 /app

Now use following command to reconfigure it back.

btrfs device add -f /dev/bcache0 /app
btrfs balance start -dconvert=raid1 -mconvert=raid1 /app

Configure module auto-loading

In order to load modules automatically, add iscsi_tcpline in /etc/modules. This may also avoid apt autoremove command removes linux-modules-extra package.

Possible cause

I think the issue was caused by spliting the kernel module package in to two in 1008, because linux-modules-extra only exists in the version 1008 in package searching list. I think ubnutu forgot this when releasing 1009. At this time, apt hasn't auto updated system to 1010.

References

Ubuntu Raspberry Package Repo
Ubuntu Package Searching
How to list files of a Debian package without install
How to install specific Ubuntu packages, with exact version?

iSCSI service failed after do-release-update

Posted on by Bian Xi Leave a Comment on iSCSI service failed after do-release-update

iSCSI service failed after do-release-update

The issue was caused by missing iscsi driver.

Errors

  • iscsid.service

Got following error

Oct 28 16:45:03 ubuntupi iscsid[6813]: can not create NETLINK_ISCSI socket [Protocol not supported]
  • iscsid.socket
     Active: failed (Result: service-start-limit-hit) since Thu 2021-10-28 16:40:07 +08; 1min 30s ago
  • depmod -a

Don't have iscsi module loaded

  • Error in dmesg
open-iscsi.service - Login to default iSCSI targets

    Process: 2333 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=21)

Proxmox with UEFI iSCSI root

Posted on by Bian Xi Leave a Comment on Proxmox with UEFI iSCSI root

Proxmox with UEFI iSCSI root

Note: I only record down the steps as draft, some steps are not required in this document, will update next time if I need to do same task.

I planed to convert TrueNAS to TrueNAS on Proxmox as many people implemented. The first step is install Proxmox.

As Proxmox uses local drive very frequently, slow USB disk is not suggested as mentioned in Internet, so I tried to convert Proxmox to iSCSI root.

Reasons

Some advantages

  • Proxmox is a proper virtual environment with container support (LXC)
  • Proxmox is on a customized debian environment, which uses apt to update. But TrueNAS, at least now, apt upgrade breaks installation.
  • Proxmox can perform passthru controller or disk, give VM better storage management
  • Boot partition only used during booting and Kernel update, and it is very small
  • The root partition on iSCSI can take iSCSI advantages, such as snapshot, etc.

Other reasons

  • TrueNAS VM can be converted to Proxmox as others do
  • TrueNAS can not be iSCSI boot using normal method, because it is not a normal Linux

Another reason is, 10 days ago, my TrueNAS self rebooted even night, I could not find reason. And I also want to compare the performance between TrueNAS VM and Proxmox VM.

Preparation

I installed Proxmox on TrueNAS VM first, then move VM EFI boot partition to physical USB drive.

  • Create VM on TrueNAS
  • 16GB USB drive for EFI boot partition (Only requires 513M USB drive)
  • 10GB local disk for OS installation
  • 16GB iSCSI LUN on NAS (10GB is enough)

Steps

Install OS

Download proxmox image from Proxmox VE 7.1 ISO Installer proxmox-ve_7.1-2.iso from website https://www.proxmox.com/en/downloads

Install proxmox in the VM with selection of zfs as filesystem.

It has 3 partitions

  • BIOS boot partition (23-2047 sector)
  • EFI partition, 512MB, vfat
  • /, zfs

The target is to move / to iSCSI LUN, and others to USB drive.

OS structure

After OS installed, the some system structured info as below

  • BIOS boot partition is not mountable
  • EFI partition contains kernel, this is to avoid /boot partition
  • EFI partition is not mounted after boot up, this is to avoid corruption
  • Proxmox kernel parameter file /etc/kernel/cmdline
  • Proxmox boot loader files loader/entries/entry.conf and loader/loader.conf
  • The zfs in proxmox uses partition device as disk, not partition id, better use partition id because it will not be changed.

Other files

  • (Not required) Ubuntu grub parameter file - /etc/default/grub

This file is used to build actual /boot/grub/grub.cfg, it is not required unless you like to boot into grub to verify configuration, which gives more error info.

Note: I used grub to detected error in cmdline. Installation of grub on boot disk is required, and reinstall proxmox boot is needed after troubleshooting.

Configure iSCSI service

The service was installed by default in proxmox.

  • Configure /etc/iscsi/initiatorname.iscsi
InitiatorName=<YOUR_INITIATOR_NAME>
  • Configure /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
  • Start iscsid service

After iscsid configuration changed, restart iscsid service is required.

systemctl restart iscsid
  • Discover and Login
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

Identify block device

Use lsblk command to identify device file, it should be something like /dev/sdX.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • USB device backup

Partition /dev/sda2: vfat, 512MB (EFI)
Partition /dev/sda3: for root filesystem (Label as )

Note: The iSCSI LUN appears as /dev/sda

FYI, the first partition can not be created using fdisk command.

Format iSCSI LUN

Format /dev/sda1 as vfat

mkfs.vfat /dev/sdd1

Copy EFI data

Note: The disk in VM appears as /dev/vda

mkdir /mnt/1 /mnt/2
mount /dev/vda2 /mnt/1
mount /dev/sda2 /mnt/2
cd /mnt/1
cp -a . /mnt/2/
umount /mnt/1 /mnt/2
rmdir /mnt/1 /mnt/2

Note: Do not use dd command at this stage, because there will be two partitions have same partition id, proxmox-boot-tool will not update correctly

Duplicate /root data

Find uuid

blkid

Attach iSCSI LUN to local disk

zpool attach rpool vda3 <partition_id of iSCSI LUN>

Update initramfs

This is to enable ubuntu load iscsi driver during boot

echo "ISCSI_AUTO=true" > /etc/iscsi/iscsi.initramfs
update-initramfs -u
update-initramfs -v -k $(uname -r) -c

Note: I ran both update-initramfs command, one suggested by proxmox, another one I used before in other systems' migrations

Update /etc/kernel/cmdline

Append following code into first line

ip=192.168.1.51::192.168.1.254:255.255.255.0:<hostname>:[interface]::192.168.1.250:: ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD>

Note: interface can be empty if only has one network card.

Update Proxmox boot script

proxmox-boot-tool refresh

Verify Proxmox Loader files

Verify Proxmox boot loader files loader/entries/entry.conf and loader/loader.conf whether up to date.

Note: If two partitions has same partition id, the proxmox-boot-tool might updated other partition.

Disable iSCSI logout

The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (/) filesystem. Disable iSCSI logout during service stop.

systemctl edit --full open-iscsi.service

Comment out following line

#ExecStop=/lib/open-iscsi/logout-all.sh

Test Reboot

This is the first time test reboot, the outcome should be

  • No hanging issue
  • New iSCSI disks can be found after reboot without running iscsiadm command manually. Verify using lsblk command.
  • The command zpool status shows both local and iSCSI LUN are listed correctly.

The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.

Detach local disk

zpool detach rpool vda3

Reboot Test

  • Verify the rpool only has iSCSI LUN

Copy boot partitions to USB drive

This is to copy both BIOS boot and EFI partition. I used dd command because the first partition could not be created manually by fdisk, I think it can be created using proxmox-boot-tool, but I didn't try that.

  • Check size of partitions, and record down the first sector number of root partition
fdisk -l /dev/vda
  • Use dd create image
dd if=/dev/vda of=/tmp/efi.dsk count=<the first sector number of root partition>

*Note: the count should be equal to the sectors cover both BIOS boot and EFI partition. Bigger is ok too, because we will remove the root partition.

Copy disk image to USB drive

dd if=/tmp/efi.dsk of=/dev/sdd

Note: /dev/sdd is the USB device. I used another VM to do this

Remove third partition on USB drive

# fdisk /dev/sdd
d
3
w

Boot from USB drive

Insert USB drive into a physical server, and boot from USB

Troubleshooting

Booting can not detect iSCSI server

If you have multiple network card, need to update the interface name in ip parameter in boot menu.

  • After booting into ramdisk run ip a find correct interface name, then reboot
  • Press e key to editor boot menu when showing boot menu
  • Update network interface name
  • Press Enter to boot

iSCSI login error

  • Check multiple connection allowed option in iSCSI target configuration, especially if iSCSI logout disabled

Slow network caused iSCSI connection error

This is a strange issue, because it only happened before I successfully booted.

  • After booting into ramdisk, run ping <iSCSI server>, verify network connection
  • Reboot and press Control-s after iSCSI login, press Control-q to release after detected iSCSI LUN

Verify iSCSI configuration

Update grub

  • Ubuntu grub parameter file /etc/default/grub as listed in previous section.
...
GRUB_CMDLINE_LINUX_DEFAULT="ip=192.168.1.51::192.168.1.254:255.255.255.0:<hostname>:[interface]::192.168.1.250:: ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD>"
  • Mount EFI partition
mount /dev/vda2 /boot/efi
  • Update both /boot/grub/grub.cfg and /boot/efi/EFI/ubuntu/grub.cfg
dpkg-reconfigure grub-efi-amd64
  • Install grub on disk
grub-install.real /dev/vda

Note: grub-install is disabled by default

Return back to Proxmox Boot

proxmox-boot-tool init /dev/vda2
proxmox-boot-tool refresh

Follow up actions

bridge network

Change bridge network interface depending on the correct physical interface.

USB disk free space

Use following command to create zfs on USB free space

  • Create partition to cover USB drive free space

  • Create zpool

zpool create upoolb <new_free_partition>
  • Add /upoolb directory to save ISO images or LXC templates

Both type of data are not updated fequently.

References

Host Bootloader
Installation
Proxmox ISCSI installation
Install Proxmox VE on Debian Buster

Convert Armbian to iSCSI btrfs root

Posted on by Bian Xi Leave a Comment on Convert Armbian to iSCSI btrfs root

Convert Armbian to iSCSI btrfs root

In order to have bad SD card worry-free, I decided to convert Armbian root filesystem to btrfs and move to Synology iSCSI LUN.

Pros

  • Backup can be done by iSCSI LUN snapshot
  • Only very small SD card needs to be used
  • iSCSI LUN can increase space easily
  • SD card data can be recreated easily
  • Faster even use slow SD card
  • Harddisk is cheaper than SD card

Steps

Separate /boot and root (/) partition

Move root filesystem to new SD card

This step needs to have another SD card, which needs to be able to hold all the original SD card files.

  • Insert the new SD card via USB card reader

  • Identify the SD card device name using lsblk command, normally should be /dev/sda

  • Format SD card to have two partitions, such as

/dev/sda1          2048  4196351  4194304   2G 83 Linux
/dev/sda2       4196352 33554431 29358080  14G 83 Linux

Note: There are some soft link in /boot filesystem, if /dev/sda1 is vfat, those soft link files will not be copied. I'm not sure if any issue will be encountered.

  • Run nand-sata-install
    • select following option
Boot from SD - system on SATA, USB or NVMe

  • Select /dev/sda2 as destination to install system to /dev/sda2

  • Select btrfs as filesystem type, after that, the system will format /dev/sda2 and transfer all files into this new partition.

  • Once completed, reboot the system

Verify system after reboot

Now, the system should have two filesystems

  • root (/), which is in new USB drive
  • /boot that binds to /media/mmcboot, which point to SD card, /dev/mmcblk0p1.

Copy all files from /boot to /dev/sda1

We create the partition /dev/sda1 has same filesystem type as original partition, and maintain the same structure /boot as well.

mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt
cp -a /boot /mnt

Note: The boot partition files can be in subdirectory of /mnt/boot, as well as in /mnt. If they are in /mnt/, then the path in /etc/fstab` needs to be changed. If failed to do this, the system is still bootable, just can not mount boot filesystem and taking time to scan filesystems as well.

Modify root partition UUID

Note: This should have been updated., because the root (/) is already running on new SD card

  • Find out the UUIDs for root (/) filesystem and /boot filesystem.
blkid
  • Update following line in /mnt/boot/armbianEnv.txt to root UUID if required.
rootdev=UUID=1c82450c-9013-43d9-9554-1049c264bfb8

  • Update root (/) filesystem UUID in /etc/fstab if required.

  • Update /boot filesystem UUID in /etc/fstab if required.

Shutdown

  • Shutdown the system
  • Take out old SD card from SD card slot
  • Remove new SD card USB device
  • Insert new SD card into SD card slot
  • Then power on system

Verify system

Now, the system should have /boot and root (/) filesystems on new SD card.

Move root to iSCSI

Install/configure iSCSI service

  • Install iSCSI package
apt install open-iscsi
  • Edit /etc/iscsi/initiatorname.iscsi, update following line
InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

  • Edit /etc/iscsi/iscsid.conf, update following lines
node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
  • Enable iscsid service and restart it
systemctl enable iscsid
systemctl restart iscsid
  • Login into iSCSI
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

/dev/sda1          2048  2099199  2097152   1G 83 Linux
/dev/sda2       2099200 33554431 31455232  15G 83 Linux

Note: The first partition can be used for iSCSI boot or /boot filesystem backup

Update initramfs

This is to enable kernel load iscsi driver during boot up

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Configure iSCSI kernel parameters

extraargs=ip=<ip_address>::<gateway>:<mask>:<host>:<interface_name>::<dns0>:<dns1>: ISCSI_INITIATOR=<ISCSI_INITIATOR> ISCSI_TARGET_NAME=<ISCSI_TARGET_NAME> ISCSI_TARGET_IP=<ISCSI_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: This includes IP configuration for fix IP. If use dhcp, just change to ip=dhcp

Reboot and verify

Make sure the iSCSI drive automatically loaded after reboot, and IP address assigned correctly.

This is to confirm that iSCSI is working during reboot.

Disable iscsi stop action

Disable iscsid service stop action is to prevent reboot hanging issue.

There is no need to logout as Allow multiple sessions is set to true in Synology iSCSI Target configuration, and the iSCSI Initiator used in both kernel and iscsid configure are the same.

systemctl edit --full open-iscsi.service

Comment out following line

#ExecStop=/lib/open-iscsi/logout-all.sh

Move root filesystem to iSCSI LUN

  • Run nand-sata-install, select following option
Boot from SD - system on SATA, USB or NVMe

  • Select /dev/sda2 as destination to install system to /dev/sda2

  • Select btrfs as filesystem type, then the system will format /dev/sda2, and transfer all files into this new partition.

  • Once completed, reboot the system

Verify system after reboot

Now, the system should have two filesystems

  • root (/), which is in iSCSI drive
  • /boot, binds to /media/mmcboot, which point to SD card, mmcblk0p1.

Other consideration

Recreate /boot/boot.scr

To make sure /boot/boot.scr is up to date, run following command is necessary especially if /boot/boot.cmd was modified.

mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

Backup /boot to iSCSI

Create filesystems

  • Create filesystem in iSCSI LUN
mkfs.ext4 /dev/sda1
  • Copy files into /dev/sda1
mount /dev/sda1 /mnt
cp -a /boot /mnt
umount /mnt

Recreate /boot SD card

Assuming the new SD card is named as /dev/sdb

  • Create /boot partition using fdisk

  • Create filesystem

mkfs.ext4 /dev/sdb1
  • Copy files
mount /dev/sdb1 /mnt
cp -a /boot /mnt
umount /mnt

  • Find out /dev/sdb1 UUID using command blkid

  • Modify /etc/fstab

    Update /boot filesystem UUID, which should be under /media/mmcboot entry.

  • Change SD card to the new card and reboot

SD card as cache

Thinking of how to use rest of space in SD card. Maybe can use it as bcache caching device to reduce the network traffic.

References

quick way to create SD card with separate /boot and / partitions?
Mounting the root filesystem via NFS (nfsroot)
Setting a Static IP Address Using the Kernel Command Line
Shutdown hang on 16.04 with iscsi targets
mkimage - Generate image for U-Boot
How to install to eMMC, NAND, SATA & USB?
Diskless iSCSI boot with PXE HOWTO

ESXi with UEFI iSCSI boot on Raspberry Pi

Posted on by Bian Xi Leave a Comment on ESXi with UEFI iSCSI boot on Raspberry Pi

ESXi with UEFI iSCSI boot on Raspberry Pi

Steps

Setup iSCSI disk

  • Create iSCSI Target and LUN in Synology
  • Download RPi4 UEFI Firmware, and unzip it to a SD card which formatted as FAT32 partition
  • Boot from the SD card, and perform following tasks using UEFI menu
    • Disable 3G memory limit
      Device Manager => Raspberry Pi Configuration => Advanced Configuration => Limit RAM to 3 GB)
    • Create device which mapped to iSCSI target
      Device Manager => iSCSI Configuration => Add an Attempt

After Attempt 1 created, Reset (restart) Raspberry Pi. Now, in Boot Manager, should see UEFI SYNOLOGY iSCSI Storage.

Setup boot order

  • Change Boot order and let it before other network boot, otherwise, there will be too much waiting time.

Prepare ESXi installation disk

  • Download and flush VMware-VMvisor-Installer-7.0.0-xxxx.aarch64.iso to USB device

Install ESXi

  • Reset (Reboot) again, and in UEFI menu select boot from USB device
  • Then perform ESXi installation, and select iSCSI disk as target

After installation completed, take out ESXi installation USB, then another reset is required,

Configure ESXi

  • Boot into iSCSI
  • Change ESXi name, etc.

Troubleshooting

Unable to see iSCSI disk in Boot Manager

Most likely is the iSCSI configuration wrong.

  • Check iSCSI Target Name
  • Check iSCSI Target IP
  • Check iSCSI LUN ID (This issue costed me a few hours)
  • Check User/Password

Synchronous Exception

After installation complete, suddenly cannot boot into any destination, and just show error Synchronous Exception.

End up, I have to recopy UEFI image into micro SD card, redo iSCSI configuration. Luckily the iSCSI has no issue, which contains installed ESXi image.

References

Boot ESXi-Arm Fling on a Raspberry Pi 4 Using ISCSI
Raspberry Pi 4 UEFI Firmware Images
ESXi on Arm 10/22 更新
Raspberry Pi 4 Model B 8GBにESXi for ARM 7.0.0をインストール
Synchronous Exception at 0x00000000371013D8 #97

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Posted on by Bian Xi Leave a Comment on Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

This is to move root file system of Raspberry Pi Ubuntu OS to iSCSI LUN and convert it to btrfs file system

Pros

  • Cheaper than using small SD card
  • Backup and restore easy, only need to backup 150MB boot partition on SD card
  • Able to perform snapshot at LUN level or OS level (btrfs)
  • Should be Faster

Steps

Install iscsi packages

apt install open-iscsi
systemctl enable open-iscsi
systemctl start open-iscsi
systemctl enable iscsid
systemctl start iscsid

Configure iscsi

Edit /etc/iscsi/initiatorname.iscsi, update following line

InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • SD card backup

Partition /dev/sda1: vfat, 2GB
Partition /dev/sda2: for root filesystem

Create filesystems

mkfs.vfat /dev/sda1
mkfs.btrfs /dev/sda2

Identify UUID for root filesystem

blkid /dev/sda2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Duplicate files

mount /dev/sda2 /mnt
rsync -avhP --exclude /boot/firmware --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/firmware,mnt}

Modify /etc/fstab in LUN

Note: Don't modify the file in /etc, it will not be used during iSCSI boot.

vi /mnt/etc/fstab

Change root mounting to

UUID=<YOUR_DEV_UUID> / btrfs defaults 1 1

Modify /boot/firmware/cmdline.txt

First create a backup of this file

cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.sav

Change the content of /boot/firmware/cmdline.txt

Note: Beware of rootfstype=btrfs

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=btrfs elevator=deadline rootwait fixrtc ip=dhcp root=UUID=<YOUR_DEV_UUID> ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: Root partition can also be identified by label if assigned one, see next topic

Reboot

umount /mnt
reboot

Use LABEL for root

Using label instead of UUID for root filesystem.

Assign a label

For mounted filesystem

sudo btrfs filesystem label <mountpoint> <newlabel>

For not mounted filesystem

sudo btrfs filesystem label <device> <newlabel>

Change /etc/fstab

LABEL=<newlabel> / btrfs defaults 1 1

Change /boot/fireware/cmdline.txt

... root=LABEL=<newlabel> ...

Use Static IP

To use static IP for iSCSI connection, the ip definition in cmdline.txt needs to be changed to

ip=192.168.1.200::192.168.1.1:255.255.255.0:rpi:eth0:off

This will create an IP 192.168.1.200 on interface eth0 as below.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ef:07:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.23/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 43036sec preferred_lft 43036sec
    inet6 fe80::dea6:32ff:feef:70f/64 scope link
       valid_lft forever preferred_lft forever

One way to just maintain one IP is, disabling OS IP address.

Reboot

Other cmdline.txt format

I tried following format, but failed. I think it maybe work if no partition in iSCSI LUN

root=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

Use UUID is better, because there is no need to worry about LUN id and partition as UUID is unique ideitifier.

To specfic UUID, following format can be used according to kernel parameters, but it doesn't work too.

root=UUID=<UUID>
netroot=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

I also tried rd.* format, such as rd.iscsi.initiator, etc., they are new format to replace old format ISCSI_INITIATOR, etc., but also not failed. I think the dracut.cmdline version used by ubuntu in raspberry pi is old. Maybe should try grub2 used in Fedora OS for raspberry pi, or uefi used in Windows 10.

Backup and restore using pre-backup data

Backup data into iSCSI LUN partition 1

First create vfat in iSCSI LUN as partition 1, then backup /boot/fireware data into that partition

mkfs.vfat /dev/sda1
mount /dev/sda1 /mnt
cp /boot/firmware/. /mnt
umount /mnt

Restore to an empty SD card

Create partition in new SSD as type c, which W95 FAT32 (LBA), with Boot flag.

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048 1050623 1048576  512M  c W95 FAT32 (LBA)

Format SD card and give label as system-boot, the LABEL is defined in /etc/fstab, it can be changed to UUID if needed.

mkfs.vfat -n system-boot /dev/sdb1
mount /dev/sdb1 /mnt
cp -a <backup_filesystem> /mnt
umount /mnt

Make sure system reported correct LABEL on the newly created vfat filesystem, using blkid command to verify.

Troubleshooting

iscsi_tcp missing

If following error occurred, install package linux-modules-extra.

libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='iscsi_tcp'

Please read post Missing iSCSI module in Ubuntu 20.10

Reboot error

If failed to boot, initram command prompt will appear. In this case, following commands can be used to recover back the booting.

mkdir /mnt
mount /dev/mmcblk0p1 /mnt
cd /mnt
cp cmdline.txt.sav cmdline.txt
cd /
umount /mnt
reboot

Note: the umount is very important, otherwise, the changes wouldn't be saved.

There are very minimum commands can be used, such as no vi. So let it boots into previous status, then troubleshooting from there.

References

dracut kernel command line options
Kernel command line parameters
introduction to boot time parameters of the Linux kernel
Raspberry Pi 4 UEFI Boot
RPi cmdline.txt
RPi config.txt
kernel-parameters.txt
The config.txt file
Raspberry Pi iSCSI Root on Ubuntu 20.04
btrfs root filesystem on raspbian
[Howto] booting from iSCSI
Ubuntu Server 20.10 on Raspberry Pi 4: installation guide with USB Boot (no SD card) and full disk encryption (excluding /boot) using btrfs-inside-luks and auto-apt snapshots with Timeshift
Raspberry Pi 4 - Ubuntu 20.04 w/Btrfs root
dracut.cmdline(7) — Linux manual page

Network filesystem timeout settings

Posted on by Bian Xi Leave a Comment on Network filesystem timeout settings

Network filesystem timeout settings

Network disruptions are always happening, network filesystems on different OS have different behaviors.

NFS

During Synology disk migration and SSD cache reconfiguration, my Fedora 34 on iSCSI mounted NFS disk kept hanging, I checked the default NFS mount options, then found that it was using hard option with out intr as below,

192.168.1.10:/volume1/kvm on /kvm type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.9,local_lock=none,addr=192.168.1.10)

I think maybe this is giving the factor of hanging.

iSCSI

After I changed NFS setting to soft, I suddenly realized that my iSCSI used by Fedora OS might not able to handle interupt as well, not sure whether iSCSI got similar options.

Samba on MacOS

My MacOS also got issue on samba filesystem, always disconnected after communication dropped, but my Windows machine has no such issue.

References

What are the differences between hard mount and soft mount?