Convert Armbian to iSCSI btrfs root

In order to have bad SD card worry-free, I decided to convert Armbian root filesystem to btrfs and move to Synology iSCSI LUN.

Pros

• Backup can be done by iSCSI LUN snapshot
• Only very small SD card needs to be used
• iSCSI LUN can increase space easily
• SD card data can be recreated easily
• Faster even use slow SD card
• Harddisk is cheaper than SD card

Steps

Separate /boot and root (/) partition

Move root filesystem to new SD card

This step needs to have another SD card, which needs to be able to hold all the original SD card files.

• Insert the new SD card via USB card reader

• Identify the SD card device name using lsblk command, normally should be /dev/sda

• Format SD card to have two partitions, such as

/dev/sda1          2048  4196351  4194304   2G 83 Linux
/dev/sda2       4196352 33554431 29358080  14G 83 Linux

Note: There are some soft link in /boot filesystem, if /dev/sda1 is vfat, those soft link files will not be copied. I'm not sure if any issue will be encountered.

• Run nand-sata-install
  • select following option

Boot from SD - system on SATA, USB or NVMe

• Select /dev/sda2 as destination to install system to /dev/sda2

• Select btrfs as filesystem type, after that, the system will format /dev/sda2 and transfer all files into this new partition.

• Once completed, reboot the system

Verify system after reboot

Now, the system should have two filesystems

• root (/), which is in new USB drive
• /boot that binds to /media/mmcboot, which point to SD card, /dev/mmcblk0p1.

Copy all files from /boot to /dev/sda1

We create the partition /dev/sda1 has same filesystem type as original partition, and maintain the same structure /boot as well.

mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt
cp -a /boot /mnt

Note: The boot partition files can be in subdirectory of /mnt/boot, as well as in /mnt. If they are in /mnt/, then the path in /etc/fstab` needs to be changed. If failed to do this, the system is still bootable, just can not mount boot filesystem and taking time to scan filesystems as well.

Modify root partition UUID

Note: This should have been updated., because the root (/) is already running on new SD card

• Find out the UUIDs for root (/) filesystem and /boot filesystem.

blkid

• Update following line in /mnt/boot/armbianEnv.txt to root UUID if required.

rootdev=UUID=1c82450c-9013-43d9-9554-1049c264bfb8

• Update root (/) filesystem UUID in /etc/fstab if required.

• Update /boot filesystem UUID in /etc/fstab if required.

Shutdown

• Shutdown the system
• Take out old SD card from SD card slot
• Remove new SD card USB device
• Insert new SD card into SD card slot
• Then power on system

Verify system

Now, the system should have /boot and root (/) filesystems on new SD card.

Move root to iSCSI

Install/configure iSCSI service

• Install iSCSI package

apt install open-iscsi

• Edit /etc/iscsi/initiatorname.iscsi, update following line

InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

• Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>

• Enable iscsid service and restart it

systemctl enable iscsid
systemctl restart iscsid

• Login into iSCSI

# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

/dev/sda1          2048  2099199  2097152   1G 83 Linux
/dev/sda2       2099200 33554431 31455232  15G 83 Linux

Note: The first partition can be used for iSCSI boot or /boot filesystem backup

Update initramfs

This is to enable kernel load iscsi driver during boot up

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Configure iSCSI kernel parameters

extraargs=ip=<ip_address>::<gateway>:<mask>:<host>:<interface_name>::<dns0>:<dns1>: ISCSI_INITIATOR=<ISCSI_INITIATOR> ISCSI_TARGET_NAME=<ISCSI_TARGET_NAME> ISCSI_TARGET_IP=<ISCSI_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: This includes IP configuration for fix IP. If use dhcp, just change to ip=dhcp

Reboot and verify

Make sure the iSCSI drive automatically loaded after reboot, and IP address assigned correctly.

This is to confirm that iSCSI is working during reboot.

Disable iscsi stop action

Disable iscsid service stop action is to prevent reboot hanging issue.

There is no need to logout as Allow multiple sessions is set to true in Synology iSCSI Target configuration, and the iSCSI Initiator used in both kernel and iscsid configure are the same.

systemctl edit --full open-iscsi.service

Comment out following line

#ExecStop=/lib/open-iscsi/logout-all.sh

Move root filesystem to iSCSI LUN

• Run nand-sata-install, select following option

Boot from SD - system on SATA, USB or NVMe

• Select /dev/sda2 as destination to install system to /dev/sda2

• Select btrfs as filesystem type, then the system will format /dev/sda2, and transfer all files into this new partition.

• Once completed, reboot the system

Verify system after reboot

Now, the system should have two filesystems

• root (/), which is in iSCSI drive
• /boot, binds to /media/mmcboot, which point to SD card, mmcblk0p1.

Other consideration

Recreate /boot/boot.scr

To make sure /boot/boot.scr is up to date, run following command is necessary especially if /boot/boot.cmd was modified.

mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

Backup /boot to iSCSI

Create filesystems

• Create filesystem in iSCSI LUN

mkfs.ext4 /dev/sda1

• Copy files into /dev/sda1

mount /dev/sda1 /mnt
cp -a /boot /mnt
umount /mnt

Recreate /boot SD card

Assuming the new SD card is named as /dev/sdb

• Create /boot partition using fdisk

• Create filesystem

mkfs.ext4 /dev/sdb1

• Copy files

mount /dev/sdb1 /mnt
cp -a /boot /mnt
umount /mnt

• Find out /dev/sdb1 UUID using command blkid

• Modify /etc/fstab

  Update /boot filesystem UUID, which should be under /media/mmcboot entry.

• Change SD card to the new card and reboot

SD card as cache

Thinking of how to use rest of space in SD card. Maybe can use it as bcache caching device to reduce the network traffic.

References

quick way to create SD card with separate /boot and / partitions?
Mounting the root filesystem via NFS (nfsroot)
Setting a Static IP Address Using the Kernel Command Line
Shutdown hang on 16.04 with iscsi targets
mkimage - Generate image for U-Boot
How to install to eMMC, NAND, SATA & USB?
Diskless iSCSI boot with PXE HOWTO