Category: raspberrypi

BerryBoot on Raspberry Pi

BerryBoot on Raspberry Pi

Raspberry Pi 2/3/4 requires SD card to be used for booting, which saves OS filesystems. There are some issues with SD card with OS boot.

  • Limited size
  • Hard to backup

I was thinking a solution to move root filesystem to iSCSI, but grub is not used in OS of Raspberry Pi. As the design of BerryBoot, it can install various OSes on iSCSI disk with multi-boot environment. So started testing it with Ubuntu OS.

Hardware

Although, BerryBoot declares supporting Raspberry Pi 1/2/3/4, I tried Pi 3B+, but very slow and hanging. End up used Pi 4 (8GB).

Boot up error

After installation completed, got following error. It is caused iSCSI LUN import requires time, at time of error reported, LUN had not detected by iSCSI driver.

iSCSI target does not have any partitions

The solution give is changing iscsi.sh file in boot partition (the SD card), adding loop for delay.

IP address and VNC

The BerryBoot interface used same IP as ubuntu OS, but it doesn't configure DNS. So after bootup, the ubuntu OS has no DNS setting, then manual update of file /etc/systemd/resolv.conf is required.

In order to remotely control BerryBoot, add following command to enable VNC. Then when booting error or like to select another OS, VNC viewer can be used to connect to BerryBoot.

Wired network headless installation

  • Append following at end of line in cmdline.txt (same line append)
... vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1

Wireless headless installation

  • Append following at end of line in cmdline.txt (same line append)
vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1/wlan0
  • Create a file called wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1

network={
    ssid="ssid-of-accesspoint"
    psk="wpa-password"
}

Update error

The most important task is update, which includes package update and kernel update

Package update

It is quite smooth during update, but complaining /boot doesn't exist. I checked the OS structure, there is a /boot folder, with one kernel, and a filesystem /media/user_id/boot, which saves SD contents.

Kernel update

Kernel update was totally failed with following error

Can't find /boot/vmlinuz- (see /tmp/flash-kernel-no-kernel-error.log)

The solution given by other is removing two packages

sudo apt remove u-boot-rpi:arm64
sudo apt remove flash-kernel

Note: According to internet, BerryBoot uses its own kernel, meaning it can not be updated using package update command.

do-release-update

This is a very strange action I took. The OS I installed via BerryBoot is Ubuntu 20.10 Desktop, but the OS still can perform do-release-update to same version. The even more strange thing is, after I upgrade, it still appear as can perform do-release-update, then I spent another upgrade, another few hours upgrade, then it can fininally upgraded.

I think the package was not at the correct release stage, but don't know why got 980+ packages to upgrade after first run. Felt like perform exactly the same action once more.

The output of uname after patched ubuntu screen as below

  • BerryBoot ubuntu
root@camel:~# uname -a
Linux camel 5.10.43v64 #1 SMP PREEMPT Tue Jun 15 00:52:48 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux
root@camel:~# 
  • Normal ubuntu
root@ubuntupi:/usr/lib# uname -a
Linux ubuntupi 5.13.0-1009-raspi #10-Ubuntu SMP PREEMPT Mon Oct 25 13:58:43 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntupi:/usr/lib# 

Filesystem structure

The BerryBoot ubuntu has different filesystem structure, the root filesystem is mapped to none, and no entry in /etc/fstab, I think the reason behind, is root filesystem is mounted via kernel, which is the iSCSI disk, and cannot be changed.

Output of df

root@camel:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
none            16646144 6422360   8396520  44% /
tmpfs            3957172       0   3957172   0% /dev/shm
tmpfs            1582872    1404   1581468   1% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs            3957172       0   3957172   0% /run/qemu
tmpfs             791432      72    791360   1% /run/user/126
tmpfs             791432      64    791368   1% /run/user/1000
root@camel:~# 

Output of mount

root@camel:~# mount | grep -w /
none on / type overlay (rw,relatime,lowerdir=/mnt/shared:/squashfs,upperdir=/mnt/data/Ubuntu_20.10_Desktop.img192,workdir=/mnt/data/Ubuntu_20.10_Desktop.img192.work,redirect_dir=on)
root@camel:~# 

The normal ubuntu installed has same structure of other distributions, using /etc/fstab for root filesystem as well. So the normal operations can be taken on root filesystem

Output of df

root@ubuntupi:/usr/lib# df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs             794548    12732    781816   2% /run
/dev/mmcblk0p2  30358480 24453720   4573672  85% /
tmpfs            3972736        0   3972736   0% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
tmpfs            3972736        0   3972736   0% /run/qemu
tmpfs            3972736     8364   3964372   1% /var/log
tmpfs            3972736        8   3972728   1% /tmp
tmpfs            3972736        0   3972736   0% /var/tmp
/dev/mmcblk0p1    258095    97201    160894  38% /boot/firmware
tmpfs             794544       92    794452   1% /run/user/126
tmpfs             794544       84    794460   1% /run/user/1000
root@ubuntupi:/usr/lib# 

Output of mount

root@ubuntupi:/usr/lib# mount | grep -w /
/dev/mmcblk0p2 on / type ext4 (rw,noatime,nobarrier)
root@ubuntupi:/usr/lib# 

Pros

Following advantages over normal ubuntu

  • Root filesystem type can be selected during installation. (Normal ubuntu on Raspberry Pi can not)
  • The iSCSI module is loaded by default.
    Note: Recent update of my normal ubuntu caused iSCSI module missing issue.
  • OS multiboot

Cons

Duo to BerryBoot has different filesystem structure and kernel, some diffculties would be facing later.

  • Specific Kernel
    • No kernel upgrade can cause package incompatible, for example, docker was mentioned by others
    • Auto rebuild kernel is almost impossible
  • Filesystem
    • Unable to verify root filesystem, scrubbing wasn't allowed during my testing
    • Unable to know root filesystem type even mounted
    • Cannot increase root filesystem size easily

Conclusion

The way BerryBoot manage the system is highly customized, which might not be adapted anywhere else. The customization also leads system compatibility issues later, and migration, backup, restore, etc., cannot be implemented in common way. So there could be no future-proof for this implemenation.

Compare local boot partition solution, which only holds root partition on iSCSI, the advantage of BerryBoot is iSCSI dependency issue reduced and multiboot OS capability. But the root on iSCSI uses bootload, which reduced OS compatibility issue.

My decision is still use normal bootload way for future systems to achieve maintennance free system.

References

Headless installation
Problems with update/upgrade Ubuntu 20.10 on Raspberry pi 4 with SSD
Raspberry Pi iSCSI Root on Ubuntu 20.04
How to Dual Boot a Raspberry Pi Using BerryBoot
BerryBoot v2.0 - bootloader / universal operating system installer
Storing your files on a Synology NAS (using iSCSI)

Raspberry Pi Commands Basic

Raspberry Pi Commands Basic

Update

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo rpi-update

Hardware Info

cat /proc/cpuinfo discover information about the Raspberry Pi's CPU
cat /proc/meminfo displays details about the Raspberry Pi's memory.
cat /proc/partitions reveals the size and number of partitions on your SD card or HDD.
cat /proc/version shows you which version of the Pi you are using.
vcgencmd measure_temp reveals the CPU temperature (vital if you're concerned about airflow).
vcgencmd get_mem arm && vcgencmd get_mem gpu will reveal the memory split between the CPU and GPU, which can be adjusted in the config screen.
free -o -h will display the available system memory.
top d1 checks the load on your CPU, displaying details for all cores.

Shutdown

sudo shutdown -h now will commence the shutdown process with immediate effect.
sudo shutdown -h 21:55 Schedule a timed shutdown at 21:55

References

Raspberry Pi Terminal Commands: A Quick Guide for Raspberry Pi Users

ESXi with UEFI iSCSI boot on Raspberry Pi

ESXi with UEFI iSCSI boot on Raspberry Pi

Steps

Setup iSCSI disk

  • Create iSCSI Target and LUN in Synology
  • Download RPi4 UEFI Firmware, and unzip it to a SD card which formatted as FAT32 partition
  • Boot from the SD card, and perform following tasks using UEFI menu
    • Disable 3G memory limit
      Device Manager => Raspberry Pi Configuration => Advanced Configuration => Limit RAM to 3 GB)
    • Create device which mapped to iSCSI target
      Device Manager => iSCSI Configuration => Add an Attempt

After Attempt 1 created, Reset (restart) Raspberry Pi. Now, in Boot Manager, should see UEFI SYNOLOGY iSCSI Storage.

Setup boot order

  • Change Boot order and let it before other network boot, otherwise, there will be too much waiting time.

Prepare ESXi installation disk

  • Download and flush VMware-VMvisor-Installer-7.0.0-xxxx.aarch64.iso to USB device

Install ESXi

  • Reset (Reboot) again, and in UEFI menu select boot from USB device
  • Then perform ESXi installation, and select iSCSI disk as target

After installation completed, take out ESXi installation USB, then another reset is required,

Configure ESXi

  • Boot into iSCSI
  • Change ESXi name, etc.

Troubleshooting

Unable to see iSCSI disk in Boot Manager

Most likely is the iSCSI configuration wrong.

  • Check iSCSI Target Name
  • Check iSCSI Target IP
  • Check iSCSI LUN ID (This issue costed me a few hours)
  • Check User/Password

Synchronous Exception

After installation complete, suddenly cannot boot into any destination, and just show error Synchronous Exception.

End up, I have to recopy UEFI image into micro SD card, redo iSCSI configuration. Luckily the iSCSI has no issue, which contains installed ESXi image.

References

Boot ESXi-Arm Fling on a Raspberry Pi 4 Using ISCSI
Raspberry Pi 4 UEFI Firmware Images
ESXi on Arm 10/22 更新
Raspberry Pi 4 Model B 8GBにESXi for ARM 7.0.0をインストール
Synchronous Exception at 0x00000000371013D8 #97

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

This is to move root file system of Raspberry Pi Ubuntu OS to iSCSI LUN and convert it to btrfs file system

Pros

  • Cheaper than using small SD card
  • Backup and restore easy, only need to backup 150MB boot partition on SD card
  • Able to perform snapshot at LUN level or OS level (btrfs)
  • Should be Faster

Steps

Install iscsi packages

apt install open-iscsi
systemctl enable open-iscsi
systemctl start open-iscsi
systemctl enable iscsid
systemctl start iscsid

Configure iscsi

Edit /etc/iscsi/initiatorname.iscsi, update following line

InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • SD card backup

Partition /dev/sda1: vfat, 2GB
Partition /dev/sda2: for root filesystem

Create filesystems

mkfs.vfat /dev/sda1
mkfs.btrfs /dev/sda2

Identify UUID for root filesystem

blkid /dev/sda2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Duplicate files

mount /dev/sda2 /mnt
rsync -avhP --exclude /boot/firmware --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/firmware,mnt}

Modify /etc/fstab in LUN

Note: Don't modify the file in /etc, it will not be used during iSCSI boot.

vi /mnt/etc/fstab

Change root mounting to

UUID=<YOUR_DEV_UUID> / btrfs defaults 1 1

Modify /boot/firmware/cmdline.txt

First create a backup of this file

cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.sav

Change the content of /boot/firmware/cmdline.txt

Note: Beware of rootfstype=btrfs

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=btrfs elevator=deadline rootwait fixrtc ip=dhcp root=UUID=<YOUR_DEV_UUID> ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: Root partition can also be identified by label if assigned one, see next topic

Reboot

umount /mnt
reboot

Use LABEL for root

Using label instead of UUID for root filesystem.

Assign a label

For mounted filesystem

sudo btrfs filesystem label <mountpoint> <newlabel>

For not mounted filesystem

sudo btrfs filesystem label <device> <newlabel>

Change /etc/fstab

LABEL=<newlabel> / btrfs defaults 1 1

Change /boot/fireware/cmdline.txt

... root=LABEL=<newlabel> ...

Use Static IP

To use static IP for iSCSI connection, the ip definition in cmdline.txt needs to be changed to

ip=192.168.1.200::192.168.1.1:255.255.255.0:rpi:eth0:off

This will create an IP 192.168.1.200 on interface eth0 as below.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ef:07:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.23/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 43036sec preferred_lft 43036sec
    inet6 fe80::dea6:32ff:feef:70f/64 scope link
       valid_lft forever preferred_lft forever

One way to just maintain one IP is, disabling OS IP address.

Reboot

Other cmdline.txt format

I tried following format, but failed. I think it maybe work if no partition in iSCSI LUN

root=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

Use UUID is better, because there is no need to worry about LUN id and partition as UUID is unique ideitifier.

To specfic UUID, following format can be used according to kernel parameters, but it doesn't work too.

root=UUID=<UUID>
netroot=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

I also tried rd.* format, such as rd.iscsi.initiator, etc., they are new format to replace old format ISCSI_INITIATOR, etc., but also not failed. I think the dracut.cmdline version used by ubuntu in raspberry pi is old. Maybe should try grub2 used in Fedora OS for raspberry pi, or uefi used in Windows 10.

Backup and restore using pre-backup data

Backup data into iSCSI LUN partition 1

First create vfat in iSCSI LUN as partition 1, then backup /boot/fireware data into that partition

mkfs.vfat /dev/sda1
mount /dev/sda1 /mnt
cp /boot/firmware/. /mnt
umount /mnt

Restore to an empty SD card

Create partition in new SSD as type c, which W95 FAT32 (LBA), with Boot flag.

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048 1050623 1048576  512M  c W95 FAT32 (LBA)

Format SD card and give label as system-boot, the LABEL is defined in /etc/fstab, it can be changed to UUID if needed.

mkfs.vfat -n system-boot /dev/sdb1
mount /dev/sdb1 /mnt
cp -a <backup_filesystem> /mnt
umount /mnt

Make sure system reported correct LABEL on the newly created vfat filesystem, using blkid command to verify.

Troubleshooting

iscsi_tcp missing

If following error occurred, install package linux-modules-extra.

libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='iscsi_tcp'

Please read post Missing iSCSI module in Ubuntu 20.10

Reboot error

If failed to boot, initram command prompt will appear. In this case, following commands can be used to recover back the booting.

mkdir /mnt
mount /dev/mmcblk0p1 /mnt
cd /mnt
cp cmdline.txt.sav cmdline.txt
cd /
umount /mnt
reboot

Note: the umount is very important, otherwise, the changes wouldn't be saved.

There are very minimum commands can be used, such as no vi. So let it boots into previous status, then troubleshooting from there.

References

dracut kernel command line options
Kernel command line parameters
introduction to boot time parameters of the Linux kernel
Raspberry Pi 4 UEFI Boot
RPi cmdline.txt
RPi config.txt
kernel-parameters.txt
The config.txt file
Raspberry Pi iSCSI Root on Ubuntu 20.04
btrfs root filesystem on raspbian
[Howto] booting from iSCSI
Ubuntu Server 20.10 on Raspberry Pi 4: installation guide with USB Boot (no SD card) and full disk encryption (excluding /boot) using btrfs-inside-luks and auto-apt snapshots with Timeshift
Raspberry Pi 4 - Ubuntu 20.04 w/Btrfs root
dracut.cmdline(7) — Linux manual page

Plan install a NextCloud server at home

Plan install a NextCloud server at home

Planning to have a NextCloud server at home. There are many solutions.

Solutions

Synology

Install NextCloud as docker in Synology ds1812+ or ds2419+, but

  • Unable to use other system next time.
  • Expose all data in NAS
  • CPU and memory limited

But consider use it as storage via iSCSI.

TrueNAS

Install NextCloud on TrueNAS is officially supported. It is installed as docker container in TrueNAS, but

  • Use more power as the system running on a normal PC
  • Migration might be hard

Raspberry Pi

Install NextCloud on Raspberry Pi, a few options there

  • Install directly on OS, but will mix up with OS as well
  • Install as docker, needs to have correct structure to be maintained
  • Install as NextCloudPi, limited storage on SD card

First plan

After considered all above, first plan will be

  • Install on a Pi which has 1GB ethernet port
  • Install BerryBoot and boot from Synology iSCSI disk
  • Configure storage utilize Synology NAS

Pros

  • Separate storage and application
  • Low power
  • Easy to replace the hardware
  • Got official update regularly

Second plan

After read some documents on line, feel the raspberry pi at current stage still isn't good for NextCloud. The major issue is the CPU speed, which also caused 1GB ethernet performance drop to 40MB/s.

I think I had better use my core 2 due machine to install NextCloud. Maybe I will try it on Raspberry Pi 4 with 8GB RAM later as well.

References

NextCloud Website
BerryBoot Installation
BerryBoot iSCSI Installation
Berryboot install NextCloudPi on an external drive step by step
Nextcloud now officially supported on TrueNAS
NextCloud Plugin on TrueNAS