BerryBoot on Raspberry Pi

Raspberry Pi 2/3/4 requires SD card to be used for booting, which saves OS filesystems. There are some issues with SD card with OS boot.

• Limited size
• Hard to backup

I was thinking a solution to move root filesystem to iSCSI, but grub is not used in OS of Raspberry Pi. As the design of BerryBoot, it can install various OSes on iSCSI disk with multi-boot environment. So started testing it with Ubuntu OS.

Hardware

Although, BerryBoot declares supporting Raspberry Pi 1/2/3/4, I tried Pi 3B+, but very slow and hanging. End up used Pi 4 (8GB).

Boot up error

After installation completed, got following error. It is caused iSCSI LUN import requires time, at time of error reported, LUN had not detected by iSCSI driver.

iSCSI target does not have any partitions

The solution give is changing iscsi.sh file in boot partition (the SD card), adding loop for delay.

IP address and VNC

The BerryBoot interface used same IP as ubuntu OS, but it doesn't configure DNS. So after bootup, the ubuntu OS has no DNS setting, then manual update of file /etc/systemd/resolv.conf is required.

In order to remotely control BerryBoot, add following command to enable VNC. Then when booting error or like to select another OS, VNC viewer can be used to connect to BerryBoot.

Wired network headless installation

• Append following at end of line in cmdline.txt (same line append)

... vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1

Wireless headless installation

• Append following at end of line in cmdline.txt (same line append)

vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1/wlan0

• Create a file called wpa_supplicant.conf

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1

network={
    ssid="ssid-of-accesspoint"
    psk="wpa-password"
}

Update error

The most important task is update, which includes package update and kernel update

Package update

It is quite smooth during update, but complaining /boot doesn't exist. I checked the OS structure, there is a /boot folder, with one kernel, and a filesystem /media/user_id/boot, which saves SD contents.

Kernel update

Kernel update was totally failed with following error

Can't find /boot/vmlinuz- (see /tmp/flash-kernel-no-kernel-error.log)

The solution given by other is removing two packages

sudo apt remove u-boot-rpi:arm64
sudo apt remove flash-kernel

Note: According to internet, BerryBoot uses its own kernel, meaning it can not be updated using package update command.

do-release-update

This is a very strange action I took. The OS I installed via BerryBoot is Ubuntu 20.10 Desktop, but the OS still can perform do-release-update to same version. The even more strange thing is, after I upgrade, it still appear as can perform do-release-update, then I spent another upgrade, another few hours upgrade, then it can fininally upgraded.

I think the package was not at the correct release stage, but don't know why got 980+ packages to upgrade after first run. Felt like perform exactly the same action once more.

The output of uname after patched ubuntu screen as below

• BerryBoot ubuntu

root@camel:~# uname -a
Linux camel 5.10.43v64 #1 SMP PREEMPT Tue Jun 15 00:52:48 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux
root@camel:~# 

• Normal ubuntu

root@ubuntupi:/usr/lib# uname -a
Linux ubuntupi 5.13.0-1009-raspi #10-Ubuntu SMP PREEMPT Mon Oct 25 13:58:43 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntupi:/usr/lib# 

Filesystem structure

The BerryBoot ubuntu has different filesystem structure, the root filesystem is mapped to none, and no entry in /etc/fstab, I think the reason behind, is root filesystem is mounted via kernel, which is the iSCSI disk, and cannot be changed.

Output of df

root@camel:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
none            16646144 6422360   8396520  44% /
tmpfs            3957172       0   3957172   0% /dev/shm
tmpfs            1582872    1404   1581468   1% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs            3957172       0   3957172   0% /run/qemu
tmpfs             791432      72    791360   1% /run/user/126
tmpfs             791432      64    791368   1% /run/user/1000
root@camel:~# 

Output of mount

root@camel:~# mount | grep -w /
none on / type overlay (rw,relatime,lowerdir=/mnt/shared:/squashfs,upperdir=/mnt/data/Ubuntu_20.10_Desktop.img192,workdir=/mnt/data/Ubuntu_20.10_Desktop.img192.work,redirect_dir=on)
root@camel:~# 

The normal ubuntu installed has same structure of other distributions, using /etc/fstab for root filesystem as well. So the normal operations can be taken on root filesystem

Output of df

root@ubuntupi:/usr/lib# df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs             794548    12732    781816   2% /run
/dev/mmcblk0p2  30358480 24453720   4573672  85% /
tmpfs            3972736        0   3972736   0% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
tmpfs            3972736        0   3972736   0% /run/qemu
tmpfs            3972736     8364   3964372   1% /var/log
tmpfs            3972736        8   3972728   1% /tmp
tmpfs            3972736        0   3972736   0% /var/tmp
/dev/mmcblk0p1    258095    97201    160894  38% /boot/firmware
tmpfs             794544       92    794452   1% /run/user/126
tmpfs             794544       84    794460   1% /run/user/1000
root@ubuntupi:/usr/lib# 

Output of mount

root@ubuntupi:/usr/lib# mount | grep -w /
/dev/mmcblk0p2 on / type ext4 (rw,noatime,nobarrier)
root@ubuntupi:/usr/lib# 

Pros

Following advantages over normal ubuntu

• Root filesystem type can be selected during installation. (Normal ubuntu on Raspberry Pi can not)
• The iSCSI module is loaded by default.
  Note: Recent update of my normal ubuntu caused iSCSI module missing issue.
• OS multiboot

Cons

Duo to BerryBoot has different filesystem structure and kernel, some diffculties would be facing later.

• Specific Kernel
  • No kernel upgrade can cause package incompatible, for example, docker was mentioned by others
  • Auto rebuild kernel is almost impossible
• Filesystem
  • Unable to verify root filesystem, scrubbing wasn't allowed during my testing
  • Unable to know root filesystem type even mounted
  • Cannot increase root filesystem size easily

Conclusion

The way BerryBoot manage the system is highly customized, which might not be adapted anywhere else. The customization also leads system compatibility issues later, and migration, backup, restore, etc., cannot be implemented in common way. So there could be no future-proof for this implemenation.

Compare local boot partition solution, which only holds root partition on iSCSI, the advantage of BerryBoot is iSCSI dependency issue reduced and multiboot OS capability. But the root on iSCSI uses bootload, which reduced OS compatibility issue.

My decision is still use normal bootload way for future systems to achieve maintennance free system.

References

Headless installation
Problems with update/upgrade Ubuntu 20.10 on Raspberry pi 4 with SSD
Raspberry Pi iSCSI Root on Ubuntu 20.04
How to Dual Boot a Raspberry Pi Using BerryBoot
BerryBoot v2.0 - bootloader / universal operating system installer
Storing your files on a Synology NAS (using iSCSI)