BerryBoot on Raspberry Pi
Raspberry Pi 2/3/4 requires SD card to be used for booting, which saves OS filesystems. There are some issues with SD card with OS boot.
- Limited size
- Hard to backup
I was thinking a solution to move root filesystem to iSCSI, but grub is not used in OS of Raspberry Pi. As the design of BerryBoot, it can install various OSes on iSCSI disk with multi-boot environment. So started testing it with Ubuntu OS.
Hardware
Although, BerryBoot declares supporting Raspberry Pi 1/2/3/4, I tried Pi 3B+, but very slow and hanging. End up used Pi 4 (8GB).
Boot up error
After installation completed, got following error. It is caused iSCSI LUN import requires time, at time of error reported, LUN had not detected by iSCSI driver.
iSCSI target does not have any partitions
The solution give is changing iscsi.sh file in boot partition (the SD card), adding loop for delay.
IP address and VNC
The BerryBoot interface used same IP as ubuntu OS, but it doesn't configure DNS. So after bootup, the ubuntu OS has no DNS setting, then manual update of file /etc/systemd/resolv.conf is required.
In order to remotely control BerryBoot, add following command to enable VNC. Then when booting error or like to select another OS, VNC viewer can be used to connect to BerryBoot.
Wired network headless installation
- Append following at end of line in
cmdline.txt (same line append)
... vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1
Wireless headless installation
- Append following at end of line in
cmdline.txt (same line append)
vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1/wlan0
- Create a file
called wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1
network={
ssid="ssid-of-accesspoint"
psk="wpa-password"
}
Update error
The most important task is update, which includes package update and kernel update
Package update
It is quite smooth during update, but complaining /boot doesn't exist. I checked the OS structure, there is a /boot folder, with one kernel, and a filesystem /media/user_id/boot, which saves SD contents.
Kernel update
Kernel update was totally failed with following error
Can't find /boot/vmlinuz- (see /tmp/flash-kernel-no-kernel-error.log)
The solution given by other is removing two packages
sudo apt remove u-boot-rpi:arm64
sudo apt remove flash-kernel
Note: According to internet, BerryBoot uses its own kernel, meaning it can not be updated using package update command.
do-release-update
This is a very strange action I took. The OS I installed via BerryBoot is Ubuntu 20.10 Desktop, but the OS still can perform do-release-update to same version. The even more strange thing is, after I upgrade, it still appear as can perform do-release-update, then I spent another upgrade, another few hours upgrade, then it can fininally upgraded.
I think the package was not at the correct release stage, but don't know why got 980+ packages to upgrade after first run. Felt like perform exactly the same action once more.
The output of uname after patched ubuntu screen as below
root@camel:~# uname -a
Linux camel 5.10.43v64 #1 SMP PREEMPT Tue Jun 15 00:52:48 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux
root@camel:~#
root@ubuntupi:/usr/lib# uname -a
Linux ubuntupi 5.13.0-1009-raspi #10-Ubuntu SMP PREEMPT Mon Oct 25 13:58:43 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntupi:/usr/lib#
Filesystem structure
The BerryBoot ubuntu has different filesystem structure, the root filesystem is mapped to none, and no entry in /etc/fstab, I think the reason behind, is root filesystem is mounted via kernel, which is the iSCSI disk, and cannot be changed.
Output of df
root@camel:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
none 16646144 6422360 8396520 44% /
tmpfs 3957172 0 3957172 0% /dev/shm
tmpfs 1582872 1404 1581468 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 3957172 0 3957172 0% /run/qemu
tmpfs 791432 72 791360 1% /run/user/126
tmpfs 791432 64 791368 1% /run/user/1000
root@camel:~#
Output of mount
root@camel:~# mount | grep -w /
none on / type overlay (rw,relatime,lowerdir=/mnt/shared:/squashfs,upperdir=/mnt/data/Ubuntu_20.10_Desktop.img192,workdir=/mnt/data/Ubuntu_20.10_Desktop.img192.work,redirect_dir=on)
root@camel:~#
The normal ubuntu installed has same structure of other distributions, using /etc/fstab for root filesystem as well. So the normal operations can be taken on root filesystem
Output of df
root@ubuntupi:/usr/lib# df
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 794548 12732 781816 2% /run
/dev/mmcblk0p2 30358480 24453720 4573672 85% /
tmpfs 3972736 0 3972736 0% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 3972736 0 3972736 0% /run/qemu
tmpfs 3972736 8364 3964372 1% /var/log
tmpfs 3972736 8 3972728 1% /tmp
tmpfs 3972736 0 3972736 0% /var/tmp
/dev/mmcblk0p1 258095 97201 160894 38% /boot/firmware
tmpfs 794544 92 794452 1% /run/user/126
tmpfs 794544 84 794460 1% /run/user/1000
root@ubuntupi:/usr/lib#
Output of mount
root@ubuntupi:/usr/lib# mount | grep -w /
/dev/mmcblk0p2 on / type ext4 (rw,noatime,nobarrier)
root@ubuntupi:/usr/lib#
Pros
Following advantages over normal ubuntu
- Root filesystem type can be selected during installation. (Normal ubuntu on Raspberry Pi can not)
- The iSCSI module is loaded by default.
Note: Recent update of my normal ubuntu caused iSCSI module missing issue.
- OS multiboot
Cons
Duo to BerryBoot has different filesystem structure and kernel, some diffculties would be facing later.
- Specific Kernel
- No kernel upgrade can cause package incompatible, for example, docker was mentioned by others
- Auto rebuild kernel is almost impossible
- Filesystem
- Unable to verify root filesystem, scrubbing wasn't allowed during my testing
- Unable to know root filesystem type even mounted
- Cannot increase root filesystem size easily
Conclusion
The way BerryBoot manage the system is highly customized, which might not be adapted anywhere else. The customization also leads system compatibility issues later, and migration, backup, restore, etc., cannot be implemented in common way. So there could be no future-proof for this implemenation.
Compare local boot partition solution, which only holds root partition on iSCSI, the advantage of BerryBoot is iSCSI dependency issue reduced and multiboot OS capability. But the root on iSCSI uses bootload, which reduced OS compatibility issue.
My decision is still use normal bootload way for future systems to achieve maintennance free system.
References
Headless installation
Problems with update/upgrade Ubuntu 20.10 on Raspberry pi 4 with SSD
Raspberry Pi iSCSI Root on Ubuntu 20.04
How to Dual Boot a Raspberry Pi Using BerryBoot
BerryBoot v2.0 - bootloader / universal operating system installer
Storing your files on a Synology NAS (using iSCSI)