Tag: ubuntu

BerryBoot on Raspberry Pi

BerryBoot on Raspberry Pi

Raspberry Pi 2/3/4 requires SD card to be used for booting, which saves OS filesystems. There are some issues with SD card with OS boot.

  • Limited size
  • Hard to backup

I was thinking a solution to move root filesystem to iSCSI, but grub is not used in OS of Raspberry Pi. As the design of BerryBoot, it can install various OSes on iSCSI disk with multi-boot environment. So started testing it with Ubuntu OS.

Hardware

Although, BerryBoot declares supporting Raspberry Pi 1/2/3/4, I tried Pi 3B+, but very slow and hanging. End up used Pi 4 (8GB).

Boot up error

After installation completed, got following error. It is caused iSCSI LUN import requires time, at time of error reported, LUN had not detected by iSCSI driver.

iSCSI target does not have any partitions

The solution give is changing iscsi.sh file in boot partition (the SD card), adding loop for delay.

IP address and VNC

The BerryBoot interface used same IP as ubuntu OS, but it doesn't configure DNS. So after bootup, the ubuntu OS has no DNS setting, then manual update of file /etc/systemd/resolv.conf is required.

In order to remotely control BerryBoot, add following command to enable VNC. Then when booting error or like to select another OS, VNC viewer can be used to connect to BerryBoot.

Wired network headless installation

  • Append following at end of line in cmdline.txt (same line append)
... vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1

Wireless headless installation

  • Append following at end of line in cmdline.txt (same line append)
vncinstall ipv4=192.168.88.88/255.255.255.0/192.168.88.1/wlan0
  • Create a file called wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1

network={
    ssid="ssid-of-accesspoint"
    psk="wpa-password"
}

Update error

The most important task is update, which includes package update and kernel update

Package update

It is quite smooth during update, but complaining /boot doesn't exist. I checked the OS structure, there is a /boot folder, with one kernel, and a filesystem /media/user_id/boot, which saves SD contents.

Kernel update

Kernel update was totally failed with following error

Can't find /boot/vmlinuz- (see /tmp/flash-kernel-no-kernel-error.log)

The solution given by other is removing two packages

sudo apt remove u-boot-rpi:arm64
sudo apt remove flash-kernel

Note: According to internet, BerryBoot uses its own kernel, meaning it can not be updated using package update command.

do-release-update

This is a very strange action I took. The OS I installed via BerryBoot is Ubuntu 20.10 Desktop, but the OS still can perform do-release-update to same version. The even more strange thing is, after I upgrade, it still appear as can perform do-release-update, then I spent another upgrade, another few hours upgrade, then it can fininally upgraded.

I think the package was not at the correct release stage, but don't know why got 980+ packages to upgrade after first run. Felt like perform exactly the same action once more.

The output of uname after patched ubuntu screen as below

  • BerryBoot ubuntu
root@camel:~# uname -a
Linux camel 5.10.43v64 #1 SMP PREEMPT Tue Jun 15 00:52:48 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux
root@camel:~# 
  • Normal ubuntu
root@ubuntupi:/usr/lib# uname -a
Linux ubuntupi 5.13.0-1009-raspi #10-Ubuntu SMP PREEMPT Mon Oct 25 13:58:43 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntupi:/usr/lib# 

Filesystem structure

The BerryBoot ubuntu has different filesystem structure, the root filesystem is mapped to none, and no entry in /etc/fstab, I think the reason behind, is root filesystem is mounted via kernel, which is the iSCSI disk, and cannot be changed.

Output of df

root@camel:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
none            16646144 6422360   8396520  44% /
tmpfs            3957172       0   3957172   0% /dev/shm
tmpfs            1582872    1404   1581468   1% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs            3957172       0   3957172   0% /run/qemu
tmpfs             791432      72    791360   1% /run/user/126
tmpfs             791432      64    791368   1% /run/user/1000
root@camel:~# 

Output of mount

root@camel:~# mount | grep -w /
none on / type overlay (rw,relatime,lowerdir=/mnt/shared:/squashfs,upperdir=/mnt/data/Ubuntu_20.10_Desktop.img192,workdir=/mnt/data/Ubuntu_20.10_Desktop.img192.work,redirect_dir=on)
root@camel:~# 

The normal ubuntu installed has same structure of other distributions, using /etc/fstab for root filesystem as well. So the normal operations can be taken on root filesystem

Output of df

root@ubuntupi:/usr/lib# df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs             794548    12732    781816   2% /run
/dev/mmcblk0p2  30358480 24453720   4573672  85% /
tmpfs            3972736        0   3972736   0% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
tmpfs            3972736        0   3972736   0% /run/qemu
tmpfs            3972736     8364   3964372   1% /var/log
tmpfs            3972736        8   3972728   1% /tmp
tmpfs            3972736        0   3972736   0% /var/tmp
/dev/mmcblk0p1    258095    97201    160894  38% /boot/firmware
tmpfs             794544       92    794452   1% /run/user/126
tmpfs             794544       84    794460   1% /run/user/1000
root@ubuntupi:/usr/lib# 

Output of mount

root@ubuntupi:/usr/lib# mount | grep -w /
/dev/mmcblk0p2 on / type ext4 (rw,noatime,nobarrier)
root@ubuntupi:/usr/lib# 

Pros

Following advantages over normal ubuntu

  • Root filesystem type can be selected during installation. (Normal ubuntu on Raspberry Pi can not)
  • The iSCSI module is loaded by default.
    Note: Recent update of my normal ubuntu caused iSCSI module missing issue.
  • OS multiboot

Cons

Duo to BerryBoot has different filesystem structure and kernel, some diffculties would be facing later.

  • Specific Kernel
    • No kernel upgrade can cause package incompatible, for example, docker was mentioned by others
    • Auto rebuild kernel is almost impossible
  • Filesystem
    • Unable to verify root filesystem, scrubbing wasn't allowed during my testing
    • Unable to know root filesystem type even mounted
    • Cannot increase root filesystem size easily

Conclusion

The way BerryBoot manage the system is highly customized, which might not be adapted anywhere else. The customization also leads system compatibility issues later, and migration, backup, restore, etc., cannot be implemented in common way. So there could be no future-proof for this implemenation.

Compare local boot partition solution, which only holds root partition on iSCSI, the advantage of BerryBoot is iSCSI dependency issue reduced and multiboot OS capability. But the root on iSCSI uses bootload, which reduced OS compatibility issue.

My decision is still use normal bootload way for future systems to achieve maintennance free system.

References

Headless installation
Problems with update/upgrade Ubuntu 20.10 on Raspberry pi 4 with SSD
Raspberry Pi iSCSI Root on Ubuntu 20.04
How to Dual Boot a Raspberry Pi Using BerryBoot
BerryBoot v2.0 - bootloader / universal operating system installer
Storing your files on a Synology NAS (using iSCSI)

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

This is to move root file system of Raspberry Pi Ubuntu OS to iSCSI LUN and convert it to btrfs file system

Pros

  • Cheaper than using small SD card
  • Backup and restore easy, only need to backup 150MB boot partition on SD card
  • Able to perform snapshot at LUN level or OS level (btrfs)
  • Should be Faster

Steps

Install iscsi packages

apt install open-iscsi
systemctl enable open-iscsi
systemctl start open-iscsi
systemctl enable iscsid
systemctl start iscsid

Configure iscsi

Edit /etc/iscsi/initiatorname.iscsi, update following line

InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • SD card backup

Partition /dev/sda1: vfat, 2GB
Partition /dev/sda2: for root filesystem

Create filesystems

mkfs.vfat /dev/sda1
mkfs.btrfs /dev/sda2

Identify UUID for root filesystem

blkid /dev/sda2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Duplicate files

mount /dev/sda2 /mnt
rsync -avhP --exclude /boot/firmware --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/firmware,mnt}

Modify /etc/fstab in LUN

Note: Don't modify the file in /etc, it will not be used during iSCSI boot.

vi /mnt/etc/fstab

Change root mounting to

UUID=<YOUR_DEV_UUID> / btrfs defaults 1 1

Modify /boot/firmware/cmdline.txt

First create a backup of this file

cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.sav

Change the content of /boot/firmware/cmdline.txt

Note: Beware of rootfstype=btrfs

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=btrfs elevator=deadline rootwait fixrtc ip=dhcp root=UUID=<YOUR_DEV_UUID> ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: Root partition can also be identified by label if assigned one, see next topic

Reboot

umount /mnt
reboot

Use LABEL for root

Using label instead of UUID for root filesystem.

Assign a label

For mounted filesystem

sudo btrfs filesystem label <mountpoint> <newlabel>

For not mounted filesystem

sudo btrfs filesystem label <device> <newlabel>

Change /etc/fstab

LABEL=<newlabel> / btrfs defaults 1 1

Change /boot/fireware/cmdline.txt

... root=LABEL=<newlabel> ...

Use Static IP

To use static IP for iSCSI connection, the ip definition in cmdline.txt needs to be changed to

ip=192.168.1.200::192.168.1.1:255.255.255.0:rpi:eth0:off

This will create an IP 192.168.1.200 on interface eth0 as below.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ef:07:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.23/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 43036sec preferred_lft 43036sec
    inet6 fe80::dea6:32ff:feef:70f/64 scope link
       valid_lft forever preferred_lft forever

One way to just maintain one IP is, disabling OS IP address.

Reboot

Other cmdline.txt format

I tried following format, but failed. I think it maybe work if no partition in iSCSI LUN

root=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

Use UUID is better, because there is no need to worry about LUN id and partition as UUID is unique ideitifier.

To specfic UUID, following format can be used according to kernel parameters, but it doesn't work too.

root=UUID=<UUID>
netroot=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

I also tried rd.* format, such as rd.iscsi.initiator, etc., they are new format to replace old format ISCSI_INITIATOR, etc., but also not failed. I think the dracut.cmdline version used by ubuntu in raspberry pi is old. Maybe should try grub2 used in Fedora OS for raspberry pi, or uefi used in Windows 10.

Backup and restore using pre-backup data

Backup data into iSCSI LUN partition 1

First create vfat in iSCSI LUN as partition 1, then backup /boot/fireware data into that partition

mkfs.vfat /dev/sda1
mount /dev/sda1 /mnt
cp /boot/firmware/. /mnt
umount /mnt

Restore to an empty SD card

Create partition in new SSD as type c, which W95 FAT32 (LBA), with Boot flag.

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048 1050623 1048576  512M  c W95 FAT32 (LBA)

Format SD card and give label as system-boot, the LABEL is defined in /etc/fstab, it can be changed to UUID if needed.

mkfs.vfat -n system-boot /dev/sdb1
mount /dev/sdb1 /mnt
cp -a <backup_filesystem> /mnt
umount /mnt

Make sure system reported correct LABEL on the newly created vfat filesystem, using blkid command to verify.

Troubleshooting

iscsi_tcp missing

If following error occurred, install package linux-modules-extra.

libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='iscsi_tcp'

Please read post Missing iSCSI module in Ubuntu 20.10

Reboot error

If failed to boot, initram command prompt will appear. In this case, following commands can be used to recover back the booting.

mkdir /mnt
mount /dev/mmcblk0p1 /mnt
cd /mnt
cp cmdline.txt.sav cmdline.txt
cd /
umount /mnt
reboot

Note: the umount is very important, otherwise, the changes wouldn't be saved.

There are very minimum commands can be used, such as no vi. So let it boots into previous status, then troubleshooting from there.

References

dracut kernel command line options
Kernel command line parameters
introduction to boot time parameters of the Linux kernel
Raspberry Pi 4 UEFI Boot
RPi cmdline.txt
RPi config.txt
kernel-parameters.txt
The config.txt file
Raspberry Pi iSCSI Root on Ubuntu 20.04
btrfs root filesystem on raspbian
[Howto] booting from iSCSI
Ubuntu Server 20.10 on Raspberry Pi 4: installation guide with USB Boot (no SD card) and full disk encryption (excluding /boot) using btrfs-inside-luks and auto-apt snapshots with Timeshift
Raspberry Pi 4 - Ubuntu 20.04 w/Btrfs root
dracut.cmdline(7) — Linux manual page

Change user name in ubuntu

Change user name in ubuntu

To change default user name ubuntu created in Raspberry Pi image, without adding a temporary user or enable root id, can be done using background job.

Commands

Change user name and home directory

usermod -l newUsername -d /home/newUsername oldUsername

Change group name

groupmod -n <newgroup> <oldgroup>

Job

The actual execution should be triggered as background job. I used screen command as below

  1. Login as ubuntu (default id)
  2. Run sudo -i to root
  3. Run screen to launch screen session
  4. Run following command, and press Ctrl-A d detach screen session immediately, without command executed.
newname=<new_user_name>
sleep 30; ps -ef | grep ^ubuntu | awk '{print $2}' | xargs kill -9 ; usermod -l $newname -d /home/$newname ubuntu; groupmod -n $newname ubuntu; mv /home/ubuntu /home/$newname
  1. Quit from console or ssh session
  2. After 30 seconds, login to the system using newname if succeeded. Othewise, use ubuntu login again.
  3. Then sudo to root and run screen -r to check if error or quit screen session.

References

How do I change my username?

Add bluetooth device from ubuntu console

Add bluetooth device from ubuntu console

I used following steps to add bluetooth keyboard.

Steps

  • Run bluetoothctl, then get following prompt
[bluetooth]# 
  • Run following commands to initialize bluetooth
power on
agent on
default-agent
scan on
  • Find the bluetooth device mac address

  • Run following command to connect to it

trust XX:XX:XX:XX:XX:XX
pair XX:XX:XX:XX:XX:XX
connect XX:XX:XX:XX:XX:XX
  • Then disable scan and quit
scan off
exit

References

How to connect bluetooth headset via command line on ubuntu 12.04

Verify package using debsums

Verify package using debsums

Verify every installed package

debsums

Verify every installed package (including configuration files).

debsums -a

Verify installed packages and report errors only

debsums -s

Verify every installed package and report changed files only

debsums -c

Verify every installed package (including configuration files) and report changed files only.

debsums -ca

Verify every installed package and report changed configuration files only.

sudo debsums -ce

Verify specific package

debsums -a bash

Create mismatch list

dpkg-query -S $(sudo debsums -c 2>&1 | sed -e "s/.*file \(.*\) (.*/\1/g") | cut -d: -f1 | sort -u

To reinstall them

apt-get install --reinstall <package name>

References

How to verify installed packages

Ubuntu grub-efi-amd64-signed error after do-release-upgrade

Ubuntu grub-efi-amd64-signed error after do-release-upgrade

Following error occurred whenever run apt upgrade after perform do-release-upgrade

dpkg: error processing package grub-efi-amd64-signed (–configure):
installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32

Solution

Reinstall all grub group packages using following commands

sudo apt-get purge grub\*
sudo apt-get install grub-efi
sudo apt-get autoremove
sudo update-grub

Remove ubuntu zfs snapshots

Remove ubuntu zfs snapshots

There are so many snapshots when using zfs in ubuntu.

Issue

When tried to do release update, got following error

# do-release-update
...
...
Not enough free disk space 

The upgrade has aborted. The upgrade needs a total of 256 M free 
space on disk '/boot'. Please free at least an additional 91.4 M of 
disk space on '/boot'. You can remove old kernels using 'sudo apt 
autoremove' and you could also set COMPRESS=xz in 
/etc/initramfs-tools/initramfs.conf to reduce the size of your 
initramfs. 
...

This error messsage was occurred many times before, but those systems had very small /boot partition or many old kernels kept. If it is the first case, total repartitioning and moving root filesystem are required.

Space on /boot

Examing disk space for bpool, found that zfs reported 675MB used in bpool, but actual usage is only 242MB.

root@ubuntu:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   960M   675M   285M        -         -    30%    70%  1.00x    ONLINE  -
rpool  17.5G  7.99G  9.51G        -         -    21%    45%  1.00x    ONLINE  -
root@ubuntu:~# zfs list bpool
NAME    USED  AVAIL     REFER  MOUNTPOINT
bpool   675M   157M       96K  /boot
root@ubuntu:~# du -cshx /boot
242M    /boot
242M    total
root@ubuntu:~# 

Then found many snapshots both in bpool and data pool

root@ubuntu:~# zfs list -t snapshot | head
NAME                                                               USED  AVAIL     REFER  MOUNTPOINT
bpool/BOOT/ubuntu_e8m8h0@autozsys_ywm1ok                             0B      -      238M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_ms74md                             0B      -      238M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_ugu9z7                            80K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_r3xqau                            72K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_nkagbh                             0B      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_xdbwsy                             0B      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_zrt7vi                            72K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_jbmnwk                            72K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_0e5p2e                            64K      -      242M  -
root@ubuntu:~# 
root@ubuntu:~# zfs list -t snapshot | wc
    301    1505   27701

Too many! Not sure how many snapshots ubuntu likes to create

Removing snapshots

List all snapshots for /boot

root@ubuntu:~# df /boot
Filesystem               1K-blocks   Used Available Use% Mounted on
bpool/BOOT/ubuntu_e8m8h0    408192 247808    160384  61% /boot
root@ubuntu:~# zfs list -H -o name -t snapshot bpool/BOOT/ubuntu_e8m8h0
bpool/BOOT/ubuntu_e8m8h0@autozsys_ywm1ok
bpool/BOOT/ubuntu_e8m8h0@autozsys_ms74md
bpool/BOOT/ubuntu_e8m8h0@autozsys_ugu9z7
bpool/BOOT/ubuntu_e8m8h0@autozsys_r3xqau
bpool/BOOT/ubuntu_e8m8h0@autozsys_nkagbh
bpool/BOOT/ubuntu_e8m8h0@autozsys_xdbwsy
bpool/BOOT/ubuntu_e8m8h0@autozsys_zrt7vi
bpool/BOOT/ubuntu_e8m8h0@autozsys_jbmnwk
bpool/BOOT/ubuntu_e8m8h0@autozsys_0e5p2e
bpool/BOOT/ubuntu_e8m8h0@autozsys_b17dwn
bpool/BOOT/ubuntu_e8m8h0@autozsys_uad1rb
bpool/BOOT/ubuntu_e8m8h0@autozsys_mxhvc9
bpool/BOOT/ubuntu_e8m8h0@autozsys_9athz8
bpool/BOOT/ubuntu_e8m8h0@autozsys_61umv1
bpool/BOOT/ubuntu_e8m8h0@autozsys_1q65cz
root@ubuntu:~# 

Then remove them

zfs list -H -o name -t snapshot bpool/BOOT/ubuntu_e8m8h0 | xargs -n 1 zfs destroy

Now, it is ok to upgrade

root@ubuntu:~# zfs list -o space bpool
NAME   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
bpool   589M   243M        0B     96K             0B       243M
root@ubuntu:~# 

Docker Compose – wordpress

Docker Compose - wordpress

Simple steps to start using docker compose to create wordpress dockers.

Installation

Install docker-compose package

Run following command on ubuntu and armbian servers.

apt install docker-compose

Create dockers

Create folder as project name wp

The project name will be used as a part of docker container name.

mkdir -p /app/wp

Create docker compose file

Using vi to create file docker-compose.yml in directory /app/wp

version: "3.3"

services:
  db:
    image: mariadb:latest
    volumes:
      - db_data:/var/lib/mysql
    ports:
      - "3306:3306"
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress

  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    volumes:
      - wordpress_data:/var/www/html
    ports:
      - "80:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_NAME: wordpress
volumes:
  db_data: {}
  wordpress_data: {}

Run docker compose command

docker-compose up -d

Destroy dockers

Run docker compose command

docker-compose down

Destroy dockers and their volumes

docker-compose down --volumes