Day: November 4, 2021

Overlay Filesystem Basic

Overlay Filesystem Basic

Overlay filesystem merges lower and upper directories into merged directory.

Mount writable

mount -t overlay overlay -o lowerdir=/lower1:/lower2:/lower3,upperdir=/upper,workdir=/work /merged

The above example will have the order:


  • The lower directory can be read-only or could be an overlay itself.
  • The upper directory is normally writable.
  • The work directory is used to prepare files as they are switched between the layers, it needs to be an empty directory on the same filesystem mount as the upper directory.
  • All changes in the merged directory are still reflected in upper.
  • New files created in lower and upper will be shown in merged.
  • All files before opened directory, the content is still mapped according to layer.
  • All files after opened in merged directory, the content will not be reflected in merged directory.

Mount read-only

To mount as read only, no upper and work directory are required.

mount -t overlay overlay -o lowerdir=/lower1:/lower2 /merged

Whiteout files

Whiteout is to simulate a file removed from upper layer directory. It is created as a character device with 0/0 device number.

Opaque directories

Opaque is to simulate a directory removed from upper layer directory. It is made by setting the xattr “trusted.overlay.opaque” to “y”.


Overlay filesystem
Overlay Filesystem
Explaining OverlayFS – What it Does and How it Works

Placing fingers for FPS gaming

Placing fingers for FPS gaming

Frames per second (FPS) gaming requires muscle memory, fast response is not only needs for screen, but also for player, there is no time to move eyes from screen to hands at all.


Place fingers correctly while gaming can take advantage, such as fast response, less mistake, less stress, etc. Following placement just a guide for some games, such as genshin impact, etc.

  • Middle finger on W and alternate it between S
  • Index finger on D
  • Ring finger on A
  • Pinky finger on shift alternating to Ctrl when needing to crouch
  • Thumb finger over space alternating to C/X/Z for the commands

If you put keyboard flat, you may able to use the palm near to your pinky finger to hit control. In this case, you need to use Middle/Index/Ring to hit C/X/Z, anyway, thumb is not easy to reach them as well.


Most of buttons and sticks are very obvious, except bumpers and triggers.

  • Index finger on bumper
  • Middle finger on trigger

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

This is to move root file system of Raspberry Pi Ubuntu OS to iSCSI LUN and convert it to btrfs file system


  • Cheaper than using small SD card
  • Backup and restore easy, only need to backup 150MB boot partition on SD card
  • Able to perform snapshot at LUN level or OS level (btrfs)
  • Should be Faster


Install iscsi packages

apt install open-iscsi
systemctl enable open-iscsi
systemctl start open-iscsi
systemctl enable iscsid
systemctl start iscsid

Configure iscsi

Edit /etc/iscsi/initiatorname.iscsi, update following line


Note: The YOUR_INITIATOR_NAME is the iSCSI client name

Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.


Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • SD card backup

Partition /dev/sda1: vfat, 2GB
Partition /dev/sda2: for root filesystem

Create filesystems

mkfs.vfat /dev/sda1
mkfs.btrfs /dev/sda2

Identify UUID for root filesystem

blkid /dev/sda2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Duplicate files

mount /dev/sda2 /mnt
rsync -avhP --exclude /boot/firmware --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/firmware,mnt}

Modify /etc/fstab in LUN

Note: Don't modify the file in /etc, it will not be used during iSCSI boot.

vi /mnt/etc/fstab

Change root mounting to

UUID=<YOUR_DEV_UUID> / btrfs defaults 1 1

Modify /boot/firmware/cmdline.txt

First create a backup of this file

cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.sav

Change the content of /boot/firmware/cmdline.txt

Note: Beware of rootfstype=btrfs

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=btrfs elevator=deadline rootwait fixrtc ip=dhcp root=UUID=<YOUR_DEV_UUID> ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: Root partition can also be identified by label if assigned one, see next topic


umount /mnt

Use LABEL for root

Using label instead of UUID for root filesystem.

Assign a label

For mounted filesystem

sudo btrfs filesystem label <mountpoint> <newlabel>

For not mounted filesystem

sudo btrfs filesystem label <device> <newlabel>

Change /etc/fstab

LABEL=<newlabel> / btrfs defaults 1 1

Change /boot/fireware/cmdline.txt

... root=LABEL=<newlabel> ...

Use Static IP

To use static IP for iSCSI connection, the ip definition in cmdline.txt needs to be changed to


This will create an IP on interface eth0 as below.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ef:07:0f brd ff:ff:ff:ff:ff:ff
    inet brd scope global eth0
       valid_lft forever preferred_lft forever
    inet brd scope global secondary dynamic eth0
       valid_lft 43036sec preferred_lft 43036sec
    inet6 fe80::dea6:32ff:feef:70f/64 scope link
       valid_lft forever preferred_lft forever

One way to just maintain one IP is, disabling OS IP address.


Other cmdline.txt format

I tried following format, but failed. I think it maybe work if no partition in iSCSI LUN


Use UUID is better, because there is no need to worry about LUN id and partition as UUID is unique ideitifier.

To specfic UUID, following format can be used according to kernel parameters, but it doesn't work too.


I also tried rd.* format, such as rd.iscsi.initiator, etc., they are new format to replace old format ISCSI_INITIATOR, etc., but also not failed. I think the dracut.cmdline version used by ubuntu in raspberry pi is old. Maybe should try grub2 used in Fedora OS for raspberry pi, or uefi used in Windows 10.

Backup and restore using pre-backup data

Backup data into iSCSI LUN partition 1

First create vfat in iSCSI LUN as partition 1, then backup /boot/fireware data into that partition

mkfs.vfat /dev/sda1
mount /dev/sda1 /mnt
cp /boot/firmware/. /mnt
umount /mnt

Restore to an empty SD card

Create partition in new SSD as type c, which W95 FAT32 (LBA), with Boot flag.

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048 1050623 1048576  512M  c W95 FAT32 (LBA)

Format SD card and give label as system-boot, the LABEL is defined in /etc/fstab, it can be changed to UUID if needed.

mkfs.vfat -n system-boot /dev/sdb1
mount /dev/sdb1 /mnt
cp -a <backup_filesystem> /mnt
umount /mnt

Make sure system reported correct LABEL on the newly created vfat filesystem, using blkid command to verify.


iscsi_tcp missing

If following error occurred, install package linux-modules-extra.

libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='iscsi_tcp'

Please read post Missing iSCSI module in Ubuntu 20.10

Reboot error

If failed to boot, initram command prompt will appear. In this case, following commands can be used to recover back the booting.

mkdir /mnt
mount /dev/mmcblk0p1 /mnt
cd /mnt
cp cmdline.txt.sav cmdline.txt
cd /
umount /mnt

Note: the umount is very important, otherwise, the changes wouldn't be saved.

There are very minimum commands can be used, such as no vi. So let it boots into previous status, then troubleshooting from there.


dracut kernel command line options
Kernel command line parameters
introduction to boot time parameters of the Linux kernel
Raspberry Pi 4 UEFI Boot
RPi cmdline.txt
RPi config.txt
The config.txt file
Raspberry Pi iSCSI Root on Ubuntu 20.04
btrfs root filesystem on raspbian
[Howto] booting from iSCSI
Ubuntu Server 20.10 on Raspberry Pi 4: installation guide with USB Boot (no SD card) and full disk encryption (excluding /boot) using btrfs-inside-luks and auto-apt snapshots with Timeshift
Raspberry Pi 4 - Ubuntu 20.04 w/Btrfs root
dracut.cmdline(7) — Linux manual page