Blog
Swim Stars 2022 – Failed
WordPress Error – bypass `reCaptcha v3 test failed`
WordPress Error - bypass reCaptcha v3 test failed
When encounter error when login to WordPress, can temporarily bypass reCaptcha verification
reCaptcha v3 test failed
Solution
Rename folder wp-content/plugins/google-captcha
to wp-content/plugins/google-captcha.old
, then try to login again.
To enable back, rename back the folder.
References
Enable HTTPS for NextCloud Docker using `docker-compose`
Table of Contents
Enable HTTPS for NextCloud Docker using docker-compose
By default, NextCloud Docker doesn't enable HTTPS.
Steps
Create
- Create folders
/app/nextcloud/data/db
: for MySQL database/app/nextcloud/data/db_conf
: for MySQL database configuration (don't see anything in it)/app/nextcloud/data/cert
: for nextcloud certificates/app/nextcloud/data/html
: for nextcloud data and packages
- Create
docker-compose.yml
- Create
Dockerfile.nextcloud
- Run
docker-compose build
- Run
docker-compose up -d
Destroy
- Run
docker-compose down
Docker Compose
docker-compose.yml
:
version: '3'
services:
db:
image: mariadb:latest
restart: always
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
volumes:
- /app/nextcloud/data/db:/var/lib/mysql
- /app/nextcloud/data/db_conf:/etc/mysql/conf.d
environment:
- MYSQL_ROOT_PASSWORD=<mysql_root_password>
- MYSQL_PASSWORD=<mysql_user_password>
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
build:
context: .
dockerfile: Dockerfile.nextcloud
restart: always
ports:
- 80:80
- 443:443
links:
- db
volumes:
- /app/nextcloud/data/html:/var/www/html
- /app/nextcloud/data/cert/fullchain.pem:/etc/ssl/certs/ssl-cert-snakeoil.pem
- /app/nextcloud/data/cert/privkey.pem:/etc/ssl/private/ssl-cert-snakeoil.key
environment:
- MYSQL_PASSWORD=<mysql_user_password>
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_HOST=db
Dockerfile.nextcloud
- Define base docker image as
nextcloud:latest
- Enable
ssl
module - Enable
default-ssl
site
FROM nextcloud:latest
RUN a2enmod ssl
RUN a2ensite default-ssl
References
Unable to login after set admin password when setting up NextCloud
Unable to login after set admin password when setting up
After installed NextCloud using NextCloud Docker, unable to login using newly created admin password.
Solution
Use Google Chrome to login, it doesn't work in Firefox and Microsoft Edge. After fully completed setup, Firefox works again.
Disable Windows Defender Realtime Scanning
Disable Windows Defender Realtime Scanning
Windows Defender Realtime Scanning takes lots of CPU time, to disable it, disable following items together.
Real-time Protection
Tamper Protection
References
Ubuntu iSCSI connect to multiple targets on boot
Ubuntu iSCSI connect to multiple targets on boot
Using iSCSI grub parameters can login to one iSCSI target at booting time, but for more than one targets, auto iSCSI session login might not be possible, at least I haven't found a way to do that.
In order to login to other multiple targets, modification need to be done in iSCSI discovered configuration
Login on boot
After discover the iSCSI targets, the targets will be in /etc/iscsi/nodes
folder. Then found the correct target which in the format as below.
/etc/iscsi/nodes/<IQN>/<IP>,<Port>,1/default
Modify the above file and change following two parameters
...
node.startup = automatic
...
node.conn[0].startup = onboot
...
Ubuntu with UEFI iSCSI root on x86_64
Table of Contents
Ubuntu with UEFI iSCSI root on x86_64
If I'm not wrong, the old ubuntu server can be installed directly on iSCSI disk for MBR type of PC. But I like to convert a Core 2 Due MacBook Pro to Ubuntu server, which only has UEFI and can not boot up into MBR.
Note: This is just my observation. Maybe I'm wrong.
Preparation
- Ubuntu 20.04 installation USB drive
- 16GB USB drive for OS installation
Steps
Install OS
Partition the USB drive into the same format as Fedora root on iSCSI which I had done before.
- EFI partition, 512MB, vfat, on USB drive
- /boot, btrfs, 1GB, on USB drive
- /, btrfs, on iSCSI LUN
Note: I separated /boot and /, because I need to move / into iSCSI LUN, and I'm also not sure UEFI can configure iSCSI to detect iSCSI LUN before grub find out /boot partition. I tried UEFI in raspberry pi, which can configure iSCSI, but I didn't see such menu in MacBook Pro.
OS structure
After installed, the system structured as below
- UEFI - /boot/efi/EFI
# find /boot/efi -ls
1 4 drwxr-xr-x 3 root root 4096 Jan 1 1970 /boot/efi
4 4 drwxr-xr-x 4 root root 4096 Nov 9 23:31 /boot/efi/EFI
7 4 drwxr-xr-x 2 root root 4096 Nov 9 23:31 /boot/efi/EFI/BOOT
36 936 -rwxr-xr-x 1 root root 955656 Nov 10 10:46 /boot/efi/EFI/BOOT/BOOTX64.EFI
37 84 -rwxr-xr-x 1 root root 85672 Nov 10 10:46 /boot/efi/EFI/BOOT/fbx64.efi
38 840 -rwxr-xr-x 1 root root 856232 Nov 10 10:46 /boot/efi/EFI/BOOT/mmx64.efi
11 4 drwxr-xr-x 2 root root 4096 Nov 10 08:56 /boot/efi/EFI/ubuntu
44 4 -rwxr-xr-x 1 root root 108 Nov 10 10:46 /boot/efi/EFI/ubuntu/BOOTX64.CSV
45 4 -rwxr-xr-x 1 root root 121 Nov 10 10:46 /boot/efi/EFI/ubuntu/grub.cfg
46 1696 -rwxr-xr-x 1 root root 1734528 Nov 10 10:46 /boot/efi/EFI/ubuntu/grubx64.efi
47 840 -rwxr-xr-x 1 root root 856232 Nov 10 10:46 /boot/efi/EFI/ubuntu/mmx64.efi
48 936 -rwxr-xr-x 1 root root 955656 Nov 10 10:46 /boot/efi/EFI/ubuntu/shimx64.efi
In above list, there following two files are important
The file /boot/efi/EFI/ubuntu/BOOTX64.CSV
has following content
shimx64.efi,ubuntu,,This is the boot entry for ubuntu
The EFI grub configuration file, /boot/efi/EFI/ubuntu/grub.cfg
has following content, which contains uuid of boot partition and location info, and it is named as root hd3,gpt2
search.fs_uuid 812cce04-3b56-4e17-8e38-b325304293f2 root hd3,gpt2
set prefix=($root)'/grub'
configfile $prefix/grub.cfg
Note: Although the USB device location number is changing depending on the sequence of device detection, but the uuid will never be changed. Here, names it as hd3,gpt2
, is because boot partition was the gpt partition 2 on 3rd device /dev/sdd2
. The device name hd3
doesn't need to be the real device location, but it is only the reference to be used in ubuntu grab configuration later.
- Ubuntu boot directory -
/boot
This directory includes kernel files and grub configuration file.
-rw------- 1 root root 4755119 Oct 15 17:56 System.map-5.4.0-90-generic
-rw-r--r-- 1 root root 237884 Oct 15 17:56 config-5.4.0-90-generic
drwxr-xr-x 3 root root 4096 Jan 1 1970 efi
drwxr-xr-x 1 root root 82 Nov 10 08:54 grub
lrwxrwxrwx 1 root root 27 Nov 9 23:29 initrd.img -> initrd.img-5.4.0-90-generic
-rw-r--r-- 1 root root 84224544 Nov 10 02:51 initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root 27 Nov 9 23:29 initrd.img.old -> initrd.img-5.4.0-90-generic
lrwxrwxrwx 1 root root 24 Nov 9 23:29 vmlinuz -> vmlinuz-5.4.0-90-generic
-rw------- 1 root root 11780352 Oct 15 19:36 vmlinuz-5.4.0-90-generic
lrwxrwxrwx 1 root root 24 Nov 9 23:29 vmlinuz.old -> vmlinuz-5.4.0-90-generic
- Ubuntu grub -
/boot/grub/grub.cfg
This is the grub configuration for ubuntu boot, the importent parts are, ip configuration, iscsi configuration, and turn off screen configuration.
linux /vmlinuz-5.4.0-90-generic root=UUID=<YOUR_DEV_UUID> ro ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30
Note: This file generated using /etc/default/grub
, I'm not sure how to change root to label based, and there is option ro
, which conflicts with rw
that I was given in /etc/default/grub
.
- Ubuntu grub parameter file -
/etc/default/grub
This file is used to build actual /boot/grub/grub.cfg
.
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="ip=dhcp ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw consoleblank=30"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL=console
By default, nothing disappers on screen when booting up, uncomment the GRUB_TERMINAL=console
to fix the issue.
Note: I prefer grub menu, it can be used to edit kernel parameter when anything went wrong. Troubleshooting can be easier during kernel upgrade.
Two kinds of commands use this file to update grub configuration
update-grub
orupdate-grub2
Note: update-grub2
is a soft link of update-grub
This is to update /boot/grub/grub.cfg
using /etc/default/grub
file.
Some people reported issue that update-grub
or update-grub2
which used this file to build grub configuration, doesn't update /boot/efi/EFI/ubuntu/grub.cfg
file.
dpkg-reconfigure grub-efi-amd64
This command will update both grub configuration file /boot/grub/grub.cfg
and EFI grub configuration file /boot/efi/EFI/ubuntu/grub.cfg
.
Configure iSCSI service
The service was installed by default in ubuntu server 20.04.
- Enable iscsid service
systemctl enable iscsid
- Configure
/etc/iscsi/initiatorname.iscsi
InitiatorName=<YOUR_INITIATOR_NAME>
- Configure
/etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
- Start iscsid service
After iscsid configuration changed, restart iscsid service is required.
systemctl restart iscsid
- Discover and Login
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login
Note: If can not login, restart iscsid
and try again.
Identify block device
Use lsblk
command to identify device file, it should be something like /dev/sdX
.
Partitioning
Creating two partitions using fdisk, the first partition is to prepare following for future used, such as
- Network boot
- UEFI iSCSI boot
- USB device backup
Partition /dev/sdd1: vfat, 1GB, for /boot
filesystem
Partition /dev/sdd2: for root filesystem
Note: The iSCSI LUN appears as /dev/sdd
Format iSCSI LUN
Format /dev/sdd1
as vfat and /dev/sdd2
as btrfs
mkfs.vfat /dev/sdd1
mkfs.btrfs /dev/sdd2
Update initramfs
This is to enable ubuntu load iscsi driver during boot
touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c
Note: Verifying iscsi module in updating list is important
Update grub
Ubuntu grub parameter file /etc/default/grub
as listed in previous section. Beware of iSCSI parameters.
Update both /boot/grub/grub.cfg
and /boot/efi/EFI/ubuntu/grub.cfg
dpkg-reconfigure grub-efi-amd64
Test Reboot
This is the first time test reboot, the outcome should be
- No hanging issue
- New iSCSI disks can be found after reboot without running
iscsiadm
command manually. Verify usinglsblk
command.
The objective of this reboot is to test iSCSI module and finding out any misconfiguration for grub.
Duplicate files
Duplicate files to iSCSI LUN
mount /dev/sdd2 /mnt
mkdir /mnt/boot
mount /dev/sdd1 /mnt/boot
rsync -avhP --exclude /boot/efi --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/efi,mnt}
Change root partition in grub
Use UUID
Identify UUID for new root filesystem
blkid /dev/sdd2
Replace root device definitions as root=UUID=<UUID>
in /boot/grub/grub.cfg
using block id found
Use LABEL
Assign LABEL to new root filesystem
btrfs fi label /mnt ROOT
Replace root device definitions as root=LABEL=ROOT
in /boot/grub/grub.cfg
.
Update /etc/fstab
Replace root filesystem (/) uuid using the found in previous section (in iSCSI LUN), or if LABEL is assigned, then following line can be used.
LABEL=ROOT / btrfs defaults 0 1
Test Reboot
The reboot is to test root partition switchs to iSCSI LUN. Verify using df
command.
/dev/sdc2 15727596 5091296 10317712 34% /
...
/dev/sda2 1048576 111400 819256 12% /boot
/dev/sda1 523248 5356 517892 2% /boot/efi
Now, the root (/) is in different device as /boot
and /boot/efi
.
Update grub again
Run dpkg-reconfigure grub-efi-amd64
again, then reboot the system. This is to verify all configuration are correct.
Test Reboot
After reboot, the system should have expected setup, which archives
- The root (/) partition is in iSCSI LUN
- Filesystems are structured according to
/boot/efi/EFI/ubuntu/grub.cfg
and/etc/fstab
.
Clone to smaller USB drive
Due to both EFI and /boot
partitions are all small partition, and they are only needed during boot up, a smaller and slower USB drive can be used.
The outcome is also testing the root filesystem fully moved.
Create partition
- EFI partition, 512MB, vfat, on USB drive
- /boot, btrfs, 1GB, on USB drive
Create filesystem
/dev/sdc1
is EFI partition, and /dev/sdc2
is /boot
partition
mkfs.vfat /dev/sdc1
mkfs.btrfs /dev/sdc2
Deplicate files
mount /dev/sdc2 /mnt
mkdir /mnt/efi
mount /dev/sdc1 /mnt/efi
rsync -avhP /boot/ /mnt/
Umount filesystems
umount /mnt/efi
umount /mnt
umount /boot/efi
umount /boot
Update /etc/fstab
Edit /etc/fstab
and update UUID for both /boot
and /boot/efi
as below
/dev/disk/by-uuid/812cce04-3b56-4e17-8e38-b325304293f2 /boot btrfs defaults 0 1
/dev/disk/by-uuid/6B77-6F14 /boot/efi vfat defaults 0 1
Mount filesystem
This is also to confirm /etc/fstab
is correct.
mount -a
Update grub
Run following command, and verify grub settings, include /boot/grub/grub.cfg
, /boot/efi/EFI/ubuntu/grub.cfg
.
dpkg-reconfigure grub-efi-amd64
Final test boot
Shutdown system and remove original USB, after that power on device. Make sure everything are expected after system boot up.
Other considerations
Disable iSCSI logout
The early iSCSI logout, can cause BTRFS filesystem closing issue, especially on root (/
) filesystem. Disable iSCSI logout during service stop.
systemctl edit --full open-iscsi.service
Comment out following line
#ExecStop=/lib/open-iscsi/logout-all.sh
Use fix IP address for iSCSI
Following IP configuration can be used to configure fix IP (192.168.1.51) in kernel
ip=192.168.1.51::192.168.1.254:255.255.255.0:fish:enp0s10::192.168.1.250::
If the IP address is different than OS, and they are using same interface, then there will be two IP entries for same interface, for example,
2: enp0s10: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:26:4a:18:82:c6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.51/24 brd 192.168.1.255 scope global enp0s10
valid_lft forever preferred_lft forever
inet 192.168.1.9/24 brd 192.168.1.255 scope global secondary dynamic enp0s10
valid_lft 43172sec preferred_lft 43172sec
inet6 fe80::226:4aff:fe18:82c6/64 scope link
valid_lft forever preferred_lft forever
With two IP address, the dedicated iSCSI IP range can be setup if required. Furthermore, dedicated network interface/LAN can be used for iSCSI network.
Future considerations
Missing iSCSI module
A recent issue I encountered for ubuntu `do-release-upgrade`, is missing `iscsi_tcp.ko` module, and the package `linux-modules-extra` is not in apt list. The apt dependency doesn't include linux-modules-extra, but the package can be downloaded from apt repository, manual installation is required.
- Update:
The latest missing package can be installed using following command.
apt install linux-modules-extra-raspi
Backup USB device
Because USB devices are not mirror, backing up is required, and restoration is essential. The first partition in LUN can be used for backup.
Boot partition on iSCSI
If UEFI supports iSCSI, the boot device should able to be in iSCSI LUN as well. The advantage is, boot device also can be part of LUN snapshot for backup. But three stages involve iSCSI connection maybe having issues.
- UEFI iSCSI connection
- Grub iSCSI connection
- OS iSCSI connection
In fact, the OS doesn't need iSCSI connection if no additional iSCSI targets required except the one connected by Grub, because the LUNs connected in Grub are represented as local disks.
Multiboot
If move Boot partition to iSCSI LUN, then left one small configuration in EFI partition, such as UUID for boot device, etc. If setting up multiboot in EFI is possible, then the same USB device can be used for different boot devices.
Grub in Fedora way
In Fedora, the format of kernel parameters are different, and IP address and it's bridge configuration can be done in following way. If can be done in same way as Fedora, the IP address of iSCSI can be fixed, no need to be dhcp, and iSCSI definition can be shorter.
GRUB_CMDLINE_LINUX="netroot=iscsi::@::3260:: rd.iscsi.initiator=YOUR_INITIATOR_NAME rhgb quiet ip=192.168.1.9::192.168.1.254:255.255.255.0::br0:off nameserver=192.168.1.250 ifname=enp0s10:00:26:4a:18:82:c6 bridge=br0:enp0s10"
Prior to start
I had tried a few times to install ubuntu server 20.04 on MacBook Pro on iSCSI, but failed. I also tried Fedora 34, and it was successfully installed root partition on iSCSI LUN.
For Fedora 34, there are 3 partitions,
- EFI partition, 512MB, vfat, on USB drive
- /boot, btrfs, 2GB, on USB drive
- /, btrfs, on iSCSI LUN
It is using GRUB2.
After observation, I think I'm able to do the same for ubuntu.
In fact, I prefer ubuntu, because do-release-upgrade
is doing well for ubuntu. For Fedora, I had done version upgrade many years ago too, but very manual and it is not official supported. Meaning one day, the upgrade method can never work again.
Troubleshooting
grub configuration error
Such as iSCSI configuration was given wrongly, due to no grub menu, the USB drive needs to be connected to another linux system to modify. An ubuntu VM is handy in this case.
References
Convert Raspberry Pi Ubuntu to iSCSI btrfs root
How to Configure the GRUB2 Boot Loader’s Settings
The kernel’s command-line parameters
update-grub does not update /boot/efi/EFI/ubuntu/grub.cfg
Missing iSCSI module in Ubuntu 20.10
Table of Contents
Missing iSCSI module in Ubuntu 20.10
Recently, I have upgraded Ubuntu 20.04 to 20.10, then iscsi_tcp module is missing, which caused iscsi LUN could not be loaded.
I checked /var/log/syslog, showed zram (reported by init-zram-swapping) and autofs (reported by containerd) module were missing too.
Error
Looking at error message shown during iscsi discovery, the command was looking for file iscsi_tcp.ko. Which can not be found in the system.
Update
The latest missing package can be installed using following command.
apt install linux-modules-extra-raspi
apt search
When performing apt search, shows three packages should include such file, but when listing all files in those packages, there is no iscsi_tcp.ko file at all.
# apt search iscsi_tcp.ko
Sorting... Done
Full Text Search... Done
libopeniscsiusr/impish,now 2.1.4-0ubuntu1 arm64 [installed]
iSCSI userspace library
libopeniscsiusr-dev/impish 2.1.4-0ubuntu1 all
iSCSI userspace library headers
open-iscsi/impish,now 2.1.4-0ubuntu1 arm64 [installed]
iSCSI initiator tools
Reinstall OS
Then I decided to install a new fresh OS, but got the same result.
Search in ubuntu website
When searching packages content file iscsi_tcp.ko
from ubuntu website, found there is no module package called linux-modules-extra
for kernel 5.13.0-1009-raspi, the latest is 5.13.0-1008-raspi, which is for 20.04.
In other words, 20.10 doesn't have iscsi_tcp module, at all.
# apt install linux-modules-extra
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-modules-extra
#
Search in ubuntu repo
Then I go to ubuntu packages repo site below
http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/
...
/lib/modules/5.13.0-1008-oracle/kernel/drivers/scsi/libiscsi_tcp.ko linux-modules-5.13.0-1008-oracle [arm64, amd64]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/iscsi_tcp.ko linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/libiscsi_tcp.ko linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/iscsi_tcp.ko linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/libiscsi_tcp.ko linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-19-generic-64k/kernel/drivers/scsi/iscsi_tcp.ko linux-modules-5.13.0-19-generic-64k [arm64]
...
found that there are linux-modules-extra package for both 1009 and 1010.
...
[ ] linux-modules-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb 2021-10-29 10:29 23M
[ ] linux-modules-extra-5.13.0-1008-raspi-nolpae_5.13.0-1008.9_armhf.deb 2021-09-29 20:44 21M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_arm64.deb 2021-09-29 20:43 23M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_armhf.deb 2021-09-29 20:44 21M
[ ] linux-modules-extra-5.13.0-1009-raspi-nolpae_5.13.0-1009.10_armhf.deb 2021-10-28 08:33 21M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb 2021-10-28 08:33 23M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_armhf.deb 2021-10-28 08:33 21M
[ ] linux-modules-extra-5.13.0-1010-raspi-nolpae_5.13.0-1010.11_armhf.deb 2021-10-29 10:29 21M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_arm64.deb 2021-10-29 10:29 23M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb 2021-10-29 10:29 21M
[ ] linux-raspi-headers-5.4.0-1008_5.4.0-1008.8_arm64.deb 2020-04-10 11:18 11M
...
Install linux-modules-extra
Then I decided to do download the package and check the contents
wget http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb
dpkg -c linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb | grep iscsi
found the iscsi_tcp.ko is in the package
Install package via dpkg
dpkg -i linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb
Testing
Tested using iscsiadm, shows the problem fixed
iscsiadm --mode discovery --op update --type sendtargets --portal 192.168.1.17
In fact, this also fixed some other similar issues, such as, zram model missing, etc.
Reconfigure btrfs filesystem
In this system, the btrfs filesystem which was mirrored to iscsi LUN using bcache, the iscsi LUN was removed using following command during troubleshooting
btrfs balance start -f -sconvert=single -mconvert=single -dconvert=single /app
btrfs device remove /dev/bcache0 /app
Now use following command to reconfigure it back.
btrfs device add -f /dev/bcache0 /app
btrfs balance start -dconvert=raid1 -mconvert=raid1 /app
Configure module auto-loading
In order to load modules automatically, add iscsi_tcp
line in /etc/modules
. This may also avoid apt autoremove
command removes linux-modules-extra
package.
Possible cause
I think the issue was caused by spliting the kernel module package in to two in 1008, because linux-modules-extra only exists in the version 1008 in package searching list. I think ubnutu forgot this when releasing 1009. At this time, apt hasn't auto updated system to 1010.
References
Ubuntu Raspberry Package Repo
Ubuntu Package Searching
How to list files of a Debian package without install
How to install specific Ubuntu packages, with exact version?
Bcache Basic
Table of Contents
Bcache Basic
Update
After finished first round of writing, I think it needs to be described in clearer way.
During first time setup, I got very confused by the document I was using, and didn't understand fully. After observation of the devices behavior, I think I will review this document later.
Install
apt install bcache-tools
Concept
Bcache adds in one more layer between the actual filesystem and the block device (partition, raid, etc.) which filesystem located in. This is done by relocating filesystem header behind bcache header in block device, which offset the ordinary filesystem header and data 8KiB behind.
Note: Use partition as example
Ordinary filesystem partition = (Ordinary filesystem header + Ordinary filesystem data)
bcache partition = bcache header (8KiB) + (Ordinary filesystem header + Ordinary filesystem data)
-------------------------------------------------------
bcache data
In such case, the device (partition) is represented as a bcache partition, and bcache driver creates a new device called /dev/bcacheX without bcache header, then OS will detect /dev/bcacheX as ordinary filesystem.
This method is widely used in disk encryption as well, which allows encryption driver translates encrypted data device to OS in a newly created device.
Mount a filesystem bypass bcache header
By using following method, the ordinary filesystem in bcache partition can be mounted by ordinary filesystem driver without bcache driver.
- Create Loopback Device
losetup -o 8192 /dev/loop0 /dev/[BCACHE DEVICE]
- Mount Device
mount /dev/loop0 -o loop /mnt/[LOCATION]
Devices
Backing Device
Backing device is the actual device holding data, for example, /dev/sdb1
, it can be 10TB hard disk, disk raid, software raid, etc. It can be created by following command
make-bcache -B /dev/sdb1
It will create a bcache device as well, such as /dev/bcache0
Bcache device
Bcache device is the device created together with backing device (/dev/sdb1
), such as /dev/bcache0
, all ordinary filesystem operation will be operated on bcache device (/dev/bcache0
), such as mkfs, etc.
Caching Device
Caching device is the temporary device used as cache, for example /dev/sdc1
, it can be 128GB SSD. It can be created using following command
make-bcache -C /dev/sdc1
It has cset.uuid as below
# bcache-super-show /dev/sdc1 | grep cset
cset.uuid f0e01318-f4fd-4fab-abbb-d76d870503ec
Before recreate caching device, use following command clean up the device header.
wipefs -a /dev/<device>
Attach
Attach action allows caching device starts working for backing device.
# echo <caching_device_uuid> > /sys/block/<bcache_device>/bcache/attach
echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/bcache0/bcache/attach
# or to the backing device
echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/sdb/sdb1/bcache/attach
Note: If following error occurred, and no output when ran bcache-status
command, then run partprobe
command to rescan partition tables
# bcache-status
#
# echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/bcache0/bcache/attach
-bash: echo: write error: No such file or directory
# partprobe
*Note: bcache-status
is a free bcache tool can be downloaded from github.
Operation on backing device and bcache device
The backing device and the bcache device have 1-to-1 relationship, because they are created at same time using one make-bcache -C
command.
In fact, both bcache
folders in both bcache and backing device are the same.
# ls -Hdi /sys/block/sdb/sdb1/bcache
64954 /sys/block/sdb/sdb1/bcache
# ls -Hdi /sys/block/bcache0/bcache
64954 /sys/block/bcache0/bcache
I think the correct thinking should be attaching to bcache device, but the bcache folder is created under backing device.
# readlink -f /sys/block/bcache0/bcache
/sys/devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sdb/sdb1/bcache
I think this is because that bcache device is a virtual device created during backing device (real device) creation, so the actual device structure should assign to real device (backing device).
With or without caching device
Without caching device, bcache driver will directly translate ordinary filesystem driver read/write into bcache device read/write.
With cache device, bcache driver will utilize caching device before backing device operation.
ordinary filesystem operation => bcache driver => bcache filesystem operation => backing device
||
||
caching device
So, without caching device, bcache is still operating correctly.
Attach caching device to multiple bcache devices
One caching device can support multiple bcache devices as below
# echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/bcache0/bcache/attach
# echo 4b05ce02-19f4-4cc6-8ca0-1f765671ceda > /sys/block/bcache1/bcache/attach
# echo 4b05ce02-19f4-4cc6-8ca0-1f765671ceda > /sys/block/bcache2/bcache/attach
# echo 4b05ce02-19f4-4cc6-8ca0-1f765671ceda > /sys/block/bcache3/bcache/attach
# echo 75ff0598-7624-46f6-bcac-c27a3cf1a09f > /sys/block/bcache4/bcache/attach
A bcache device has only one caching device
The uuid of caching device which attached to bcache device can be found as below.
# ls -la /sys/block/<device>/bcache/cache
lrwxrwxrwx 1 root root 0 Jun 19 18:42 /sys/block/<device>/bcache/set -> ../../../../../../../../fs/bcache/<UUID>
Detach
To detach a caching device, needs to send 1
or cache-set-uuid to bcache device or backing device
Safely remove the caching device from bcache device
echo cache-set-uuid > /sys/block/bcache0/bcache/detach
# or
echo cache-set-uuid > /sys/block/sdb/sdb1/bcache/detach
Detach the caching device from bcache device
echo 1 > /sys/block/bcache0/bcache/detach
# or
echo 1 > /sys/block/sdb/sdb1/bcache/detach
Stop
bcache/backing device
Stop a bcache device, is the same as stop the backing device.
echo 1 > /sys/block/bcache0/bcache/stop
# or
echo 1 > /sys/block/sdb/sdb1/bcache/stop
After stopped bcache/backing device,
- The
/sys/block/sdb/sdb1/bcache
folder disappears - The
/sys/block/bcache0
virtual device disappers - But no impact to caching device, and still registered in
/sys/fs/bcache/<uuid>
.
caching device
Stop caching device, will impact all caching, bcache and backing devices
echo 1 > /sys/fs/bcache/cache-set-uuid/stop
After stopped caching device, all bcache setup related to that caching device disapper
- The
/sys/block/sdb/sdb1/bcache
folder disappers if/dev/sdb1
is backing device attached - The bcache devices
/sys/block/bcache0
disappers if/dev/bcache0
is the bcache device attached - The caching device
/sys/fs/bcache/<uuid>
disappers
To resume
The first way to resume whole setup, can be done by run partprobe
.
The second way is using register to resume device one by one.
Register
In fact, registering is needed every bootup, but attaching only have to be done once.
Register is required if the caching or backing device missing during system start up or they are stopped manually.
Register the backing device as below
echo /dev/sdb1 > /sys/fs/bcache/register # backing device
After registered, the system will
- Creates
/sys/block/sdb/sdb1/bcache
folder - If didn't attach caching device before stopped or caching device had been registered, then creates
/sys/block/bcache0
virtual device
If /sys/block/bcache0
is not created due to missing caching device,
- The
/sys/block/bcache0
device will be created after register missing caching device - Or use following command to force to create
/sys/block/bcache0
and start running
echo 1 > /sys/block/sdb/sdb1/bcache/running
Warning: If force start, all write cache in caching device will be lost, this can cause filesystem corruption
Register caching device
To register caching device, following command can be used
echo /dev/sdc1 > /sys/fs/bcache/register # caching device
It will create directory /sys/fs/bcache/<uuid>
folder.
If attached backing device had been registered, the /sys/block/bcache0
will be created and running.
Ordinary filesystem operation
All ordinary filesystem operations will be operated on bcache device (/dev/bcache0
), for example
mkfs.btrfs /dev/bcache0
mount /dev/bcache0 /mnt
...
Caching state
The caching state can be viewed using following command
cat /sys/block/bcache0/bcache/state
Output:
- no cache: this means you have not attached a caching device to your backing bcache device
- clean: this means everything is ok. The cache is clean.
- dirty: this means everything is setup fine and that you have enabled writeback and that the cache is dirty.
- inconsistent: you are in trouble because the backing device is not in sync with the caching device
Caching mode
There are 4 caching modes, writethrough
, writeback
, writearound
, and none
.
echo writeback > /sys/block/bcache0/bcache/cache_mode
Show caching device info
bcache-super-show /dev/sdXY
Writeback Percent
echo 100 > /sys/block/bcache0/bcache/writeback_percent
Dirty data
How much data in cache has not written into backing device.
cat /sys/block/sda/sda3/bcache/dirty_data
Flush cache to backing device
This might be required if filesystem maintenance needed.
Run following command to disable writeback mode
echo writethrough > /sys/block/bcache0/bcache/cache_mode
Wait until state reports "clean"
watch cat /sys/block/bcache0/bcache/state
Force flush of cache to backing device
echo 0 > /sys/block/bcache0/bcache/writeback_percent
Errors
The /sys/fs/bcache/
folder does not exist
The bcache
module was not loaded.
sh: echo: write error: Invalid argument
If dmesg shows
bcache: bch_cached_dev_attach() Couldn't attach sdc: block size less than set's block size
Then the --block 4k parameter was not set on either device and defaults can mismatch.
Otherwise, the device might already be attached.
sh: echo: write error: No such file or directory
The UUID is not a valid cache.
Other considerations
Boot from bcache device
Grub2 does not offer support for bcache, but it is fully supported by UEFI. Check the following link for details
https://wiki.archlinux.org/title/Bcache