Author: Bian Xi

Script to delete old docker images

Posted on by Bian Xi Leave a Comment on Script to delete old docker images

Script to delete old docker images

To schedule a task to delete old docker images.

Assumption

Assuming there is a local copy of image normally. If not, need to directly operate in registry server or pull image locally before execution.

Prereq

Following information should be set as environment variable.

user=username
password=password
registry=registry.example.net
repo=repo

List images

List by image date

This is for local images, list all images' digests created 30 days ago.

docker images $registry/$repo --digests --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Digest}} {{.CreatedAt}}" | awk '{if ($5 < strftime("%Y-%m-%d", systime()-3600*24*30)) {print $1,$2,$3,$4}}'

List by tags

List all tags in format of YYYYDDMM older than 30 days.

TAG_LIST=`curl -s https://$user:$password@$registry/v2/$repo/tags/list | jq '.tags[] | select(. < (now-3600*24*30 | strftime("%Y%m%d")) )'`

Note: strflocaltime can be used if your jq supports it.

Query remote manifest by tag

Using docker command

docker manifest inspect $registry/$repo:$tag -v | jq '.Descriptor.digest'

or using rest api

curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://username:password@registry.example.net/v2/${repo}/manifests/20210624 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

Delete tags

Delete from registry

curl -X DELETE https://$username:$password@$registry/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete from local

Delete image using image id.

docker rmi $image_id

Script

#!/bin/bash

. ./set_env.sh

user=USERNAME
password=$PASSWORD

repo=${1:-repo_name}
registry=REGISTRY_NAME
keep_days=30

## List all docker images

docker images $registry/$repo --digests --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Digest}} {{.CreatedAt}}" | awk -v keep_days=$keep_days '{if ($5 < strftime("%Y-%m-%d", systime()-3600*24*keep_days)) {print $1,$2,$3,$4}}' | while read line
do
        read repo_path tag id manifest <<< $line

        echo Deleting from registry: $line
        curl -X DELETE https://$user:$password@$registry/v2/${repo}/manifests/$manifest

        echo Deleting from local: $line
        docker rmi $id
done

## List all docker images with tags in registry server
curl -s https://$user:$password@$registry/v2/$repo/tags/list | jq '.tags[] | select(. < (now-3600*24*30 | strftime("%Y%m%d")) )' | tr -d '"' | while read tag
do
        echo Deleting from registry: $tag
        manifest=`docker manifest inspect $registry/$repo:$tag -v | jq '.Descriptor.digest' | tr -d '"'`
        curl -X DELETE https://$user:$password@$registry/v2/${repo}/manifests/$manifest
done

References

Docker Registry HTTP API V2
Can I get an image digest without downloading the image?

JFrog artifactory open-source edition

Posted on by Bian Xi Leave a Comment on JFrog artifactory open-source edition

JFrog artifactory open-source edition

JFrog artifactory contains many feature, such as git, docker, etc.

Download URL

https://jfrog.com/open-source/#artifactory

Docker

Docker image can be pulled via docker command

docker pull releases-docker.jfrog.io/jfrog/artifactory-oss:latest

Docker compose can be found in download url as well.

Clean up docker registry

Posted on by Bian Xi Leave a Comment on Clean up docker registry

Clean up docker registry

After docker containers are backed up daily to docker registry, clean up process is required.

Note: If the image without tag, and if it can be found client locately, then can run command docker inspect to find out. Otherwise, can check "Run command from server" section find out.

List _catalog (repositries)

Following curl command can be used to display all _catalog (repositries).

curl https://username:password@registry.example.net/v2/_catalog

List images (tags) for each repositry

Following curl command can be used to display tags for specific repo

curl https://username:password@registry.example.net/v2/${repo}/tags/list

Get Digest

Using pull request

Use pull image command to get digest as below.

docker pull registry.example.net/${repo}:20210624
20210624: Pulling from user/host/dnsmasq
...
22b5d63ad977: Already exists
8e2e66517d7e: Pull complete
Digest: sha256:7535af1f65524f9200b901fc31b9c779819e45c0502ef99605666842a319908f

Get digest when deleting local image

Using delete local image action also can get digest as well

docker rmi registry.example.net/${repo}:20210619
# Untagged: registry.example.net/user/host/dnsmasq:20210619
# Untagged: registry.example.net/user/host/dnsmasq@sha256:e300ff463dc18c7b3bf3964dc5a9832f613d829285a0da49e5fd37519dc7d0fc
# Deleted: sha256:35baba3d5948b5844b67adcd6a236905039e929f8647d4e4afc9e64e9460d557
# Deleted: sha256:bd681f3956f55dc028bae7ca4c2657457824a0e356c59705302fb084660a669b

The Digest is the second tag.

Note: The first deleted sha256 allowed to be deleted too, but not the second deleted. Don't understand why

Run command from client

Run following command to get Digest

curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://username:password@registry.example.net/v2/${repo}/manifests/20210624 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

Run command from server

This is usefull if there is no image at client and no tag for specfic image.

docker exec privateregistry_registry_1 bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml | grep '${repo}: marking manifest' | awk '{print $4}'

Check manifests exists or not

curl https://username:password@registry.example.net/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete tag

curl -X DELETE https://username:password@registry.example.net/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete _catalog

The v2 registry doesn't allow deleting only certain tags from an image, and deletion of tags is in an open PR for a future version of the Registry (https://github.com/docker/distribution/pull/2169).

rm -rf docker/registry/v2/repositories/${repo}/

*Note: After delete, garbage-collect, restart, the repo is still reporting out as empty _catalog.

Run garbage-collect

Login to registry server and run following command

docker exec registry bin/registry garbage-collect --delete-untagged /etc/docker/registry/config.yml

If the repo is still listed in the output, need to delete them too.

Restart registry if necessary

docker restart registry

References

Docker Private Registry - Deleted all images, but still showing in catalog
Clean Up Your Docker Registry
Delete repository from v2 private registry
Pull Request - New Tags API #2169
Docker Registry HTTP API V2

GPU via M.2

Posted on by Bian Xi Leave a Comment on GPU via M.2

GPU via M.2

Product

ATD-Link R43SG

Supports

Morefine S500+

Where to buy

https://www.aliexpress.com/item/1005003279448856.html?spm=a2g0o.productlist.0.0.7cffa20dWbpXKF&algo_pvid=3274993e-85b9-4c9a-9b2f-40609e37fa3d&algo_exp_id=3274993e-85b9-4c9a-9b2f-40609e37fa3d-3&pdp_ext_f=%7B%22sku_id%22%3A%2212000024996792414%22%7D

Review

Morefine S500+ Using RTX 3080 TI

M.2 PCIe x4 to External eGPU - Faster than TB3 eGPU's

ADT-link R43SG M.2 PCIe 3.0 x4 / NVMe to external GPU review. Cards tested GTX 1060 6GB & GTX 1080 Ti. Where to buy: http://s.click.aliexpress.com/e/INILpRw

8750H mini PC with 2 x M.2 NMVe slots: http://s.click.aliexpress.com/e/bpWvyIlE

Create LABEL for filesystems in Linux

Posted on by Bian Xi Leave a Comment on Create LABEL for filesystems in Linux

Create LABEL for filesystems in Linux

LABEL is used in /etc/fstab and system boot, such as cmdline.txt and grub, to allow the mounting identify the filesystem..

In order to allow system recognize the LABEL, blkid needs to show the filesystem has the LABEL.

Assign LABEL during filesystem creation

vfat

mkfs.vfat -n "label" /dev/XXX

ext4

mkfs.ext4 -L "label" /dev/XXX

btrfs

mkfs.btrfs -L "label" /dev/XXX

Change LABEL

LABEL can be changed after filesystem created.

swap

swaplabel -L "new label" /dev/XXX using util-linux

ext2/3/4

e2label /dev/XXX "new label" using e2fsprogs

btrfs

btrfs filesystem label /dev/XXX "new label" using btrfs-progs

reiserfs

reiserfstune -l "new label" /dev/XXX using reiserfsprogs

jfs

jfs_tune -L "new label" /dev/XXX using jfsutils

xfs

xfs_admin -L "new label" /dev/XXX using xfsprogs

fat/vfat

fatlabel /dev/XXX "new label" using dosfstools
mlabel -i /dev/XXX ::"new label" using mtools

exfat

tune.exfat -L "new label" /dev/XXX using exfatprogs
exfatlabel /dev/XXX "new label" using exfatprogs or exfat-utils

ntfs

ntfslabel /dev/XXX "new label" using ntfs-3g

udf

udflabel /dev/XXX "new label" using udftools

crypto_LUKS (LUKS2 only)

cryptsetup config --label="new label" /dev/XXX using cryptsetup

References

Persistent block device naming
How do I change the "label" reported by lsblk? [duplicate]

Overlay Filesystem Basic

Posted on by Bian Xi Leave a Comment on Overlay Filesystem Basic

Overlay Filesystem Basic

Overlay filesystem merges lower and upper directories into merged directory.

Mount writable

mount -t overlay overlay -o lowerdir=/lower1:/lower2:/lower3,upperdir=/upper,workdir=/work /merged

The above example will have the order:

/upper
/lower1
/lower2
/lower3

  • The lower directory can be read-only or could be an overlay itself.
  • The upper directory is normally writable.
  • The work directory is used to prepare files as they are switched between the layers, it needs to be an empty directory on the same filesystem mount as the upper directory.
  • All changes in the merged directory are still reflected in upper.
  • New files created in lower and upper will be shown in merged.
  • All files before opened directory, the content is still mapped according to layer.
  • All files after opened in merged directory, the content will not be reflected in merged directory.

Mount read-only

To mount as read only, no upper and work directory are required.

mount -t overlay overlay -o lowerdir=/lower1:/lower2 /merged

Whiteout files

Whiteout is to simulate a file removed from upper layer directory. It is created as a character device with 0/0 device number.

Opaque directories

Opaque is to simulate a directory removed from upper layer directory. It is made by setting the xattr “trusted.overlay.opaque” to “y”.

References

Overlay filesystem
Overlay Filesystem
Explaining OverlayFS – What it Does and How it Works

Placing fingers for FPS gaming

Posted on by Bian Xi Leave a Comment on Placing fingers for FPS gaming

Placing fingers for FPS gaming

Frames per second (FPS) gaming requires muscle memory, fast response is not only needs for screen, but also for player, there is no time to move eyes from screen to hands at all.

Keyboard

Place fingers correctly while gaming can take advantage, such as fast response, less mistake, less stress, etc. Following placement just a guide for some games, such as genshin impact, etc.

  • Middle finger on W and alternate it between S
  • Index finger on D
  • Ring finger on A
  • Pinky finger on shift alternating to Ctrl when needing to crouch
  • Thumb finger over space alternating to C/X/Z for the commands

If you put keyboard flat, you may able to use the palm near to your pinky finger to hit control. In this case, you need to use Middle/Index/Ring to hit C/X/Z, anyway, thumb is not easy to reach them as well.

Controller

Most of buttons and sticks are very obvious, except bumpers and triggers.

  • Index finger on bumper
  • Middle finger on trigger

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Posted on by Bian Xi Leave a Comment on Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

This is to move root file system of Raspberry Pi Ubuntu OS to iSCSI LUN and convert it to btrfs file system

Pros

  • Cheaper than using small SD card
  • Backup and restore easy, only need to backup 150MB boot partition on SD card
  • Able to perform snapshot at LUN level or OS level (btrfs)
  • Should be Faster

Steps

Install iscsi packages

apt install open-iscsi
systemctl enable open-iscsi
systemctl start open-iscsi
systemctl enable iscsid
systemctl start iscsid

Configure iscsi

Edit /etc/iscsi/initiatorname.iscsi, update following line

InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • SD card backup

Partition /dev/sda1: vfat, 2GB
Partition /dev/sda2: for root filesystem

Create filesystems

mkfs.vfat /dev/sda1
mkfs.btrfs /dev/sda2

Identify UUID for root filesystem

blkid /dev/sda2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Duplicate files

mount /dev/sda2 /mnt
rsync -avhP --exclude /boot/firmware --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/firmware,mnt}

Modify /etc/fstab in LUN

Note: Don't modify the file in /etc, it will not be used during iSCSI boot.

vi /mnt/etc/fstab

Change root mounting to

UUID=<YOUR_DEV_UUID> / btrfs defaults 1 1

Modify /boot/firmware/cmdline.txt

First create a backup of this file

cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.sav

Change the content of /boot/firmware/cmdline.txt

Note: Beware of rootfstype=btrfs

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=btrfs elevator=deadline rootwait fixrtc ip=dhcp root=UUID=<YOUR_DEV_UUID> ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: Root partition can also be identified by label if assigned one, see next topic

Reboot

umount /mnt
reboot

Use LABEL for root

Using label instead of UUID for root filesystem.

Assign a label

For mounted filesystem

sudo btrfs filesystem label <mountpoint> <newlabel>

For not mounted filesystem

sudo btrfs filesystem label <device> <newlabel>

Change /etc/fstab

LABEL=<newlabel> / btrfs defaults 1 1

Change /boot/fireware/cmdline.txt

... root=LABEL=<newlabel> ...

Use Static IP

To use static IP for iSCSI connection, the ip definition in cmdline.txt needs to be changed to

ip=192.168.1.200::192.168.1.1:255.255.255.0:rpi:eth0:off

This will create an IP 192.168.1.200 on interface eth0 as below.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ef:07:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.23/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 43036sec preferred_lft 43036sec
    inet6 fe80::dea6:32ff:feef:70f/64 scope link
       valid_lft forever preferred_lft forever

One way to just maintain one IP is, disabling OS IP address.

Reboot

Other cmdline.txt format

I tried following format, but failed. I think it maybe work if no partition in iSCSI LUN

root=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

Use UUID is better, because there is no need to worry about LUN id and partition as UUID is unique ideitifier.

To specfic UUID, following format can be used according to kernel parameters, but it doesn't work too.

root=UUID=<UUID>
netroot=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

I also tried rd.* format, such as rd.iscsi.initiator, etc., they are new format to replace old format ISCSI_INITIATOR, etc., but also not failed. I think the dracut.cmdline version used by ubuntu in raspberry pi is old. Maybe should try grub2 used in Fedora OS for raspberry pi, or uefi used in Windows 10.

Backup and restore using pre-backup data

Backup data into iSCSI LUN partition 1

First create vfat in iSCSI LUN as partition 1, then backup /boot/fireware data into that partition

mkfs.vfat /dev/sda1
mount /dev/sda1 /mnt
cp /boot/firmware/. /mnt
umount /mnt

Restore to an empty SD card

Create partition in new SSD as type c, which W95 FAT32 (LBA), with Boot flag.

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048 1050623 1048576  512M  c W95 FAT32 (LBA)

Format SD card and give label as system-boot, the LABEL is defined in /etc/fstab, it can be changed to UUID if needed.

mkfs.vfat -n system-boot /dev/sdb1
mount /dev/sdb1 /mnt
cp -a <backup_filesystem> /mnt
umount /mnt

Make sure system reported correct LABEL on the newly created vfat filesystem, using blkid command to verify.

Troubleshooting

iscsi_tcp missing

If following error occurred, install package linux-modules-extra.

libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='iscsi_tcp'

Please read post Missing iSCSI module in Ubuntu 20.10

Reboot error

If failed to boot, initram command prompt will appear. In this case, following commands can be used to recover back the booting.

mkdir /mnt
mount /dev/mmcblk0p1 /mnt
cd /mnt
cp cmdline.txt.sav cmdline.txt
cd /
umount /mnt
reboot

Note: the umount is very important, otherwise, the changes wouldn't be saved.

There are very minimum commands can be used, such as no vi. So let it boots into previous status, then troubleshooting from there.

References

dracut kernel command line options
Kernel command line parameters
introduction to boot time parameters of the Linux kernel
Raspberry Pi 4 UEFI Boot
RPi cmdline.txt
RPi config.txt
kernel-parameters.txt
The config.txt file
Raspberry Pi iSCSI Root on Ubuntu 20.04
btrfs root filesystem on raspbian
[Howto] booting from iSCSI
Ubuntu Server 20.10 on Raspberry Pi 4: installation guide with USB Boot (no SD card) and full disk encryption (excluding /boot) using btrfs-inside-luks and auto-apt snapshots with Timeshift
Raspberry Pi 4 - Ubuntu 20.04 w/Btrfs root
dracut.cmdline(7) — Linux manual page

Bcache Basic

Posted on by Bian Xi Leave a Comment on Bcache Basic

Bcache Basic

Update

After finished first round of writing, I think it needs to be described in clearer way.

During first time setup, I got very confused by the document I was using, and didn't understand fully. After observation of the devices behavior, I think I will review this document later.

Install

apt install bcache-tools

Concept

Bcache adds in one more layer between the actual filesystem and the block device (partition, raid, etc.) which filesystem located in. This is done by relocating filesystem header behind bcache header in block device, which offset the ordinary filesystem header and data 8KiB behind.

Note: Use partition as example

Ordinary filesystem partition = (Ordinary filesystem header + Ordinary filesystem data)
bcache partition = bcache header (8KiB) + (Ordinary filesystem header + Ordinary filesystem data)
                                          -------------------------------------------------------
                                                                  bcache data

In such case, the device (partition) is represented as a bcache partition, and bcache driver creates a new device called /dev/bcacheX without bcache header, then OS will detect /dev/bcacheX as ordinary filesystem.

This method is widely used in disk encryption as well, which allows encryption driver translates encrypted data device to OS in a newly created device.

Mount a filesystem bypass bcache header

By using following method, the ordinary filesystem in bcache partition can be mounted by ordinary filesystem driver without bcache driver.

  • Create Loopback Device
losetup -o 8192 /dev/loop0 /dev/[BCACHE DEVICE]
  • Mount Device
mount /dev/loop0 -o loop /mnt/[LOCATION]

Devices

Backing Device

Backing device is the actual device holding data, for example, /dev/sdb1, it can be 10TB hard disk, disk raid, software raid, etc. It can be created by following command

make-bcache -B /dev/sdb1

It will create a bcache device as well, such as /dev/bcache0

Bcache device

Bcache device is the device created together with backing device (/dev/sdb1), such as /dev/bcache0, all ordinary filesystem operation will be operated on bcache device (/dev/bcache0), such as mkfs, etc.

Caching Device

Caching device is the temporary device used as cache, for example /dev/sdc1, it can be 128GB SSD. It can be created using following command

make-bcache -C /dev/sdc1

It has cset.uuid as below

# bcache-super-show /dev/sdc1 | grep cset
cset.uuid       f0e01318-f4fd-4fab-abbb-d76d870503ec

Before recreate caching device, use following command clean up the device header.

wipefs -a /dev/<device>

Attach

Attach action allows caching device starts working for backing device.

# echo <caching_device_uuid> > /sys/block/<bcache_device>/bcache/attach
echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/bcache0/bcache/attach
# or to the backing device
echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/sdb/sdb1/bcache/attach

Note: If following error occurred, and no output when ran bcache-status command, then run partprobe command to rescan partition tables

# bcache-status
#
# echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/bcache0/bcache/attach
-bash: echo: write error: No such file or directory
# partprobe

*Note: bcache-status is a free bcache tool can be downloaded from github.

Operation on backing device and bcache device

The backing device and the bcache device have 1-to-1 relationship, because they are created at same time using one make-bcache -C command.

In fact, both bcache folders in both bcache and backing device are the same.

# ls -Hdi /sys/block/sdb/sdb1/bcache
64954 /sys/block/sdb/sdb1/bcache
# ls -Hdi /sys/block/bcache0/bcache
64954 /sys/block/bcache0/bcache

I think the correct thinking should be attaching to bcache device, but the bcache folder is created under backing device.

# readlink -f /sys/block/bcache0/bcache
/sys/devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sdb/sdb1/bcache

I think this is because that bcache device is a virtual device created during backing device (real device) creation, so the actual device structure should assign to real device (backing device).

With or without caching device

Without caching device, bcache driver will directly translate ordinary filesystem driver read/write into bcache device read/write.

With cache device, bcache driver will utilize caching device before backing device operation.

ordinary filesystem operation => bcache driver => bcache filesystem operation => backing device
                                       ||
                                       ||
                                 caching device

So, without caching device, bcache is still operating correctly.

Attach caching device to multiple bcache devices

One caching device can support multiple bcache devices as below

# echo f0e01318-f4fd-4fab-abbb-d76d870503ec > /sys/block/bcache0/bcache/attach
# echo 4b05ce02-19f4-4cc6-8ca0-1f765671ceda > /sys/block/bcache1/bcache/attach
# echo 4b05ce02-19f4-4cc6-8ca0-1f765671ceda > /sys/block/bcache2/bcache/attach
# echo 4b05ce02-19f4-4cc6-8ca0-1f765671ceda > /sys/block/bcache3/bcache/attach
# echo 75ff0598-7624-46f6-bcac-c27a3cf1a09f > /sys/block/bcache4/bcache/attach

A bcache device has only one caching device

The uuid of caching device which attached to bcache device can be found as below.

# ls -la /sys/block/<device>/bcache/cache
lrwxrwxrwx 1 root root 0 Jun 19 18:42 /sys/block/<device>/bcache/set -> ../../../../../../../../fs/bcache/<UUID>

Detach

To detach a caching device, needs to send 1 or cache-set-uuid to bcache device or backing device

Safely remove the caching device from bcache device

echo cache-set-uuid > /sys/block/bcache0/bcache/detach
# or
echo cache-set-uuid > /sys/block/sdb/sdb1/bcache/detach

Detach the caching device from bcache device

echo 1 > /sys/block/bcache0/bcache/detach
# or
echo 1 > /sys/block/sdb/sdb1/bcache/detach

Stop

bcache/backing device

Stop a bcache device, is the same as stop the backing device.

echo 1 > /sys/block/bcache0/bcache/stop
# or
echo 1 > /sys/block/sdb/sdb1/bcache/stop

After stopped bcache/backing device,

  • The /sys/block/sdb/sdb1/bcache folder disappears
  • The /sys/block/bcache0 virtual device disappers
  • But no impact to caching device, and still registered in /sys/fs/bcache/<uuid>.

caching device

Stop caching device, will impact all caching, bcache and backing devices

echo 1 > /sys/fs/bcache/cache-set-uuid/stop

After stopped caching device, all bcache setup related to that caching device disapper

  • The /sys/block/sdb/sdb1/bcache folder disappers if /dev/sdb1 is backing device attached
  • The bcache devices /sys/block/bcache0 disappers if /dev/bcache0 is the bcache device attached
  • The caching device /sys/fs/bcache/<uuid> disappers

To resume

The first way to resume whole setup, can be done by run partprobe.

The second way is using register to resume device one by one.

Register

In fact, registering is needed every bootup, but attaching only have to be done once.

Register is required if the caching or backing device missing during system start up or they are stopped manually.

Register the backing device as below

echo /dev/sdb1 > /sys/fs/bcache/register     # backing device

After registered, the system will

  • Creates /sys/block/sdb/sdb1/bcache folder
  • If didn't attach caching device before stopped or caching device had been registered, then creates /sys/block/bcache0 virtual device

If /sys/block/bcache0 is not created due to missing caching device,

  • The /sys/block/bcache0 device will be created after register missing caching device
  • Or use following command to force to create /sys/block/bcache0 and start running
echo 1 > /sys/block/sdb/sdb1/bcache/running

Warning: If force start, all write cache in caching device will be lost, this can cause filesystem corruption

Register caching device

To register caching device, following command can be used

echo /dev/sdc1 > /sys/fs/bcache/register     # caching device

It will create directory /sys/fs/bcache/<uuid> folder.

If attached backing device had been registered, the /sys/block/bcache0 will be created and running.

Ordinary filesystem operation

All ordinary filesystem operations will be operated on bcache device (/dev/bcache0), for example

mkfs.btrfs /dev/bcache0
mount /dev/bcache0 /mnt
...

Caching state

The caching state can be viewed using following command

cat /sys/block/bcache0/bcache/state

Output:

  • no cache: this means you have not attached a caching device to your backing bcache device
  • clean: this means everything is ok. The cache is clean.
  • dirty: this means everything is setup fine and that you have enabled writeback and that the cache is dirty.
  • inconsistent: you are in trouble because the backing device is not in sync with the caching device

Caching mode

There are 4 caching modes, writethrough, writeback, writearound, and none.

echo writeback > /sys/block/bcache0/bcache/cache_mode

Show caching device info

bcache-super-show /dev/sdXY

Writeback Percent

echo 100 > /sys/block/bcache0/bcache/writeback_percent

Dirty data

How much data in cache has not written into backing device.

cat /sys/block/sda/sda3/bcache/dirty_data

Flush cache to backing device

This might be required if filesystem maintenance needed.

Run following command to disable writeback mode

echo writethrough > /sys/block/bcache0/bcache/cache_mode

Wait until state reports "clean"

watch cat /sys/block/bcache0/bcache/state

Force flush of cache to backing device

echo 0 > /sys/block/bcache0/bcache/writeback_percent

Errors

The /sys/fs/bcache/ folder does not exist

The bcache module was not loaded.

sh: echo: write error: Invalid argument

If dmesg shows

bcache: bch_cached_dev_attach() Couldn't attach sdc: block size less than set's block size

Then the --block 4k parameter was not set on either device and defaults can mismatch.

Otherwise, the device might already be attached.

sh: echo: write error: No such file or directory

The UUID is not a valid cache.

Other considerations

Boot from bcache device

Grub2 does not offer support for bcache, but it is fully supported by UEFI. Check the following link for details

https://wiki.archlinux.org/title/Bcache

References

Bcache
A block layer cache (bcache)
bcache-status