Blog

Blog

GRUB Basic

Posted on by Bian Xi Leave a Comment on GRUB Basic

GRUB Basic

Access grub menu

For hidden menu, it can be accessed by holding down the Shift key at the very start of the boot-up process.

If still cannot be started, then provide following lines in /etc/defaults/grub could fix the issue.

GRUB_TERMINAL_OUTPUT="console"

Power down monitor

Power down monitor can be done via command line as below.

setterm --blank 1 --powerdown 2

It blanks screen after 1 min and turn it off after 2 min.

By adding the kernel parameter consoleblank=60 will lets screen off in 60 seconds if no one touch it.

Regenerate /boot/grub/grub.cfg

Run following command can regenerate /boot/grub/grub.cfg using /etc/defaults/grub

update-grub
# or
update-grub2

Update /boot/efi/EFI/ubuntu/grub.cfg

Run following command can update this file according to current envirnment structure.

dpkg-reconfigure grub-efi-amd64

It is also update /boot/grub too.

References

GRUB2 101: How to Access and Use Your Linux Distribution’s Boot Loader
How To Configure GRUB2 Boot Loader Settings In Ubuntu
How to power down display on terminal?
How to Configure the GRUB2 Boot Loader’s Settings

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Posted on by Bian Xi Leave a Comment on Convert Raspberry Pi Ubuntu to iSCSI btrfs root

Convert Raspberry Pi Ubuntu to iSCSI btrfs root

This is to move root file system of Raspberry Pi Ubuntu OS to iSCSI LUN and convert it to btrfs file system

Pros

  • Cheaper than using small SD card
  • Backup and restore easy, only need to backup 150MB boot partition on SD card
  • Able to perform snapshot at LUN level or OS level (btrfs)
  • Should be Faster

Steps

Install iscsi packages

apt install open-iscsi
systemctl enable open-iscsi
systemctl start open-iscsi
systemctl enable iscsid
systemctl start iscsid

Configure iscsi

Edit /etc/iscsi/initiatorname.iscsi, update following line

InitiatorName=<YOUR_INITIATOR_NAME>

Note: The YOUR_INITIATOR_NAME is the iSCSI client name

Edit /etc/iscsi/iscsid.conf, update following lines

node.session.auth.authmethod = CHAP
node.session.auth.username = <YOUR_USERNAME>
node.session.auth.password = <YOUR_PASSWORD>
# iscsiadm --mode discovery --type sendtargets --portal <YOUR_TARGET_IP>
# iscsiadm --mode node --targetname <YOUR_TARGET_NAME> --portal <YOUR_TARGET_IP> --login

Note: If can not login, restart iscsid and try again.

systemctl restart iscsid

Identify block device

Use lsblk command to identify device file, normally should be /dev/sda.

Partitioning

Creating two partitions using fdisk, the first partition is to prepare following for future used, such as

  • Network boot
  • UEFI iSCSI boot
  • SD card backup

Partition /dev/sda1: vfat, 2GB
Partition /dev/sda2: for root filesystem

Create filesystems

mkfs.vfat /dev/sda1
mkfs.btrfs /dev/sda2

Identify UUID for root filesystem

blkid /dev/sda2

Update initramfs

This is to enable ubuntu load iscsi driver during boot

touch /etc/iscsi/iscsi.initramfs
update-initramfs -v -k $(uname -r) -c

Duplicate files

mount /dev/sda2 /mnt
rsync -avhP --exclude /boot/firmware --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
mkdir /mnt/{dev,proc,sys,boot/firmware,mnt}

Modify /etc/fstab in LUN

Note: Don't modify the file in /etc, it will not be used during iSCSI boot.

vi /mnt/etc/fstab

Change root mounting to

UUID=<YOUR_DEV_UUID> / btrfs defaults 1 1

Modify /boot/firmware/cmdline.txt

First create a backup of this file

cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.sav

Change the content of /boot/firmware/cmdline.txt

Note: Beware of rootfstype=btrfs

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=btrfs elevator=deadline rootwait fixrtc ip=dhcp root=UUID=<YOUR_DEV_UUID> ISCSI_INITIATOR=<YOUR_INITIATOR_NAME> ISCSI_TARGET_NAME=<YOUR_TARGET_NAME> ISCSI_TARGET_IP=<YOUR_TARGET_IP> ISCSI_TARGET_PORT=3260 ISCSI_USERNAME=<YOUR_USERNAME> ISCSI_PASSWORD=<YOUR_PASSWORD> rw

Note: Root partition can also be identified by label if assigned one, see next topic

Reboot

umount /mnt
reboot

Use LABEL for root

Using label instead of UUID for root filesystem.

Assign a label

For mounted filesystem

sudo btrfs filesystem label <mountpoint> <newlabel>

For not mounted filesystem

sudo btrfs filesystem label <device> <newlabel>

Change /etc/fstab

LABEL=<newlabel> / btrfs defaults 1 1

Change /boot/fireware/cmdline.txt

... root=LABEL=<newlabel> ...

Use Static IP

To use static IP for iSCSI connection, the ip definition in cmdline.txt needs to be changed to

ip=192.168.1.200::192.168.1.1:255.255.255.0:rpi:eth0:off

This will create an IP 192.168.1.200 on interface eth0 as below.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ef:07:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.23/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 43036sec preferred_lft 43036sec
    inet6 fe80::dea6:32ff:feef:70f/64 scope link
       valid_lft forever preferred_lft forever

One way to just maintain one IP is, disabling OS IP address.

Reboot

Other cmdline.txt format

I tried following format, but failed. I think it maybe work if no partition in iSCSI LUN

root=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

Use UUID is better, because there is no need to worry about LUN id and partition as UUID is unique ideitifier.

To specfic UUID, following format can be used according to kernel parameters, but it doesn't work too.

root=UUID=<UUID>
netroot=iscsi:[<username>:<password>[:<reverse>:<password>]@][<servername>]:[<protocol>]:[<port>][:[<iscsi_iface_name>]:[<netdev_name>]]:[<LUN>]:<targetname>

I also tried rd.* format, such as rd.iscsi.initiator, etc., they are new format to replace old format ISCSI_INITIATOR, etc., but also not failed. I think the dracut.cmdline version used by ubuntu in raspberry pi is old. Maybe should try grub2 used in Fedora OS for raspberry pi, or uefi used in Windows 10.

Backup and restore using pre-backup data

Backup data into iSCSI LUN partition 1

First create vfat in iSCSI LUN as partition 1, then backup /boot/fireware data into that partition

mkfs.vfat /dev/sda1
mount /dev/sda1 /mnt
cp /boot/firmware/. /mnt
umount /mnt

Restore to an empty SD card

Create partition in new SSD as type c, which W95 FAT32 (LBA), with Boot flag.

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048 1050623 1048576  512M  c W95 FAT32 (LBA)

Format SD card and give label as system-boot, the LABEL is defined in /etc/fstab, it can be changed to UUID if needed.

mkfs.vfat -n system-boot /dev/sdb1
mount /dev/sdb1 /mnt
cp -a <backup_filesystem> /mnt
umount /mnt

Make sure system reported correct LABEL on the newly created vfat filesystem, using blkid command to verify.

Troubleshooting

iscsi_tcp missing

If following error occurred, install package linux-modules-extra.

libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='iscsi_tcp'

Please read post Missing iSCSI module in Ubuntu 20.10

Reboot error

If failed to boot, initram command prompt will appear. In this case, following commands can be used to recover back the booting.

mkdir /mnt
mount /dev/mmcblk0p1 /mnt
cd /mnt
cp cmdline.txt.sav cmdline.txt
cd /
umount /mnt
reboot

Note: the umount is very important, otherwise, the changes wouldn't be saved.

There are very minimum commands can be used, such as no vi. So let it boots into previous status, then troubleshooting from there.

References

dracut kernel command line options
Kernel command line parameters
introduction to boot time parameters of the Linux kernel
Raspberry Pi 4 UEFI Boot
RPi cmdline.txt
RPi config.txt
kernel-parameters.txt
The config.txt file
Raspberry Pi iSCSI Root on Ubuntu 20.04
btrfs root filesystem on raspbian
[Howto] booting from iSCSI
Ubuntu Server 20.10 on Raspberry Pi 4: installation guide with USB Boot (no SD card) and full disk encryption (excluding /boot) using btrfs-inside-luks and auto-apt snapshots with Timeshift
Raspberry Pi 4 - Ubuntu 20.04 w/Btrfs root
dracut.cmdline(7) — Linux manual page

Plan install a NextCloud server at home

Posted on by Bian Xi Leave a Comment on Plan install a NextCloud server at home

Plan install a NextCloud server at home

Planning to have a NextCloud server at home. There are many solutions.

Solutions

Synology

Install NextCloud as docker in Synology ds1812+ or ds2419+, but

  • Unable to use other system next time.
  • Expose all data in NAS
  • CPU and memory limited

But consider use it as storage via iSCSI.

TrueNAS

Install NextCloud on TrueNAS is officially supported. It is installed as docker container in TrueNAS, but

  • Use more power as the system running on a normal PC
  • Migration might be hard

Raspberry Pi

Install NextCloud on Raspberry Pi, a few options there

  • Install directly on OS, but will mix up with OS as well
  • Install as docker, needs to have correct structure to be maintained
  • Install as NextCloudPi, limited storage on SD card

First plan

After considered all above, first plan will be

  • Install on a Pi which has 1GB ethernet port
  • Install BerryBoot and boot from Synology iSCSI disk
  • Configure storage utilize Synology NAS

Pros

  • Separate storage and application
  • Low power
  • Easy to replace the hardware
  • Got official update regularly

Second plan

After read some documents on line, feel the raspberry pi at current stage still isn't good for NextCloud. The major issue is the CPU speed, which also caused 1GB ethernet performance drop to 40MB/s.

I think I had better use my core 2 due machine to install NextCloud. Maybe I will try it on Raspberry Pi 4 with 8GB RAM later as well.

References

NextCloud Website
BerryBoot Installation
BerryBoot iSCSI Installation
Berryboot install NextCloudPi on an external drive step by step
Nextcloud now officially supported on TrueNAS
NextCloud Plugin on TrueNAS

Add self-signed certificate for TrueNAS

Posted on by Bian Xi Leave a Comment on Add self-signed certificate for TrueNAS

Add self-signed certificate for TrueNAS

To use self-signed certificate in TrueNAS, following steps are required.

Add Certificate into TrueNAS

  • Select Credentials -> Certificates
  • In Certificates section, click on Add button
  • In Add Certificate window, give a name, and select Import Certificate
  • In Extra Constraints section, cut and paste the contents of cert file and key file into Certificate and Private Key textboxes

Configure GUI certificate

  • Select System Settings -> General
  • In GUI section, click on Settings button
  • In GUI Settings window, select the certificate to be used in GUI SSL Certificate option
  • Click on Save button

Restart

Restart UI web server, which is done automatically.

Refresh browser, need to click reload botton.

Add root certificate to MacOS

Posted on by Bian Xi Leave a Comment on Add root certificate to MacOS

Add root certificate to MacOS

If you have your own root certificate like I do, the follow the steps below to add it in MacOS.

Browser vs OS level installation

When accessing a server which signed using your own root certificate, if it isn't installed locally, browser will prompt the warning. Then need to select trust the certifcate in order to continue.

This action only trust that machine certificate in the browser, it only does

  • Trust the machine certificate, not the root certificate
  • Only trust in the browser currently used

The benefit of installing root certificate OS level are

  • All applications will trust certificate
  • Only one time installation required
  • Trusted for all users in the system (Not Firefox Browser)

Download certificate

Provide by issuer

Go to issuer software, such as Synology NAS, download from certificate store, and extract CA certificate, such as example-ca-cert.pem.

From browser with root certificate

The root certificate can be downloaded from browser if the brower has been installed. For example, in Firefox,

  • Click on lock icon besides URL => connection secure => more information, then Page Info window appears
  • Click on View Certificate in Security tab, then certificate information page is displayed as a new browser tab.
  • Look for chain certificate. In Firefox, it is under Miscellaneous => Download PEM (chain)
  • Click on chain certificate and save it locally.

Note: The downloaded certificate file contains both server certificate and root certificate. Delete server certificate using text editor if possible. If the server certificate had been installed in keychain, it can be removed from keychain later too

Install certificate

Use following steps to install CA certificate into keychain

  • Double click the certificate file (with ".pem" or ".cer" extension)
  • Choose "System" from the keychain option. Then press "Add" to install after password provided

Set certificate "Always Trust"

To set system wide trust, use following steps.

  • Open Keychain Access application
  • Look for root certificate, double click it
  • Expand Trust section
  • Select "Always Trust" from list of When using this certificate.

Delete server certificate if needed

If the server certificate was also installed, suggest to delete it from keychain and browser certificate store. This is to avoid false information about successful installation.

  • Open Keychain Access application
  • Look for server certificate
  • Right click on it, then select delete certificate

Firefox Only

In Firefox, which has its own certificate store, the system certificates are not accepted. So use following steps to enable system certificates to be used for current user.

  • Open new tab, and type about:config
  • Search for security.enterprise_roots.enabled
  • Change it to true by double click the line.

Note: This only enable trust for current user

Reboot

Verify

Use browser to access another website which has the same root certificate, the certificate not trusted page should not appear.

References

FAQ: How to add root certificate to Mac OS X

Script to delete old docker images

Posted on by Bian Xi Leave a Comment on Script to delete old docker images

Script to delete old docker images

To schedule a task to delete old docker images.

Assumption

Assuming there is a local copy of image normally. If not, need to directly operate in registry server or pull image locally before execution.

Prereq

Following information should be set as environment variable.

user=username
password=password
registry=registry.example.net
repo=repo

List images

List by image date

This is for local images, list all images' digests created 30 days ago.

docker images $registry/$repo --digests --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Digest}} {{.CreatedAt}}" | awk '{if ($5 < strftime("%Y-%m-%d", systime()-3600*24*30)) {print $1,$2,$3,$4}}'

List by tags

List all tags in format of YYYYDDMM older than 30 days.

TAG_LIST=`curl -s https://$user:$password@$registry/v2/$repo/tags/list | jq '.tags[] | select(. < (now-3600*24*30 | strftime("%Y%m%d")) )'`

Note: strflocaltime can be used if your jq supports it.

Query remote manifest by tag

Using docker command

docker manifest inspect $registry/$repo:$tag -v | jq '.Descriptor.digest'

or using rest api

curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://username:password@registry.example.net/v2/${repo}/manifests/20210624 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

Delete tags

Delete from registry

curl -X DELETE https://$username:$password@$registry/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete from local

Delete image using image id.

docker rmi $image_id

Script

#!/bin/bash

. ./set_env.sh

user=USERNAME
password=$PASSWORD

repo=${1:-repo_name}
registry=REGISTRY_NAME
keep_days=30

## List all docker images

docker images $registry/$repo --digests --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Digest}} {{.CreatedAt}}" | awk -v keep_days=$keep_days '{if ($5 < strftime("%Y-%m-%d", systime()-3600*24*keep_days)) {print $1,$2,$3,$4}}' | while read line
do
        read repo_path tag id manifest <<< $line

        echo Deleting from registry: $line
        curl -X DELETE https://$user:$password@$registry/v2/${repo}/manifests/$manifest

        echo Deleting from local: $line
        docker rmi $id
done

## List all docker images with tags in registry server
curl -s https://$user:$password@$registry/v2/$repo/tags/list | jq '.tags[] | select(. < (now-3600*24*30 | strftime("%Y%m%d")) )' | tr -d '"' | while read tag
do
        echo Deleting from registry: $tag
        manifest=`docker manifest inspect $registry/$repo:$tag -v | jq '.Descriptor.digest' | tr -d '"'`
        curl -X DELETE https://$user:$password@$registry/v2/${repo}/manifests/$manifest
done

References

Docker Registry HTTP API V2
Can I get an image digest without downloading the image?

Clean up docker registry

Posted on by Bian Xi Leave a Comment on Clean up docker registry

Clean up docker registry

After docker containers are backed up daily to docker registry, clean up process is required.

Note: If the image without tag, and if it can be found client locately, then can run command docker inspect to find out. Otherwise, can check "Run command from server" section find out.

List _catalog (repositries)

Following curl command can be used to display all _catalog (repositries).

curl https://username:password@registry.example.net/v2/_catalog

List images (tags) for each repositry

Following curl command can be used to display tags for specific repo

curl https://username:password@registry.example.net/v2/${repo}/tags/list

Get Digest

Using pull request

Use pull image command to get digest as below.

docker pull registry.example.net/${repo}:20210624
20210624: Pulling from user/host/dnsmasq
...
22b5d63ad977: Already exists
8e2e66517d7e: Pull complete
Digest: sha256:7535af1f65524f9200b901fc31b9c779819e45c0502ef99605666842a319908f

Get digest when deleting local image

Using delete local image action also can get digest as well

docker rmi registry.example.net/${repo}:20210619
# Untagged: registry.example.net/user/host/dnsmasq:20210619
# Untagged: registry.example.net/user/host/dnsmasq@sha256:e300ff463dc18c7b3bf3964dc5a9832f613d829285a0da49e5fd37519dc7d0fc
# Deleted: sha256:35baba3d5948b5844b67adcd6a236905039e929f8647d4e4afc9e64e9460d557
# Deleted: sha256:bd681f3956f55dc028bae7ca4c2657457824a0e356c59705302fb084660a669b

The Digest is the second tag.

Note: The first deleted sha256 allowed to be deleted too, but not the second deleted. Don't understand why

Run command from client

Run following command to get Digest

curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://username:password@registry.example.net/v2/${repo}/manifests/20210624 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

Run command from server

This is usefull if there is no image at client and no tag for specfic image.

docker exec privateregistry_registry_1 bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml | grep '${repo}: marking manifest' | awk '{print $4}'

Check manifests exists or not

curl https://username:password@registry.example.net/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete tag

curl -X DELETE https://username:password@registry.example.net/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete _catalog

The v2 registry doesn't allow deleting only certain tags from an image, and deletion of tags is in an open PR for a future version of the Registry (https://github.com/docker/distribution/pull/2169).

rm -rf docker/registry/v2/repositories/${repo}/

*Note: After delete, garbage-collect, restart, the repo is still reporting out as empty _catalog.

Run garbage-collect

Login to registry server and run following command

docker exec registry bin/registry garbage-collect --delete-untagged /etc/docker/registry/config.yml

If the repo is still listed in the output, need to delete them too.

Restart registry if necessary

docker restart registry

References

Docker Private Registry - Deleted all images, but still showing in catalog
Clean Up Your Docker Registry
Delete repository from v2 private registry
Pull Request - New Tags API #2169
Docker Registry HTTP API V2

JFrog artifactory open-source edition

Posted on by Bian Xi Leave a Comment on JFrog artifactory open-source edition

JFrog artifactory open-source edition

JFrog artifactory contains many feature, such as git, docker, etc.

Download URL

https://jfrog.com/open-source/#artifactory

Docker

Docker image can be pulled via docker command

docker pull releases-docker.jfrog.io/jfrog/artifactory-oss:latest

Docker compose can be found in download url as well.

Rsync backup client

Posted on by Bian Xi Leave a Comment on Rsync backup client

Rsync backup client

Rsync backup command can be used as below.

rsync -avR --delete
        --backup-dir=/folder-`date +%Y.%m.%d` \
        --password-file=/root/password_file \
        --exclude-from='exclude-list.txt' \
        /app \
        /usr/local/share/ca-certificates/ \
        /etc/network/interfaces \
        /etc/NetworkManager/system-connections/ \
        /root \
        /home \
        rsync@server::NetBackup/folder

The password_file only contains rsync password without any other data.

In rsync@server::NetBackup/folder, rsync is user id, server is rsync server, NetBackup is rsync service (share folder), folder is where backup to be saved.

The backup-dir is the folder created in the rsync server to save changes.