Day: December 10, 2021

Resize bpool on ubuntu VM with zfs

Posted on by Bian Xi Leave a Comment on Resize bpool on ubuntu VM with zfs

Resize bpool on ubuntu VM with zfs

Got two kind of messages of disk space issue on bpool.

  • apt upgrade can not perform snapshot
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 19%.
  • do-release-upgrade can not be performed

Steps

  • Add iSCSI LUN

  • Change grub configuration

  • Partition iSCSI LUN

  • Attach partitions into zpool

  • Detach old partitions from zpool

  • Repartition rpool and bpool partition in old disk

  • Add back to the rpool and bpool

  • Run update-grub2

  • Detach iSCSI rpool and bpool

  • Run following command to set autoexpand

zpool set autoexpand=on bpool
  • Run partprobe or zfs online
zpool online -e bpool <partition_id>
  • Set autoexpand off
zpool set autoexpand=off bpool

Troubleshooting

Removed local boot partition

I also got unable to boot error due to removed local bpool, and grub can not find BOOT filesystem as it was in iSCSI LUN.

To fix this issue, use following steps

  • Boot from CDROM
  • Install open-iscsi package
  • Add iSCSI LUN
  • Use zfs import bpool to import bpool from iSCSI
  • Attach local boot partition back to bpool again
  • Reboot

Used sfdisk copy partition

This creates an issue, the two partitions has same blkid. After added the second iSCSI LUN.

References

HOWTO replace zfs bpool and rpool with larger disk - Ubuntu 20.04 (Virtualbox)
ZFS on Linux resize rpool

Commands for Signed SSH Certificates using Hashicorp Vault

Posted on by Bian Xi Leave a Comment on Commands for Signed SSH Certificates using Hashicorp Vault

Commands for Signed SSH Certificates using Hashicorp Vault

List down the commands required.

Client

Generate SSH Admin token (One time)

export VAULT_ADDR='https://vault.bx.net:8200'
export VAULT_TOKEN="<ROOT_TOKEN>"

vault token create -field token -policy=ssh-admin-policy

Renew Admin token

export VAULT_TOKEN="<SSH_ADMIN_TOKEN>"
vault token renew

Generate signed certificate

export VAULT_TOKEN="<SSH_ADMIN_TOKEN>"
vault token lookup
vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/signed-cert.pub

SSH using signed certificate

ssh -i ~/.ssh/signed-cert.pub -i ~/.ssh/id_rsa <host>

Server

Save CA key

export VAULT_ADDR='https://vault.bx.net:8200'
export VAULT_TOKEN="<ROOT_TOKEN>"

vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem

Configure /etc/ssh/sshd_config

Add following lines in /etc/ssh/sshd_config

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
CASignatureAlgorithms ^ssh-rsa

Note: Comment out last line if SSH got error

Troubleshooting

Server SSL cert

The SSL cert in vault server needs to be trusted by local client, otherwise, following server occurred.

Error writing data to ssh-client-signer/sign/my-role: Put "<role_name>": x509: certificate signed by unknown authority

References

Selection of container orchestration platform

Posted on by Bian Xi Leave a Comment on Selection of container orchestration platform

Selection of container orchestration platform

I'm trying to manage my docker containers using container orchestration platform. There are a few of them can be used.

Variations

Docker Swarm

Easy to setup and manage existing docker machines.

Kubernetes (K8s)

Widely used.

K3S

K3s is Rancher’s k3s Kubernetes distro, lightweight, single binary, and low resource usage.

Openshift

Complex

Consideration

Unlike company environments, most of my docker containers applications have individual database. The application usage is low, autoscaling isn't a requirement for me but the backup and restore is important.

References

k8s vs k3s
Docker Swarm vs Kubernetes: how to choose a container orchestration tool