Author: Bian Xi

Learning – Docker

Posted on by Bian Xi Leave a Comment on Learning – Docker

Learning - Docker

Just to refresh my Docker knowledge.

Logs

docker logs -f --tail 100 nginx

Network

docker network create mongo-network

mongo and mongo-express

docker-compose

docker-compose -f docker-compose.yaml up
docker-compose -f docker-compose.yaml down

Dockerfile

FROM nginx:1.10.2-alpine
MAINTAINER my@example.com
ENV

RUN

COPY ./nginx.conf /etc/nginx/nginx.conf

CMD

build

docker build -t my-app:1.0 .

AWS ECR

Fully-managed Docker container registry

Default registry

docker pull mongo:4.2

same as

docker pull docker.io/library/mongo:4.2

Tag

docker tag my-app:latest <reg>/my-app:latest

Push

docker push <reg>/my-app:latest

Volume

c:\programData\docker\volumes
/var/lib/docker/volumes

In Mac

# screen ~/Library/Containers/com.docker.docker/Data/com.docker.amd64-linux/tty
# ls /var/lib/docker/volumes

References

Docker Tutorial for Beginners [FULL COURSE in 3 Hours]

Ping fails but nslookup works in MacOS

Posted on by Bian Xi Leave a Comment on Ping fails but nslookup works in MacOS

Ping fails but nslookup works

When performing nslookup, the IP can be returned. But when performing ping, return following error

ping: cannot resolve <hostname>: Unknown host

Causes

Second DNS server issue

The second DNS server isn't resolving the IP address.

Another DHCP server issue

Another DHCP server provide different DNS server.

Fix

killall -HUP mDNSResponder

References

DNS lookup fails but nslookup works

Install OpenVPN client in ubuntu

Posted on by Bian Xi Leave a Comment on Install OpenVPN client in ubuntu

Install OpenVPN client in ubuntu

Due to PPTP might not available, OpenVPN can be a good choice.

Packages

apt install openvpn

Configuration file

Configuration file should be downloaded from OpenVPN server.

Test

openvpn --config client.ovpn

Enable option

In /etc/default/openvpn, enable following line

AUTOSTART="all"

Copy config file

cp client.ovpn /etc/openvpn/client.conf

Note: the file name should be client.conf

Enable password (Optional)

Change auth-user-pass to auth-user-pass pass in client.conf.

Then create account info in pass file, and change mode

chmod 400 /etc/openvpn/pass

Enable service

systemctl enable openvpn@client.service
systemctl daemon-reload
systemctl start openvpn@client

References

Linux - Autostart OpenVPN in systemd (Ubuntu)

Configure PPTP in Ubuntu

Posted on by Bian Xi Leave a Comment on Configure PPTP in Ubuntu

Configure PPTP in Ubuntu

Install packages

apt-get install pptp-linux

Configure password

vi /etc/ppp/chap-secrets

The format should be

pptpuser    PPTP    pptppassword    *

Create route script

Create file /etc/ppp/ip-up.d/route-traffic

#!/bin/bash

route add -net 192.168.1.0/8 dev ppp0

Change mode to executable

chmod +x /etc/ppp/ip-up.d/route-traffic

Create dial up configure

The name of file is defined as connection name, such as /etc/ppp/peers/<CONNECTON>

pty "pptp <HOST> --nolaunchpppd"
name <USER>
remotename PPTP
require-mppe-128
file /etc/ppp/options.pptp
ipparam <CONNECTON>

Open filewall

iptables -A INPUT -i pptp -j ACCEPT
iptables -A OUTPUT -o pptp -j ACCEPT

Connect and disconnect

pon <CONNECTON>
poff <CONNECTION>

Troubleshooting

If didn't configure PPTP Passthrough on your NAT, then PPTP will failed.

PPTP uses a TCP channel on port 1723 for control and the GRE protocol to encapsulate data and create a VPN tunnel. The issue isn’t really PTPP itself but GRE; GRE doesn’t use ports. Since a requirement of NAT is that the connection must use an IP address and port number it doesn’t work with GRE. This is what PTPP passthrough addresses.

References

Set up Linux PPTP Client from the Terminal
Connect to PPTP server via command line on Ubuntu
pptp - 'Administrative Shutdown'. Error code is 0, Cause code is 0

Troubleshooting Hashicorp Vault SSH Certificate Login

Posted on by Bian Xi Leave a Comment on Troubleshooting Hashicorp Vault SSH Certificate Login

Troubleshooting Hashicorp Vault SSH Certificate Login

If can not login via SSH, normally can use -vvv as SSH option in SSH client command to verify, but it could be very long to read.

Another way is use systemctl status sshd command at server side to check the error.

For example, the output of systemctl status sshd got following lines,

Dec 12 00:40:37 example-host systemd[1]: Started OpenBSD Secure Shell server.
Dec 12 00:40:54 example-host sshd[22712]: error: Certificate invalid: expired
Dec 12 00:40:54 example-host sshd[22712]: Connection closed by authenticating user ubuntu 101.78.78.154 port 53369 [preauth]
Dec 12 00:41:12 example-host sshd[22716]: error: Certificate invalid: name is not a listed principal
Dec 12 00:41:12 example-host sshd[22716]: Connection closed by authenticating user ubuntu 101.78.78.154 port 53372 [preauth]

The first error shows vault signed certificate at client side had expired, need to rerun vault command to regenerate signed certificate.

The second error shows the user name was used in client is not listed in vault signed certificate, so need to use correct user name or configure a new role in the vault.

Duplicate partition table in ubuntu

Posted on by Bian Xi Leave a Comment on Duplicate partition table in ubuntu

Duplicate partition table in ubuntu

Note: Haven't got chance to test this.

Duplicate partitions

sfdisk -d /dev/sdX > part_table
sfdisk /dev/sdY < part_table

This will keeping the same disk & partition IDs, mainly for backup and restore partition purpose.

Generate new partition IDs

grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdY

References

How to copy the partition layout of a whole disk using standard tools

Run do-release-upgrade to ubuntu 21.10

Posted on by Bian Xi Leave a Comment on Run do-release-upgrade to ubuntu 21.10

Run do-release-upgrade to ubuntu 21.10

Looks like a simple task, but ending with many things to do...

No screen

After run do-release-upgrade, I went to sleep, then ssh connection dropped due to iMac auto sleep. The installation stopped at asking question of grub installation.

When I check the process, got a process similar to below one, which is still at pts/1, which means the process is still running at another virtual terminal.

/usr/bin/python3 /tmp/ubuntu-release-upgrader-qzt422az/focal --mode=server --frontend=DistUpgradeViewText

When I try to run do-release-upgrade again, got following message.

# do-release-upgrade 
Checking for a new Ubuntu release
No new release found.

Then I try to run apt install screen, says apt is locked by process nnnn.

As suggested by others, I killed the process nnnn, and run following command, then the terminal change to installation virtual screen, and continued.

dpkg --configure -a

Error on grub-efi

Then got following error

# dpkg --configure -a
Setting up grub-efi-amd64-signed (1.173+2.04-1ubuntu47) ...
NTFS signature is missing.
Failed to mount '/dev/sda1': Invalid argument
The device '/dev/sda1' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 12
Errors were encountered while processing:
 grub-efi-amd64-signed

As suggested by others, I ran following commands, which remove grub2 and install grub, then no error any more.

Note: do backup of /etc/default/grub, and verify the contents. I didn't do this, caused more issues later

apt-get purge grub\*
sudo apt-get install grub-efi
sudo apt-get autoremove
sudo update-grub

No zpool found during reboot

The system go into initramfs mode after reboot, looks like iSCSI devices not found.

Then I booted from CD again, and found that iSCSI configuration in /etc/default/grub was missing.

After login to iSCSI, I mirror back the bpool and rpool back to local disk, and run following command to fix it.

update-initramfs -v -k $(uname -r) -c
update-grub

Forgot waiting for resilvering finish

Forgot wait for zpool sync finish, the I rebooted the server.

Note: Some people said it is ok, but my case is NOT OK.

After perform zpool resync again, got following error with local partition CKSUM = 1.

One or more devices has experienced an unrecoverable error.

This was caused by reboot, just perform following command to clean the flag.

zpool clear rpool

Booting hung

This is an old issue, caused by network shutdown before iSCSI drive dismount. Run following command,

systemctl edit --full open-iscsi.service

Comment out the following line

#ExecStop=/lib/open-iscsi/logout-all.sh

References

Sub-process /usr/bin/dpkg returned an error code (1)

Resize bpool on ubuntu VM with zfs

Posted on by Bian Xi Leave a Comment on Resize bpool on ubuntu VM with zfs

Resize bpool on ubuntu VM with zfs

Got two kind of messages of disk space issue on bpool.

  • apt upgrade can not perform snapshot
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 19%.
  • do-release-upgrade can not be performed

Steps

  • Add iSCSI LUN

  • Change grub configuration

  • Partition iSCSI LUN

  • Attach partitions into zpool

  • Detach old partitions from zpool

  • Repartition rpool and bpool partition in old disk

  • Add back to the rpool and bpool

  • Run update-grub2

  • Detach iSCSI rpool and bpool

  • Run following command to set autoexpand

zpool set autoexpand=on bpool
  • Run partprobe or zfs online
zpool online -e bpool <partition_id>
  • Set autoexpand off
zpool set autoexpand=off bpool

Troubleshooting

Removed local boot partition

I also got unable to boot error due to removed local bpool, and grub can not find BOOT filesystem as it was in iSCSI LUN.

To fix this issue, use following steps

  • Boot from CDROM
  • Install open-iscsi package
  • Add iSCSI LUN
  • Use zfs import bpool to import bpool from iSCSI
  • Attach local boot partition back to bpool again
  • Reboot

Used sfdisk copy partition

This creates an issue, the two partitions has same blkid. After added the second iSCSI LUN.

References

HOWTO replace zfs bpool and rpool with larger disk - Ubuntu 20.04 (Virtualbox)
ZFS on Linux resize rpool

Commands for Signed SSH Certificates using Hashicorp Vault

Posted on by Bian Xi Leave a Comment on Commands for Signed SSH Certificates using Hashicorp Vault

Commands for Signed SSH Certificates using Hashicorp Vault

List down the commands required.

Client

Generate SSH Admin token (One time)

export VAULT_ADDR='https://vault.bx.net:8200'
export VAULT_TOKEN="<ROOT_TOKEN>"

vault token create -field token -policy=ssh-admin-policy

Renew Admin token

export VAULT_TOKEN="<SSH_ADMIN_TOKEN>"
vault token renew

Generate signed certificate

export VAULT_TOKEN="<SSH_ADMIN_TOKEN>"
vault token lookup
vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/signed-cert.pub

SSH using signed certificate

ssh -i ~/.ssh/signed-cert.pub -i ~/.ssh/id_rsa <host>

Server

Save CA key

export VAULT_ADDR='https://vault.bx.net:8200'
export VAULT_TOKEN="<ROOT_TOKEN>"

vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem

Configure /etc/ssh/sshd_config

Add following lines in /etc/ssh/sshd_config

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
CASignatureAlgorithms ^ssh-rsa

Note: Comment out last line if SSH got error

Troubleshooting

Server SSL cert

The SSL cert in vault server needs to be trusted by local client, otherwise, following server occurred.

Error writing data to ssh-client-signer/sign/my-role: Put "<role_name>": x509: certificate signed by unknown authority

References

Selection of container orchestration platform

Posted on by Bian Xi Leave a Comment on Selection of container orchestration platform

Selection of container orchestration platform

I'm trying to manage my docker containers using container orchestration platform. There are a few of them can be used.

Variations

Docker Swarm

Easy to setup and manage existing docker machines.

Kubernetes (K8s)

Widely used.

K3S

K3s is Rancher’s k3s Kubernetes distro, lightweight, single binary, and low resource usage.

Openshift

Complex

Consideration

Unlike company environments, most of my docker containers applications have individual database. The application usage is low, autoscaling isn't a requirement for me but the backup and restore is important.

References

k8s vs k3s
Docker Swarm vs Kubernetes: how to choose a container orchestration tool