Category: docker

Docker Compose – Basic

Docker Compose - Basic

Commands

  • Start all services
docker-compose up
  • Stop all services
docker-compose down
  • Command to install Docker Compose
pip install -U docker-compose
  • Command to check the version of Docker Compose
docker-compose -v
  • Command to run docker compose file
docker-compose up -d
  • Command to list done all the process
docker-compose up -d --scale
  • Command to use YAML file to configure application services
docker-compose.yaml
  • Check logs
docker-compose logs

Docker vs Podman

Docker vs Podman

Daemon

Podman has no daemon, which can run under normal user.

Docker has daemon dockerd running as root.

Unix socket

Docker uses Unix socket /var/run/docker.sock

All the REST API commands are implemented using this socket.

For example,


`docker version` command is implemented via following API.

curl --unix-socket /var/run/docker.sock http:/v2/version



## References

[Watch on Captionz
what is podman vs docker (beginner - intermediate) anthony explains](https://youtu.be/lkg5QJsoCCQ)

Selection of container orchestration platform

Selection of container orchestration platform

I'm trying to manage my docker containers using container orchestration platform. There are a few of them can be used.

Variations

Docker Swarm

Easy to setup and manage existing docker machines.

Kubernetes (K8s)

Widely used.

K3S

K3s is Rancher’s k3s Kubernetes distro, lightweight, single binary, and low resource usage.

Openshift

Complex

Consideration

Unlike company environments, most of my docker containers applications have individual database. The application usage is low, autoscaling isn't a requirement for me but the backup and restore is important.

References

k8s vs k3s
Docker Swarm vs Kubernetes: how to choose a container orchestration tool

Docker folder removed after removed Docker package in Synology

Docker folder removed after removed Docker package in Synology

I was changing harddisk in Synology DSM 7 in the volume with Docker package installed by recreating the volume, it requires docker package to be removed. I thought the docker folder (/volumeX/docker) might not removed, so backed up the container images into docker folder. But I was wrong, the docker folder was removed.

The docker folder was created by Docker package, which can be moved to other volume after stopped Docker package. I didn't see any data in it, and I don't know what the usage of this folder is and it is zero in size.

Dock installation issue in Armbian with iptables

Dock installation issue in Armbian with iptables

Issue

Docker requires legacy iptables. If docker installation got network issue, following commands might fix the issue.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
apt remove apparmor

References

iptables-nft vs iptables-legacy #11612

Delete docker images from docker registry

Delete docker images from docker registry

Steps

Get Digest by pulling image

Use docker pull command can get image digest

docker pull registry.example.com/image_path/image:tag

If it is daily build for container backup purpose, the tag can be in date format, such as YYYYMMDD. The output can be

20210624: Pulling from image_path/image
...
22b5d63ad977: Already exists
8e2e66517d7e: Pull complete
Digest: sha256:7535af1f65524f9200b901fc31b9c779819e45c0502ef99605666842a319908f

Get Digest by deleting local image

The digest is also printed when deleting it.

docker rmi registry.example.com/image_path/image:tag

Sample output as below

Untagged: registry.example.com/image_path/image:tag
Untagged: registry.example.com/image_path/image@sha256:e300ff463dc18c7b3bf3964dc5a9832f613d829285a0da49e5fd37519dc7d0fc
Deleted: sha256:35baba3d5948b5844b67adcd6a236905039e929f8647d4e4afc9e64e9460d557
Deleted: sha256:bd681f3956f55dc028bae7ca4c2657457824a0e356c59705302fb084660a669b

Note: The Digest is the second tag, the sha256 in the first Deleted message is allowed to be deleted too, but the sha256 in the second Deleted message was not. Don't understand why

Get Digest by curl command

curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://$USER:$PASSWORD@registry.example.com/v
2/image_path/image/manifests/tag 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

Check manifests existence

curl https://$USER:$PASSWORD@registry.example.com/v2/image_path/image/manifests/sha256:xxxxxxxxxxxxxxxx

Delete tag

curl -X DELETE https://$USER:$PASSWORD@registry.example.com/v2/image_path/image/manifests/sha256:xxxxxxxxxxxxxxxx

Run garbage-collect

docker exec registry bin/registry garbage-collect --delete-untagged /etc/docker/registry/config.yml

Restart registry if necessary

docker restart registry

Move Docker Directory

Move Docker Directory

Docker directory is /var/lib/docker. As /var should be a system directory, and most of the files in /var are log files, security related files, etc.

Docker is a application, which contains huge container data, move /var/lib/docker to data disk should be the good practice.

Methods

The are two methods I used to move /var/lib/docker directory.

Change Docker config

In /etc/docker/daemon.json file, add following entry.

{
   "data-root": "/path/to/your/docker" 
}

This makes the Docker implementation become non-standard, but as some other settings in this file also need to be set as well, such as Docker log file size, etc., this can be a normal practise for those system not many administrators.

Create soft link

Move /var/lib/docker to another location using mv command, then create a soft link /var/lib/docker point to new location.

This can be a simple and clear way, because system administrator can easily find the location without pre-check docker configuration, especially if predefined commands are required to be provided to others.

Drawback

For the first method, changing Docker configuration, although the non-standard configuration used, but Docker knows the location of data.

For the second method, adminstrator knows file location, but Docker doesn't know. So, if Docker likes to optimize the system, it might get wrong info.

References

How to move docker data directory to another location on Ubuntu

Docker container restart policy

Docker container restart policy

Policy types

To configure the restart policy for a container, use the --restart flag when using the docker run command. The value of the --restart flag can be any of the following:

Flag Description
no Do not automatically restart the container. (the default)
on-failure Restart the container if it exits due to an error, which manifests as a non-zero exit code.
always Always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted. (See the second bullet listed in restart policy details)
unless-stopped Similar to always, except that when the container is stopped (manually or otherwise), it is not restarted even after Docker daemon restarts.

Command

Following command set restart policy of container redis to unless-stopped.

docker update --restart unless-stopped redis

References

Start containers automatically

Script to delete old docker images

Script to delete old docker images

To schedule a task to delete old docker images.

Assumption

Assuming there is a local copy of image normally. If not, need to directly operate in registry server or pull image locally before execution.

Prereq

Following information should be set as environment variable.

user=username
password=password
registry=registry.example.net
repo=repo

List images

List by image date

This is for local images, list all images' digests created 30 days ago.

docker images $registry/$repo --digests --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Digest}} {{.CreatedAt}}" | awk '{if ($5 < strftime("%Y-%m-%d", systime()-3600*24*30)) {print $1,$2,$3,$4}}'

List by tags

List all tags in format of YYYYDDMM older than 30 days.

TAG_LIST=`curl -s https://$user:$password@$registry/v2/$repo/tags/list | jq '.tags[] | select(. < (now-3600*24*30 | strftime("%Y%m%d")) )'`

Note: strflocaltime can be used if your jq supports it.

Query remote manifest by tag

Using docker command

docker manifest inspect $registry/$repo:$tag -v | jq '.Descriptor.digest'

or using rest api

curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://username:password@registry.example.net/v2/${repo}/manifests/20210624 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

Delete tags

Delete from registry

curl -X DELETE https://$username:$password@$registry/v2/${repo}/manifests/sha256:xxxxxxxxxxxxxxxx

Delete from local

Delete image using image id.

docker rmi $image_id

Script

#!/bin/bash

. ./set_env.sh

user=USERNAME
password=$PASSWORD

repo=${1:-repo_name}
registry=REGISTRY_NAME
keep_days=30

## List all docker images

docker images $registry/$repo --digests --format "{{.Repository}} {{.Tag}} {{.ID}} {{.Digest}} {{.CreatedAt}}" | awk -v keep_days=$keep_days '{if ($5 < strftime("%Y-%m-%d", systime()-3600*24*keep_days)) {print $1,$2,$3,$4}}' | while read line
do
        read repo_path tag id manifest <<< $line

        echo Deleting from registry: $line
        curl -X DELETE https://$user:$password@$registry/v2/${repo}/manifests/$manifest

        echo Deleting from local: $line
        docker rmi $id
done

## List all docker images with tags in registry server
curl -s https://$user:$password@$registry/v2/$repo/tags/list | jq '.tags[] | select(. < (now-3600*24*30 | strftime("%Y%m%d")) )' | tr -d '"' | while read tag
do
        echo Deleting from registry: $tag
        manifest=`docker manifest inspect $registry/$repo:$tag -v | jq '.Descriptor.digest' | tr -d '"'`
        curl -X DELETE https://$user:$password@$registry/v2/${repo}/manifests/$manifest
done

References

Docker Registry HTTP API V2
Can I get an image digest without downloading the image?