Blog

Blog

Enable VMmotion in VMware

Posted on by Bian Xi Leave a Comment on Enable VMmotion in VMware

Enable VMmotion in VMware

Two ESXi servers, for example esx01 and esx02.

CPU

The CPU of both esx01 and esx02 are same family.

Storage

Both esx01 and esx02 are having same shared storage.

Configure

Configure NFS or iSCSI in both ESXi servers.

Verify

Access storage, and check Hosts tab, both ESXi servers should be in the list.

Network

Add VMkernel Network Adapter with vMotion service, can be on an existing standard switch.

References

VMware Tutorial No.45 | VMware vMotion Step by Step |vMotion in VMware 6.7 |vSphere vMotion GOVMLAB

Install VMware vSphere 7.0 on Proxmox

Posted on by Bian Xi Leave a Comment on Install VMware vSphere 7.0 on Proxmox

Install VMware vSphere 7.0 on Proxmox

Verify

root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
Y

Enable

Intel CPU

echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf

AMD CPU

echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.conf

Install Module

modprobe -r kvm_intel
modprobe kvm_intel

Note: more info, check https://pve.proxmox.com/wiki/Nested_Virtualization

Install

ISO

Download ISO, such as VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso

VM Configure

  • General Tab

    • Name:

  • OS Tab

    • Type: Linux
    • Version: 5.x – 2.6 Kernel

  • System Tab

    • Graphic card: Default
    • SCSI Controller: VMware PVSCSI
    • BIOS: SeaBIOS (OVMF (UEFI) should work too)
    • Machine: q35

  • Hard Disk Tab

    • Bus/Device: SATA
    • Disk size (GiB): 16

  • CPU Tab

    • Cores: 4 (At least 2, 4 will be better if our physical CPU has enough cores)
    • Type: host (or Default (kvm64))
    • Enable NUMA: Check (if possible)

  • Memory Tab

    • Memory (MiB): 4096 (At least 4096, better if assign more)
    • Ballooning Device: Uncheck

  • Network Tab

    • Model: VMware vmxnet3

References

Nested Virtualization
How to Install/use/test VMware vSphere 7.0 (ESXi 7.0) on Proxmox VE 6.3-3 (PVE)

Learning – Prometheus Exporter

Posted on by Bian Xi Leave a Comment on Learning – Prometheus Exporter

Learning - Prometheus Exporter

Steps to monitor MongoDB metrics

  • Deploy MongoDB App
  • Deploy MongoDB Exporter
  • Deploy ServiceMonitor

Deployment

minikube start --cpus 4 --memory 8192 --vm-driver hyperkit
helm ls
kubectl get pod
kubectl get svc
kubectl port-forward prometheus-kube-prometheus-prometheus 9090
kubectl port-forward prometheus-grafana 80

servicemonitor

ServiceMonitor is a custom Kubernetes component

kubectl get servicemonitor
kubectl get servicemonitor prometheus-kube-prometheus-grafana -oyaml
...
metadata:
  labels:
    release: prometheus

spec:
  endpoints:
    - path: /metrics
      port: service
  selector:
    matchLabels:
      app.kubernetes.io/instance: prometheus
      app.kubernetes.io/name: grafana

CRD configuration

$ kubectl get crd
...
prometheuses.monitoring.coreos.com ...
...
$ kubectl get prometheuses.monitoring.coreos.com -oyaml
...
spec:
  serviceMonitorSelector:
    matchLabels:
      release: prometheus
...

Deploy MongoDB

mongodb-without-exporter.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
spec:
  selector:
    app: mongodb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
kubectl apply -f mongodb-without-exporter.yaml
kubectl get pod

Deploy MongoDB Exporter

Translator between apps data to Prometheus understandable metrics

Target (MongoDB App) <= fetches metrics <= converts to correct format <= expose /metrics <= Prometheus Server

  • Separate deployment - No need to change config files

MongoDB exporter (mongodb-exporter) can be downloaded from exporter site or dockerhub.

Exporter Site

Exporters can be downloaded from https://prometheus.io/docs/instrumenting/exporters

Nodes exporter - translates metrics of cluster Nodes, exposes /metrics

prometheus-prometheus-node-exporter-8qvwn

Components for exporter

  • Exporter application - exposes /metrics endpoint
  • Service - for connnecting to the exporter
  • ServiceMonitor - to be discovered

Helm chart for exporter

Search for mongodb-exporter helm chart

https://github.com/prometheus-community/helm-charts

Override values using chart parameters

helm show values <chart-name>

Add Helm repo

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm show values prometheus-community/prometheus-mongodb-exporter > values.yaml

Override values in values.yaml

mongodb:
  uri: "mongodb://mongodb-service:27017"

serviceMonitor:
  additionalLabels:
    release: prometheus

with the label Prometheus automatically discovers a new ServiceMonitor in the cluster

$ helm install mongodb-exporter prometheus-community/prometheus-mongodb-exporter -f values.yaml
...
$ helm ls
mongodb-exporter
...
$ kubectl get pod
...
mongodb-exporter-prometheus-mongodb-exporter-75...
...
$ kubectl get svc
...
mongodb-exporter-prometheus-mongodb-exporter
...
$ kubectl get servicemonitor
...
mongodb-exporter-prometheus-mongodb-exporter
...
$ kubectl get servicemonitor mongodb-exporter-prometheus-mongodb-exporter -o yaml
...
metadata:
  labels:
    release: prometheus
...

Check endpoint /metrics

$ kubectl get svc
...
mongodb-exporter-prometheus-mongodb-exporter
...
kubectl port-forward service/mongodb-exporter-prometheus-mongodb-exporter 9216

Access https://127.0.0.1:9216/metrics

The mongodb-exporter is added as targets in prometheus, because the label release: prometheus is set and auto discovered.

MongoDB metrics data in Grafana UI

kubectl get deployment
kubectl port-forward deployment/prometheus-grafana 3000

References

Prometheus Monitoring - Steps to monitor third-party apps using Prometheus Exporter | Part 2

Learning – Setup Prometheus Monitoring on Kubernetes

Posted on by Bian Xi Leave a Comment on Learning – Setup Prometheus Monitoring on Kubernetes

Learning - Setup Prometheus Monitoring on Kubernetes

Prometheus Server

  • Data Retrieval Worker - Retrieval - pull metrics data
  • Time Series Database - Storage - stores metrics data
  • Accepts PromQL queries - HTTP Server - accepts queries

Alertmanager

Prometheus Server => push alerts => Alertmanager => Email, Slack, etc.

Prometheus UI

  • Prometheus Web UI

  • Grafana, etc.

  • Visualize the scraped data in UI

Deployment

How to deploy the different parts in Kubernetes cluster?

  • Creating all configuration YAML files yourself and execute them in right order

    • inefficient
    • lot of effort

  • Using an operator

    • Manager of all Prometheus components
    • Find Prometheus operator
    • Deploy in K8s cluster

  • Using Helm chart to deploy operator

    • maintained by Helm community
    • Helm: initial setup
    • Operator: manage setup

Setup with Helm chart

  • Clean Minikube state
$ kubectl get pod
$ helm install prometheus stable/prometheus-operator
$ kubectl get pod
NAME ...
alertmanager-prometheus-prometheus-oper-alertmanager-0
prometheus-grafana-67...
prometheus-kube-status-metrics-c6...
prometheus-prometheus-node-explorter-jr...
prometheus-prometheus-oper-operator-78...
prometheus-prometheus-prometheus-oper-prometheus-0...

Prometheus Components

kubectl get all

2 Statfulset

Prometheus Server

statefulset.apps/prometheus-prometheus-prometheus-oper-prometheus

Alertmanager

statefulset.apps/alertmanager-prometheus-prometheus-oper-alertmanager

3 Deployments

Prometheus Operator - created Prometheus and Alertmanager StatefulSet

deployment.apps/prometheus-prometheus-oper-operator

Grafana

deployment.apps/prometheus-grafana

Kube State Metrics

deployment.apps/prometheus-kube-state-metrics
  • own Helm chart
  • dependency of this Helm chart
  • scrapes K8s components - K8s infrastructure monitoring

3 StatefulSets

Created by Deployment

replicaset.apps/prometheus-prometheus-oper-operator...
replicaset.apps/prometheus-grafana...
replicaset.apps/prometheus-kube-state-metrics...

1 DaemonSet

  • Node Exporter DaemonSet
daemonset.apps/prometheus-prometheus-node-exporter

DaemonSet runs on every Worker Node

  • connects to Server
  • translates Worker Node metrics to Prometheus metrics - CPU usage, load on server

Completed tasks

  • Monitoring Stack
  • Configuration for your K8s cluster
  • Worker Nodes monitored
  • K8s components monitored

ConfigMaps

kubectl get configmap
  • configurations for different parts
  • managed by operator
  • how to connect to default metrics

Secrets

kubectl get secret

  • for Grafana

  • for Prometheus

  • for Operator

  • certificates

  • username & passwords
    ...

CRDs

kubectl get crd

extension of Kubernetes API

  • custom resource definitions

Describe components

kubectl describe = container/image information

kubectl get statefulset
kubectl describe statefulset prometheus-prometheus-prometheus-oper-prometheus > prom.yaml
kubectl describe statefulset alertmanager-prometheus-prometheus-oper-alertmanager > alert.yaml
kubectl get deployment
kubectl describe deployment prometheus-prometheus-oper-operator > oper.yaml

Stateful oper-prometheus

Containers:

  • prometheus
    • Images: quay.io/prometheus/prometheus:v2.18.1
    • Port: 9090/TCP
    • Mounts: where Prometheus gets its configuration data mounted into Prometheus Pod
    • /etc/prometheus/certs
    • /etc/prometheus/config_out
    • /etc/prometheus/rules/...
    • /prometheus
      They are
    • Configuration file: what endpoints to scrape
    • address of applications: expose /metrics
    • Rules configuration file: alerting rules, etc.

The two sidecar/help container *-reloader, they help reloading, responsible for reloading, when configuration files changes.

  • prometheus-config-reloader

    • Image: quay.io/coreos/prometheus-config-reloader:v0.38.1
    • reloader-url: http://127.0.0.1:9090/-/reload
    • config-file: /etc/prometheus/config/prometheus.yaml.gz

  • rules-configmap-reloader

ConfigMap and Secret (States):

kubectl get configmap
kubectl get secret

In prom.yaml,

  • Args: --config-file=/etc/promtheus/config
  • Mounts:
    • /etc/prometheus/config from config
    • /etc/prometheus/config_out from config_out
  • Volumes: config, it is a secret
kubectl get secret prometheus-prometheus-prometheus-oper-prometheus -o yaml > secret.yaml
apiVersion: v1
data:
  prometheus.yaml.gz: ....

In rules file rules-configmap-reloader

Mounts: /etc/prometheus/rules/prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0 from prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0

Volumes: ConfigMap prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0

kubectl get configmap prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0 -o yaml > config.yaml
  • config.yaml rules file
apiVersion: v1
data:
  default-prometheus-prometheus-oper-alertmanager.rules.yaml
  groups:
    - name: alertmanager.rules
      rules:
      - alert: AlertmanagerConfigInconsistent
...

Stateful alertmanager

Containers:

  • alertmanager

    • Image: quay.io/prometheus/alertmanager:v0.20.0
    • config.file: /etc/alertmanager/config/alertmanager.yaml

  • config-reloader

    • Image: `docker.io/jimmidyson/configmap-reload:v0.3.0

Operator permetheus-operator

Containers:

  • prometheus-operator (orchestrator of monitoring stack)

    • Image: quay.io/coreos/prometheus-operator:v0.38.1

  • tls-proxy

Tasks

  • How to add/adjust alert rules?

  • How to adjust Prometheus configuration?

Access Grafana

$ kubectl get service
...
prometheus-grafana   ClusterIP ...

ClusterIP = Internal Services

$ kubectl get deployment
...
prometheus-grafana
...

$ kubectl get pod
...
prometheus-grafana-67....
...

$ kubectl logs prometheus-grafana-67... -c grafana
...
... user=admin
...
... address=[::]:3000 ...
...

port: 300
default user: admin

$ kubectl port-forward deployment/prometheus-grafana 3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Then the grafana can be accessed via https://localhost:3000

The default admin password is "prom-operator", which can be found in chart: https://github.com/heim/charts/tree/master/stable/prometheus-operator#...

$ kubectl get pod
...
prometheus-kube-state-metrics-c6...
prometheus-prometheus-node-exporter-jr...
...

Prometheus UI

$ kubectl get pod
...
prometheus-prometheus-prometheus-oper-prometheus-0
...

$ kubectl port-forward prometheus-prometheus-prometheus-oper-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Then Prometheus UI can be accessed via https://localhost:9090/.

Summarize

  • Deployed Prometheus stack using Helm
    • easy deployment process
  • Overview of what these different components are and do
  • Configure additional metrics endpoint

References

Setup Prometheus Monitoring on Kubernetes using Helm and Prometheus Operator | Part 1

Learning – Kubernetes Operator

Posted on by Bian Xi Leave a Comment on Learning – Kubernetes Operator

Learning - Kubernetes Operator

Used for Stateful Applications on K8s

Stateless Applications on K8s

Control loop

Observe => Check Differences => Take Action => Observe ...

  • Recreate died pods
  • restart updated pods

Stateful Applications WITHOUT Operator

Data Persistence

  • more "hand-holding" needed

  • throughout whole lifecycle

  • all 3 replicas are different

  • own state and identity

  • order important

  • Process different for each application

  • So, no standard solution

  • manual intervention necessary

  • people, who "operate" these applications

  • can not archive automation, self-healing

Stateful application WITH Operator

To manage stateful application

Replaces human operator with software operator.

  • How to deploy the app?

  • How to create cluster of replicas?

  • How to recover?

  • tasks are automated and reusable

  • One 1 standard automated process

  • more complex/more environments => more benefits

Control loop mechanism

watch for changes

Observe => Check Differences => Take Action => Observe ...

It is custom control loop

make use of CRD’s

Custom Resource Definitions

  • custom K8s component (extends K8s API)

Your own custom component

domain/app-specific knowledge

CRD's, StatefulSet, ConfigMap, Service, ...

automates entire lifecycle of the app it operates

Summary

  • Managing complete lifecycle of stateless apps
    No business logic necessary to: create, update, delete
  • K8s can't automate the process natively for stateful apps
    Operators: prometheus-operator, mysql-operator, postgres-operator, elastic-operator

For example: MySQL

  • How to create the mysql cluster
  • How to run it
  • How to synchronize the data
  • How to update

OperatorHub.io

Operator SDK to create own operator

References

Kubernetes Operator simply explained in 10 mins

Learning – Docker

Posted on by Bian Xi Leave a Comment on Learning – Docker

Learning - Docker

Just to refresh my Docker knowledge.

Logs

docker logs -f --tail 100 nginx

Network

docker network create mongo-network

mongo and mongo-express

docker-compose

docker-compose -f docker-compose.yaml up
docker-compose -f docker-compose.yaml down

Dockerfile

FROM nginx:1.10.2-alpine
MAINTAINER my@example.com
ENV

RUN

COPY ./nginx.conf /etc/nginx/nginx.conf

CMD

build

docker build -t my-app:1.0 .

AWS ECR

Fully-managed Docker container registry

Default registry

docker pull mongo:4.2

same as

docker pull docker.io/library/mongo:4.2

Tag

docker tag my-app:latest <reg>/my-app:latest

Push

docker push <reg>/my-app:latest

Volume

c:\programData\docker\volumes
/var/lib/docker/volumes

In Mac

# screen ~/Library/Containers/com.docker.docker/Data/com.docker.amd64-linux/tty
# ls /var/lib/docker/volumes

References

Docker Tutorial for Beginners [FULL COURSE in 3 Hours]

Learning – Fat-Free PHP Framework (Bootstrap & Authentication)

Posted on by Bian Xi Leave a Comment on Learning – Fat-Free PHP Framework (Bootstrap & Authentication)

Learning - Fat-Free PHP Framework (Bootstrap & Authentication)

Bootstrap & Authentication

Bootstrap

Front design, https://getbootstrap.com, includes CCS, Components, etc.

Getting start has examples, sign in page can be downloaded as well.

Bootstrap CDN - Content delivery network online

Content to be included

Include css into the head tag of the page

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">

Include the java script just before the closing of body tag

<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>

Create structure for login page

  • Create app/views/login.htm

  • Create app/models/User.php

  • Create app/controllers/UserController.php

  • Create app/css/signin.css or a static folder for this

  • Copy bootstrap login sample html to login.htm

  • Copy bootstrap login sample css to signin.css

Create page

  • Clean up login.htm

  • Create UserController.php

class UserController extends Controller {
    function render() {
        $template = new Template;

        echo $template->render('login.htm');
    }
}
  • Update routes.ini

Add

GET /login=UserController->render

The login page built as https://localhost:8088/login

Customize login page

Remove Remember me and change variable names, and name attribute for input fields

Authentication

Update routes.ini

Add following line

POST /authenticate=UserController->authenticate

Update form in login.htm

<form class="form-signin" method="POST" action="/authenticate">

Create table

Create table user with id, username, password.

Create password

$ php -a
echo password_hash('f3password', PASSWORD_DEFAULT);
$2y......

Add user and password

Add an user with username=f3user and the password=$2y......

Create model for user

Create User.php by modifying Messages.php

class User extends DB\SQL\Mapper {
    public function __construct(DB\SQL $db) {
        parent::__construct($db, 'user');
    }

    public function getByName() {
        $this->load(array('username=?', $name));
    }

    public function all() {
        $this->load();
        return $this->query;
    }

    ...

*Note: Function all() can be used to assign to a variable, getByName() is return the object itself.

Add authenticate function in UserController class

function authenticate() {
    $username = $this->f3->get('POST.username');
    $password = $this->f3->get('POST.password');

    $user = new User($this->db);
    $user->getByName($username);

    if ($user->dry()) {
        // echo 'User does not exist.';
        $this->f3->reroute('/login')
    }

    if (password_verify($password, $user->password)) {
        // echo 'password OK';
        $this->f3->reroute('/');
    }
    else {
        // echo 'password NOT OK';
        $this->f3->reroute('/login')
    }
}

Note: dry() is db function.

Create session

Look for Fat-Free Session Handler

  • Enable cache

Add following line in config.ini

CACHE=true
  • Update index.php
...
new Session();

$f3->run();
  • Add following line in function authenticate in UseController class
if (password_verify($password, $user->password)) {
    $this->f3->set('SESSION.user', $user->usename)'
    $this->f3->reroute('/');
}

Update function beforeroute in Controller class

function beforeroute() {
    if ($this->f3->get('SESSION.user') === null) {
        $this->f3->reroute('/login');
        exit;
    }
}

Update UserController class

Because this update is in Controller class, so every page will go thru the verification. To ignore this behavior for login.htm, update UserController class. Adding following empty function to overwrite beforeroute()

function beforeroute() {
}

References

Adding Bootstrap and User Authentication to Fatfree PHP MVC Project

Learning – Fat-Free PHP Framework (Template Hierarchy)

Posted on by Bian Xi Leave a Comment on Learning – Fat-Free PHP Framework (Template Hierarchy)

Learning - Fat-Free PHP Framework (Template Hierarchy)

Template Hierarchy

Copy from bootstrap

  • Go to https://getbootstrap.com/getting-started/#examples, select Dashboard.

  • Copy dashboard.htm to app/views/dashboard.htm

  • Copy dashboard.css into app/css/dashboard.css

  • Create app/views/header.htm

    • Move html header in dashboard.htm into header.htm
    • Delete unecessary javascript for IE8
    • Update bootstrap.min.css to CBN version
    • Update the path of dashboard.css

  • Create app/views/layout.htm

<include href="header.htm" />
<include href="{{ @view }}" />
  • Update MainController class
<?php

class MainController extends Controller {
    function render() {
        $this->f3->set(&#039;view&#039;, &#039;dashoboard.htm&#039;);
        $template = new Template;
        echo $template->render(&#039;layout.htm&#039;);
    }
}
  • Create nav.htm

Move the body contents before Dashboard div into file nav.htm

<body>
    ...
    ...
            <ui class="nav nav-sidebar">
                ...
            </ui>
        </div>

References

Template Hierarchy in Fat-Free PHP Framework Sample Project