Two ESXi servers, for example esx01 and esx02.
The CPU of both esx01 and esx02 are same family.
Both esx01 and esx02 are having same shared storage.
Configure NFS or iSCSI in both ESXi servers.
Access storage, and check Hosts tab, both ESXi servers should be in the list.
Add VMkernel Network Adapter with vMotion service, can be on an existing standard switch.
VMware Tutorial No.45 | VMware vMotion Step by Step |vMotion in VMware 6.7 |vSphere vMotion GOVMLAB
.xz fileunxz < file.tar.xz > file.tarroot@proxmox:~# cat /sys/module/kvm_intel/parameters/nested
YIntel CPU
echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.confAMD CPU
echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.confInstall Module
modprobe -r kvm_intel
modprobe kvm_intelNote: more info, check https://pve.proxmox.com/wiki/Nested_Virtualization
Download ISO, such as VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso
General Tab
OS Tab
System Tab
Hard Disk Tab
CPU Tab
Memory Tab
Network Tab
Nested Virtualization
How to Install/use/test VMware vSphere 7.0 (ESXi 7.0) on Proxmox VE 6.3-3 (PVE)
When press FN-F11 key to VNC browser window of Proxmox VM console, it only shows the desktop.
To send the F11 key to ESXi installation screen, press FN-CMD-F11 instead.
Introduction to Ansible
pip3 install ansible
ansible --versionCreate a file called inventory
[example]
107.20.106.183ansible -i inventory example -m ping -u centosansible.cfg file[defaults]
INVENTORY=inventoryansible example -m ping -u centosansible example -a "date" -u centos
ansible example -a "free -h" -u centosIn fact, above commands used default module -m command, and the -a option is giving the command arguments.
Then initialize vagrant
vagrant init geerlingguy/centos7This will create a file called Vagrantfile in current directory.
vagrant upvagrant sshvagrant ssh-configThis configuration can be used to update ssh configuration
vagrant haltvagrant destroyVagrantfileVagrant.configure("2") do |config|
config.vm.box = "geerlingguy/centos7"
config.vm.provsion "ansible" do |ansible|
ansible.playbook = "playbook.yml"
end
endplaybook.yml---
- name: Set up NTP on all servers.
hosts: all
become: yes
tasks:
- name: Ensure NTP is installed.
yum: name=ntp state=present
- name: Ensure NTP is running.
service: name=ntpd state=started enabled=yesThe name is optional
- yum: name=ntp state=present
- service: name=ntpd state=started enabled=yesRun provision command
vagrant provisionThe command can be run many times without change the result if success before.
But following command in playbook will run every time when triggered playbook.
- command: yum install -y ntpTo overcome this, change to following
- shell: |
if ! rpm -qa | grep -qw ntp; then
yum install -y ntp
fi
minikube start --cpus 4 --memory 8192 --vm-driver hyperkit
helm ls
kubectl get pod
kubectl get svc
kubectl port-forward prometheus-kube-prometheus-prometheus 9090
kubectl port-forward prometheus-grafana 80ServiceMonitor is a custom Kubernetes component
kubectl get servicemonitor
kubectl get servicemonitor prometheus-kube-prometheus-grafana -oyaml
...
metadata:
labels:
release: prometheus
spec:
endpoints:
- path: /metrics
port: service
selector:
matchLabels:
app.kubernetes.io/instance: prometheus
app.kubernetes.io/name: grafanaCRD configuration
$ kubectl get crd
...
prometheuses.monitoring.coreos.com ...
...
$ kubectl get prometheuses.monitoring.coreos.com -oyaml
...
spec:
serviceMonitorSelector:
matchLabels:
release: prometheus
...mongodb-without-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongodb-deployment
labels:
app: mongodb
spec:
replicas: 1
selector:
matchLabels:
app: mongodb
template:
metadata:
labels:
app: mongodb
spec:
containers:
- name: mongodb
image: mongo
ports:
- containerPort: 27017
---
apiVersion: v1
kind: Service
metadata:
name: mongodb-service
spec:
selector:
app: mongodb
ports:
- protocol: TCP
port: 27017
targetPort: 27017kubectl apply -f mongodb-without-exporter.yaml
kubectl get podTranslator between apps data to Prometheus understandable metrics
Target (MongoDB App) <= fetches metrics <= converts to correct format <= expose /metrics <= Prometheus Server
MongoDB exporter (mongodb-exporter) can be downloaded from exporter site or dockerhub.
Exporters can be downloaded from https://prometheus.io/docs/instrumenting/exporters
Nodes exporter - translates metrics of cluster Nodes, exposes /metrics
prometheus-prometheus-node-exporter-8qvwn/metrics endpointhttps://github.com/prometheus-community/helm-charts
helm show values <chart-name>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm show values prometheus-community/prometheus-mongodb-exporter > values.yamlOverride values in values.yaml
mongodb:
uri: "mongodb://mongodb-service:27017"
serviceMonitor:
additionalLabels:
release: prometheuswith the label Prometheus automatically discovers a new ServiceMonitor in the cluster
$ helm install mongodb-exporter prometheus-community/prometheus-mongodb-exporter -f values.yaml
...
$ helm ls
mongodb-exporter
...
$ kubectl get pod
...
mongodb-exporter-prometheus-mongodb-exporter-75...
...
$ kubectl get svc
...
mongodb-exporter-prometheus-mongodb-exporter
...
$ kubectl get servicemonitor
...
mongodb-exporter-prometheus-mongodb-exporter
...
$ kubectl get servicemonitor mongodb-exporter-prometheus-mongodb-exporter -o yaml
...
metadata:
labels:
release: prometheus
.../metrics$ kubectl get svc
...
mongodb-exporter-prometheus-mongodb-exporter
...
kubectl port-forward service/mongodb-exporter-prometheus-mongodb-exporter 9216Access https://127.0.0.1:9216/metrics
The mongodb-exporter is added as targets in prometheus, because the label release: prometheus is set and auto discovered.
kubectl get deployment
kubectl port-forward deployment/prometheus-grafana 3000Prometheus Monitoring - Steps to monitor third-party apps using Prometheus Exporter | Part 2
Prometheus Server => push alerts => Alertmanager => Email, Slack, etc.
Prometheus Web UI
Grafana, etc.
Visualize the scraped data in UI
How to deploy the different parts in Kubernetes cluster?
Creating all configuration YAML files yourself and execute them in right order
Using an operator
Using Helm chart to deploy operator
$ kubectl get pod
$ helm install prometheus stable/prometheus-operator
$ kubectl get pod
NAME ...
alertmanager-prometheus-prometheus-oper-alertmanager-0
prometheus-grafana-67...
prometheus-kube-status-metrics-c6...
prometheus-prometheus-node-explorter-jr...
prometheus-prometheus-oper-operator-78...
prometheus-prometheus-prometheus-oper-prometheus-0...kubectl get allPrometheus Server
statefulset.apps/prometheus-prometheus-prometheus-oper-prometheusAlertmanager
statefulset.apps/alertmanager-prometheus-prometheus-oper-alertmanagerPrometheus Operator - created Prometheus and Alertmanager StatefulSet
deployment.apps/prometheus-prometheus-oper-operatorGrafana
deployment.apps/prometheus-grafanaKube State Metrics
deployment.apps/prometheus-kube-state-metricsCreated by Deployment
replicaset.apps/prometheus-prometheus-oper-operator...
replicaset.apps/prometheus-grafana...
replicaset.apps/prometheus-kube-state-metrics...daemonset.apps/prometheus-prometheus-node-exporterDaemonSet runs on every Worker Node
kubectl get configmapkubectl get secretfor Grafana
for Prometheus
for Operator
certificates
username & passwords
...
kubectl get crdextension of Kubernetes API
kubectl describe = container/image information
kubectl get statefulset
kubectl describe statefulset prometheus-prometheus-prometheus-oper-prometheus > prom.yaml
kubectl describe statefulset alertmanager-prometheus-prometheus-oper-alertmanager > alert.yamlkubectl get deployment
kubectl describe deployment prometheus-prometheus-oper-operator > oper.yamlContainers:
/etc/prometheus/certs/etc/prometheus/config_out/etc/prometheus/rules/.../prometheus/metricsThe two sidecar/help container *-reloader, they help reloading, responsible for reloading, when configuration files changes.
prometheus-config-reloader
rules-configmap-reloader
ConfigMap and Secret (States):
kubectl get configmap
kubectl get secretIn prom.yaml,
--config-file=/etc/promtheus/config/etc/prometheus/config from config/etc/prometheus/config_out from config_outconfig, it is a secretkubectl get secret prometheus-prometheus-prometheus-oper-prometheus -o yaml > secret.yamlapiVersion: v1
data:
prometheus.yaml.gz: ....In rules file rules-configmap-reloader
Mounts: /etc/prometheus/rules/prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0 from prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0
Volumes: ConfigMap prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0
kubectl get configmap prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0 -o yaml > config.yamlconfig.yaml rules fileapiVersion: v1
data:
default-prometheus-prometheus-oper-alertmanager.rules.yaml
groups:
- name: alertmanager.rules
rules:
- alert: AlertmanagerConfigInconsistent
...Containers:
alertmanager
config.file: /etc/alertmanager/config/alertmanager.yamlconfig-reloader
Containers:
prometheus-operator (orchestrator of monitoring stack)
tls-proxy
How to add/adjust alert rules?
How to adjust Prometheus configuration?
$ kubectl get service
...
prometheus-grafana ClusterIP ...ClusterIP = Internal Services
$ kubectl get deployment
...
prometheus-grafana
...
$ kubectl get pod
...
prometheus-grafana-67....
...
$ kubectl logs prometheus-grafana-67... -c grafana
...
... user=admin
...
... address=[::]:3000 ...
...port: 300
default user: admin
$ kubectl port-forward deployment/prometheus-grafana 3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000Then the grafana can be accessed via https://localhost:3000
The default admin password is "prom-operator", which can be found in chart: https://github.com/heim/charts/tree/master/stable/prometheus-operator#...
$ kubectl get pod
...
prometheus-kube-state-metrics-c6...
prometheus-prometheus-node-exporter-jr...
...$ kubectl get pod
...
prometheus-prometheus-prometheus-oper-prometheus-0
...
$ kubectl port-forward prometheus-prometheus-prometheus-oper-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090Then Prometheus UI can be accessed via https://localhost:9090/.
Setup Prometheus Monitoring on Kubernetes using Helm and Prometheus Operator | Part 1
Used for Stateful Applications on K8s
Control loop
Observe => Check Differences => Take Action => Observe ...
Data Persistence
more "hand-holding" needed
throughout whole lifecycle
all 3 replicas are different
own state and identity
order important
Process different for each application
So, no standard solution
manual intervention necessary
people, who "operate" these applications
can not archive automation, self-healing
To manage stateful application
Replaces human operator with software operator.
How to deploy the app?
How to create cluster of replicas?
How to recover?
tasks are automated and reusable
One 1 standard automated process
more complex/more environments => more benefits
watch for changes
Observe => Check Differences => Take Action => Observe ...
It is custom control loop
Custom Resource Definitions
Your own custom component
CRD's, StatefulSet, ConfigMap, Service, ...
automates entire lifecycle of the app it operates
For example: MySQL
OperatorHub.io
Operator SDK to create own operator
Monitoring Tool for
Highly dynamic container environments
Container & Microservices Infrastructure
Traditional, bare server
constantly monitor all the services
alert when crash
identify problem before
checking memory usage
notify administrator
Trigger alert at 50%
Monitor network loads
Does the actual monitoring work
Time Series Database
Storage - stores metrics data (CPU usage, No. of exception)
Data Retrieval Worker
Retrieval - pulls metrics data (Applications, Servers, ...)
Accepts PromQL queries
HTTP Server - accepts queries
Prometheus Web UI
Grafana
etc.
What does Prometheus monitor?
Which units are monitored of those targets?
Data Retrieval Worker => pull over HTTP => Target (Linux Server, External Service)
hostaddress/metrics/metrics endpoints by default/metricsList of official exporters ...
/metrics endpointUsing client libraries you can expose /metrics endpoint
Data Retrieval Worker pulls Targets /metrics
Amazon Cloud Watch, New Relic - Applications/Servers push to a centralized collection platform
What, when target only runs for a short time?
"short-lived job" => push metrics at exit => Pushgateway
Pushgateway <= pull <= Prometheus Server
Prometheus targets <= pull <= Prometheus Server
How does Prometheus know what to scrape and when?
prometheus.yml
service discovery
service discovery <= discover targets <= Prometheus Server
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
# - "first.rules"
# = "second.rules"
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
- job_name: node_exporter
scrape_interval: 1m
scrape_timeout: 1m
static_configs:
- targets: ['localhost:9100]/metrics endpointPrometheus Server => push alerts => Alertmanager => Email, Slack, etc.
Where does Prometheus store the data?
Local - Disk (HDD/SSD)
Remote Storage Systems
Custom Time Series Format
Prometheus Web UI => PromQL => Prometheus Server
Data Visualization Tools => PromQL => Prometheus Server
Query all HTTP status codes except 4xx ones
http_requests_total{status!~"4.."}Returns the 5-minute rate of the http_requests_total metric for the past 30mins
rate(http_requests_total[5m])[30m:]How Prometheus Monitoring works | Prometheus Architecture explained