Category: proxmox

Move MicroSD boot proxmox to eMMC

Move MicroSD boot proxmox to eMMC

Steps

  • Manually deplicate partition from MicroSD to eMMC using fdisk, ignore the bios partition as EFI partition used.
  • Unmount old /boot/efi partition, then duplicate EFI partition using dd from MicroSD, this can keep UUID
  • Create PV on eMMC data partition and add it to pve VG
  • Move all data from old MicroSD partition to eMMC partition
    pvmove /dev/<MicroSD partition>
  • Check structure and UUID using following command
    lsblk -o +UUID
  • Remove MicroSD PV from pve VG using vgreduce, then use pvremove to remove PV from MicroSD
  • Mount new /boot/efi partition, then run grub-install to recreate grub.cfg file
  • Remove MicroSD from system, then reboot

References

Changing IP address for all nodes in Proxmox Cluster

Changing IP address for all nodes in Proxmox Cluster

Steps

  • Change IP in all nodes in following files /etc/network/interfaces and /etc/hosts
  • Change all IP address in /etc/pve/corosync.conf
  • Reboot all nodes.

Troubleshooting

If above failed during the synchronization, use following commands to fix it.

  • Stop cluster services on the node that wasn't synchronized
systemctl stop corosync.service
systemctl stop pve-cluster
  • Update the corosync.conf file manually
vi /etc/corosync/corosync.conf
  • Restart cluster services
systemctl start corosync.service
systemctl start pve-cluster

Verify configuration file again and cluster status

cat /etc/corosync/corosync.conf
pvecm status

Differences and Benefits Between i440fx and q35 in Proxmox

Differences and Benefits Between i440fx and q35 in Proxmox

My view

For application i440fx is enough, and it is simple, for hardware related, such as GPU passthru, then Q35 is better.

Switch between them

After switch between, the network interface name will be changed.

Q35

Q35 supports PCIe

  • Extended configuration space (MMCFG)
  • PCIe native hotplug
  • Advanced Error Reporting (AER)
  • Alternative Routing-ID Interpretation (ARI)
  • Native Power Management
  • Function Level Reset (FLR)
  • Address Translation Services (ATS)
  • AHCI storage controller
  • vIOMMU emulation
  • Secure Boot

Q35 limitations

  • No support for legacy guests (Windows XP/2000).
  • Questionable support for legacy QEMU devices.
  • Limited IO space can affect the number of devices used by a single Q35 machine

References

Differences/benefits between i440fx and q35 chipsets?
Q35 - QEMU
PCI vs PCI Express
PCI EXPRESS GUIDELINES

Fix Spice Certificate Issue in Proxmox

Fix Spice Certificate Issue in Proxmox

After changed the display to Spice and added Spice USB device, following error appeared.

swtpm_setup: Not overwriting existing state file.
kvm: warning: Spice: reds.c:2893:reds_init_ssl: Could not load certificates from /etc/pve/local/pve-ssl.pem
kvm: warning: Spice: error:0909006C:PEM routines:get_name:no start line
kvm: warning: Spice: error:140DC009:SSL routines:use_certificate_chain_file:PEM lib
kvm: failed to initialize spice server
stopping swtpm instance (pid 55260) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1

Update certificate also got following errors

root@pve01:~# pvecm updatecerts --force
(re)generate node files
generate new node certificate
Signature ok
subject=OU = PVE Cluster Node, O = Proxmox Virtual Environment, CN = pve01.xxx.net
Getting CA Private Key
CA certificate and CA private key do not match
139954545105792:error:06067099:digital envelope routines:EVP_PKEY_copy_parameters:different parameters:../crypto/evp/p_lib.c:93:
139954545105792:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:../crypto/x509/x509_cmp.c:303:
unable to generate pve ssl certificate:
command 'faketime yesterday openssl x509 -req -in /tmp/pvecertreq-56235.tmp -days 161 -out /etc/pve/nodes/pve01/pve-ssl.pem -CAkey /etc/pve/priv/pve-root-ca.key -CA /etc/pve/pve-root-ca.pem -CAserial /etc/pve/priv/pve-root-ca.srl -extfile /tmp/pvesslconf-56235.tmp' failed: exit code 1

In this case, remove keys and regenerate.

root@pve01:~# rm -f /etc/pve/pve-root-ca.pem /etc/pve/priv/pve-root-ca.* /etc/pve/local/pve-ssl.*
root@pve01:~# pvecm updatecerts -f
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
root@pve01:~# pvecm updatecerts -f
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
root@pve01:~# 

Now, problem fixed.

References

pveproxy fails to load local certificate chain after upgrade to pve 6

Reinstall Proxmox VE node in cluster

Reinstall Proxmox VE node in cluster

After node pve01 in Proxmox VE cluster crushed, reinstall new pve01 in same hardware.

Install PVE using ISO

This just follows the normal installation steps.

Try and error

Tried many steps, end up using following steps to add the replacement node.

  • Remove old node known host from all other nodes
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "<old_node>"
ssh-keygen -f "/root/.ssh/known_hosts" -R "<old_node>"
  • In the old node, run
pvecm delnode <old_node>
  • In the new node, run
pvecm add <existing_node>
pvecm updatecerts
  • Import old local pools
zpool import -f <old_local_pool>

Change expected votes

Run following commands to check and set acceptable votes in existing node in the cluster

pvecm status
pvecm expected 3

Remove old node

pvecm delnode pve01

Remove old ssh know host

ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "pve01"
ssh-keygen -f "/root/.ssh/known_hosts" -R "pve01"

or manual edit two files.

Add node

Run following command in NEW NODE

pvecm add <existing_node>

Sync certs

pvecm updatecerts

Test SSH key authentication

Make sure SSH Key authentication is working

Copy UI certificate

cp /etc/pve/nodes/pve02/pveproxy-ssl.* /etc/pve/nodes/pve01

Remove local-zfs filesystem

If the previous node was using zfs, then now change to ext4, local-zfs needs to be removed.

vi /etc/pve/storage.cfg

If need to disable cluster, following command can be used

systemctl stop pve-cluster
/usr/bin/pmxcfs -l

Restart pve01 cluster

systemctl restart pve-cluster

References

Cluster Manager
Wiki - Cluster Manager
Correct procedure for zpool removal

Convert Ubuntu VM to Proxmox

Convert Ubuntu VM to Proxmox

This is to describe how to convert Ubuntu VM to Proxmox.

VM creation

Following hardware options can be considered

  • BIOS: SeaBIOS (Should be able to see Grub menu)
  • Machine: Default (i440fx)
  • SCSI Controller: VirtIO SCSI (It might not be used as sata0 to be considered for disk)
  • Hard Disk (sata0): disk_image_file
  • Network Device (net0): vmxnet3=<mac_address> (This is default for VMware, can use other type too)

Convert the VMware disk to Proxmox disk and attach the disk to new VM

qm importdisk 121 ubuntu.vmdk pool240ssd --format qcow2

Attach the disk as sata0.

Boot

After boot up system show a GUI error screen, press Contrl + Alt + F3 to switch to console mode.

Note: Press Shift to active Grub Menu if required

Network

Find out new network interface UUID

nmcli conn

Change NetworkManager file name

cd /etc/NetworkManager/system-connections
mv Wired\ connection\ 1-<old_uuid>.nmconnection Wired\ connection\ 1-<new_uuid>.nmconnection

Update nmconnection file

[connection]
id=<new_interface_name>
uuid=<new_uuid>
type=ethernet
autoconnect-priority=-999
interface-name=<new_interface_name>
permissions=
timestamp=1628151710

[ethernet]
mac-address-blacklist=

[ipv4]
address1=192.168.1.232/24,192.168.1.254
dns=192.168.1.250;8.8.8.8;
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=disabled

[proxy]

Errors

No login GUI

After boot, only a white screen with error message appears, this was fixed by running apt update and upgrade

First, update /etc/apt/sources.list file, replace all repo URL to old-releases.ubuntu.com

Then run following commands

apt update
apt upgrade -y

References

Convert Oracle Linux 7.9 to Proxmox

Convert Oracle Linux 7.9 to Proxmox

This is to describe how to convert Oracle Linux 7.9 to Proxmox.

VM creation

Following hardware options can be considered

  • BIOS: SeaBIOS (Should be able to see Grub menu)
  • Machine: Default (i440fx)
  • SCSI Controller: VirtIO SCSI (It might not be used as sata0 to be considered for disk)
  • Hard Disk (sata0): disk_image_file
  • Network Device (net0): vmxnet3=<mac_address> (This is default for VMware, can use other type too)

Convert the VMware disk to Proxmox disk and attach the disk to new VM

qm importdisk 121 oracle18c.vmdk pool240ssd --format qcow2

Attach the disk as sata0.

Boot

Select item Oracle Linux Server (0-rescue-ed95572bd80641d79f83cd91e03c0283 with Linux) 7.9 from Grub menu to boot into rescue mode.

Note: Tried other option, all got error and unable to boot

Kernel

Find Kernel Package

Login as valid user, then find out the kernel to be used

rpm -q -a | grep kernel | sort

Got following list

kernel-3.10.0-1160.25.1.el7.x86_64
kernel-3.10.0-1160.36.2.el7.x86_64
kernel-3.10.0-1160.el7.x86_64
kernel-tools-3.10.0-1160.36.2.el7.x86_64
kernel-tools-libs-3.10.0-1160.36.2.el7.x86_64
kernel-uek-5.4.17-2102.203.6.el7uek.x86_64
kernel-uek-5.4.17-2102.204.4.2.el7uek.x86_64
kernel-uek-5.4.17-2102.204.4.4.el7uek.x86_64

Choose the latest one, which is also Unbreakable Enterprise Kernel

Recreate Grub kernel files

Find scripts in Kernel package

rpm -q kernel-uek-5.4.17-2102.204.4.4.el7uek.x86_64 --scripts

Following posttrans scriptlet shown

...
posttrans scriptlet (using /bin/sh):
/usr/sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 5.4.17-2102.204.4.4.el7uek.x86_64 || exit $?
/usr/sbin/new-kernel-pkg --package kernel --rpmposttrans 5.4.17-2102.204.4.4.el7uek.x86_64 || exit $?
...

Run above commands to rebuild Grub files, then reboot the system to the menu with kernel recreated.

Note: The error Unable to open file: /etc/keys/x509_ima.der (-2) can be ignored

Network

You can reconfigure network interface the same as VMware, this is to avoid reconfiguration of network settings

Interface Type

Check VMWare vmx file to find disk type, then set the same in Proxmox

ethernet0.virtualDev = "vmxnet3"

Mac Address

You can change Mac Address using the value in VMWare configuration

ethernet0.generatedAddress = "00:11c:22:33:44:55"

Interface Name

Find out the interface in /etc/sysconfig/network-scripts as below.

/etc/sysconfig/network-scripts/ifcfg-ens192

The interface name is ifcfg-ens192

Create the file /etc/udev/rules.d/70-custom-ifnames.rules with the following contents:

SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="00:11:22:33:44:55",ATTR{type}=="1",NAME="ens192"

Then reboot the server, then check the Interface and IP address using ip a command.

References

Consistent network interface device naming

Remove orphan disks in Proxmox

Remove orphan disks in Proxmox

If you canceled disk movement in Proxmox, an orphaned disk will be created. In such case, it will not be shown in VM hardware configuration, and it can not be removed from storage session. If you try to remove it, you will have an error as the disk is attached to a VM.

In order to remove it, rescan disk is required.

Rescan

Use following command can make the orphan disk reattached to the VM.

qm rescan --vmid <vm_id>

*Note: rescan should be done in the Proxmox node which contains VM configuration, otherwise, could not find the VM configuration file error will appear.

References

https://forum.proxmox.com/threads/cancelled-disk-move-orphaned-disk.96650/

Detect and Fix Proxmox file integrity issue

Detect and Fix Proxmox file integrity issue

The files can be easily corrupted if the Proxmox OS is installed on an USB device.

Detection

Because Proxmox uses Debian system, run following command to detect corruption

# dpkg --verify
??5?????? c /etc/apt/sources.list.d/pve-enterprise.list
??5?????? c /etc/lvm/lvm.conf
??5?????? c /etc/issue
# 

Above files can be ignored as reason below.

  • /etc/apt/sources.list.d/pve-enterprise.list: Not using enterprise repo
  • /etc/lvm/lvm.conf: LVM configuration file contains system specific info, such as UUIDs
  • /etc/issue: This file contains IP address of host

Fix

Find out the packages

# dpkg -S <list of file_name>

Reinstall the packages

apt --reinstall install <list of package_name>

References