Category: kvm

Firewalld conflict between Docker and KVM

Posted on by Bian Xi Leave a Comment on Firewalld conflict between Docker and KVM

Firewalld conflict between Docker and KVM

After install docker, KVM bridge network can not access anything on network.

Identify

To identify the issue came from firewall and created by docker, the following facts had been collected.

  • After rebooted server, VM can access network, and restart firewalld without issue
  • After start docker service, VM can not access network any more
  • Then VM can access network after stop firewalld, but docker can not start container, because iptables is not accessible

Issue

No matter how to change iptables rules, and accept all traffics from everywhere, but VM was still isolated.

Commands used

Following commands were used for troubleshooting

Firewalld

In fact, there is no chain, rule, or passthroughs in firewall-cmd output. But after stop firewalld, the iptables rules became empty.

systemctl restart firewalld
firewall-cmd --list-all
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT
firewall-cmd --reload

firewall-cmd --permanent --direct --get-all-chains
firewall-cmd --permanent --direct --get-all-rules
firewall-cmd --permanent --direct --get-all-passthroughs
firewall-cmd --permanent --direct --remove-passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT

firewall-cmd --get-default-zone
firewall-cmd --get-active-zone
firewall-cmd --get-zones
firewall-cmd --get-services
firewall-cmd --list-all-zones

iptables

iptables -L -v
iptables -L -v FORWARD
iptables -I FORWARD -i br0 -o br0 -j ACCEPT
iptables -I FORWARD -j ACCEPT
iptables -I FORWARD 1 -j ACCEPT
iptables -d FORWARD 1
iptables-save
iptables-restore

others

Following commands are used to collect info and compare the differences between before and after.

brctl-show
ip a
netstat -rn

Potential issues

Following possiblities caused this issue or wrong troubleshooting

  • The iptables might not be used in the system, but the counters are refreshing.
  • Some rules in intables might not appearred in the iptables list

Debugging

For firewald, FIREWALLD_ARGS=--debug needs to be added into /etc/sysconfig/firewalld.

For iptables, -j LOG --log-prefix "rule description" needs to be added into iptables rules which require debugging.

Suggestions from others

Add ACCEPT rules

Run following commands to add ACCEPT rules

#!/bin/sh

# If I put bridge0 in trusted zone then firewalld allows anything from 
# bridge0 on both INPUT and FORWARD chains !
# So, I've put bridge0 back into the default public zone, and this script 
# adds rules to allow anything to and from bridge0 to be FORWARDed but not INPUT.

BRIDGE=bridge0
iptables -I FORWARD -i $BRIDGE -j ACCEPT
iptables -I FORWARD -o $BRIDGE -j ACCEPT

Conclusion

After many testings, found that docker is directly adding rules into iptables, not go thru firewalld. This can be noticed using following steps.

  1. Stop both firewalld and docker, iptables has no rules
  2. Start docker, iptables has only docker's rules
  3. Start filewalld, in short period time, LIBVIRT rules appear, after seconds, replaced by docker rules

Another testing

  1. Stop both firewalld and docker again
  2. Start firewalld, only the LIBVIRT rules appear
  3. Start docker, both docker and LIBVIRT rules appear

One issue was facing during reboot, if both docker and firewalld are enabled, the server might hung during reboot, maybe this is because root filesystem is on iSCSI disk, but can not confirm.

Above behaivor shows iptables is not supporting firewalld, which directly inserts rules into iptables periodically, which corrupts firewalld rules.

Solution

Run script

This solution disables firewalld and enable docker

systemctl disable firewalld
systemctl enable docker

Then run following command to add iptables rules to enable traffics

iptables -I FORWARD -i br0 -j ACCEPT
iptables -I FORWARD -o br0 -j ACCEPT

This script can be put in /etc/rc.local, which will be executed when during boot up.

Install iptables services

This solution also disables firewalld and enable docker as previous solution, then add two FORWARD rules into default iptables rules /etc/sysconfig/iptablesas below.

# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
-A FORWARD -o br0 -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
:OUTPUT ACCEPT [0:0]
#-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#-A INPUT -p icmp -j ACCEPT
#-A INPUT -i lo -j ACCEPT
#-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
#-A INPUT -j REJECT --reject-with icmp-host-prohibited
#-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Then both LIBVIRT and docker will add their rules later after system started.

Modify firewalld rules

For this solution, failed last time, I will try it again later.

firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT

Feature

If possible, define firewalld rules which cover both LIBVIRT and docker.

References

Configure FirewallD to allow bridged virtual machine network access
Debug firewalld
How to configure iptables on CentOS

Less related topic
Do I need to restore iptable rules everytime on boot?
need iptables rule to accept all incoming traffic

VM setup in TrueNAS

Posted on by Bian Xi Leave a Comment on VM setup in TrueNAS

VM setup in TrueNAS

Setup Bridge Network for HOST

If the VM interface created on physical interface, the VM will not be able to access host, which can not use any services provided by TrueNAS.

To fix this issue, Bridge Network is necessary to be used in host. To migrate physical network to bridge network, following steps required.

Note: complete all steps before click on Test Changes

  • Remove IP address from physical interface (bond0)
  • Create a bridge interface called br0, attach physical interface (bond0)
  • Add IP on bridge interface
  • Click on Test Changes
  • Wait till the IP address reachable again
  • Then Make change permanently by click the same button again.

Create VM

Select CPU Passthru should be faster.
Select vio devices for hard disk and network.

Download driver from Fedora

Download both storage driver and network driver from following website.
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.139-1/virtio-win-0.1.139.iso

Load driver to detect hard disk

Select driver CD and point to \viostor\w10\amd64 folder for storage device driver.

Update driver after installation

For network card driver, update using driver CD.

References

10 Easy Steps To Install Windows 10 on Linux KVM – KVM Windows

KVM setup in Fedora

Posted on by Bian Xi Leave a Comment on KVM setup in Fedora

KVM setup in Fedora

Commands

virsh list --all
virsh start <vm>
virsh start <vm> --console
virsh stop <vm>

KVM vs XEN

KVM isn't kernel specific, XEN required special kernel, so XEN could have kernel upgrade issue.

Bridge Network

When creating bridging network, if grub is used to create network interface, then Network Manager should not be used to create same interface. If Network Manager used, same network interface will be appear in ifconfig -a command output twice, One is created by NetworkManager, another is created by grub. If bridge network interface created on top of grub created interface, the IP address will be still assigned to grub created interface.

In order to avoid above issue, following line in /etc/default/grub to create network interface with bridging network interface br0.

GRUB_CMDLINE_LINUX=" ... ip=192.168.1.9::192.168.1.254:255.255.255.0::br0:off nameserver=192.168.1.250 ifname=enp0s10:00:26:4a:18:82:c6 bridge=br0:enp0s10"

After br0 created, KVM manager can select bridging network for vm creation.

Update grub using following command

grub2-mkconfig -o /boot/grub2/grub.cfg

Download driver

Both Windows disk controller driver and ethernet driver can be downloaded from Fedora Website, https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.139-1/virtio-win-0.1.139.iso, and add additional CD-ROM to point to this iso.

Create VM

Using Virtual Machine Manager

Create VM requires add storage, if the storage file doesn't exist, need to select the storage location, and also input the size of disk which located above the location selection box.

Using command line

To create Ubuntu VM from local image,

virt-install \
--name ubuntu2104 \
--ram 3072 \
--vcpus 2 \
--disk path=/kvm/ubuntu2104.qcow2,size=20 \
--os-variant ubuntu20.04 \
--os-type linux \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--cdrom /kvm/ubuntu-21.04-live-server-amd64.iso \
--boot kernel=casper/vmlinuz,initrd=casper/initrd,kernel_args="console=ttyS0"

To create Fedora VM from remote server

virt-install \
--name fed34 \
--ram 2048 \
--vcpus 2 \
--disk path=/kvm/fed34.img,size=20 \
--os-variant fedora34 \
--os-type linux \
--network bridge=virbr0 \
--graphics none \
--console pty,target_type=serial \
--location 'https://mirror.arizona.edu/fedora/linux/releases/34/Server/x86_64/os/' \
--extra-args 'console=ttyS0,115200n8 serial'

Create Windows 10 VM

virt-install \
   --ram=4096 \
   --name=windows10 \
   --os-type=win10 \
   --network network=default \
   --disk path=/kvm/kvm-windows10.img,size=100 \
   --cdrom=/kvm/virtio-win-0.1.139.iso \
   --graphics spice

Cons

  • Cannot select type of CPU or Passthru mode
  • Cannot select type of disk controller type to use virtual device driver.

References

10 Easy Steps To Install Windows 10 on Linux KVM – KVM Windows