Firewalld conflict between Docker and KVM
After install docker, KVM bridge network can not access anything on network.
Identify
To identify the issue came from firewall and created by docker, the following facts had been collected.
- After rebooted server, VM can access network, and restart firewalld without issue
- After start docker service, VM can not access network any more
- Then VM can access network after stop firewalld, but docker can not start container, because iptables is not accessible
Issue
No matter how to change iptables rules, and accept all traffics from everywhere, but VM was still isolated.
Commands used
Following commands were used for troubleshooting
Firewalld
In fact, there is no chain, rule, or passthroughs in firewall-cmd output. But after stop firewalld, the iptables rules became empty.
systemctl restart firewalld
firewall-cmd --list-all
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT
firewall-cmd --reload
firewall-cmd --permanent --direct --get-all-chains
firewall-cmd --permanent --direct --get-all-rules
firewall-cmd --permanent --direct --get-all-passthroughs
firewall-cmd --permanent --direct --remove-passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT
firewall-cmd --get-default-zone
firewall-cmd --get-active-zone
firewall-cmd --get-zones
firewall-cmd --get-services
firewall-cmd --list-all-zonesiptables
iptables -L -v
iptables -L -v FORWARD
iptables -I FORWARD -i br0 -o br0 -j ACCEPT
iptables -I FORWARD -j ACCEPT
iptables -I FORWARD 1 -j ACCEPT
iptables -d FORWARD 1
iptables-save
iptables-restoreothers
Following commands are used to collect info and compare the differences between before and after.
brctl-show
ip a
netstat -rnPotential issues
Following possiblities caused this issue or wrong troubleshooting
- The iptables might not be used in the system, but the counters are refreshing.
- Some rules in intables might not appearred in the iptables list
Debugging
For firewald, FIREWALLD_ARGS=--debug needs to be added into /etc/sysconfig/firewalld.
For iptables, -j LOG --log-prefix "rule description" needs to be added into iptables rules which require debugging.
Suggestions from others
Add ACCEPT rules
Run following commands to add ACCEPT rules
#!/bin/sh
# If I put bridge0 in trusted zone then firewalld allows anything from
# bridge0 on both INPUT and FORWARD chains !
# So, I've put bridge0 back into the default public zone, and this script
# adds rules to allow anything to and from bridge0 to be FORWARDed but not INPUT.
BRIDGE=bridge0
iptables -I FORWARD -i $BRIDGE -j ACCEPT
iptables -I FORWARD -o $BRIDGE -j ACCEPTConclusion
After many testings, found that docker is directly adding rules into iptables, not go thru firewalld. This can be noticed using following steps.
- Stop both firewalld and docker, iptables has no rules
- Start docker, iptables has only docker's rules
- Start filewalld, in short period time, LIBVIRT rules appear, after seconds, replaced by docker rules
Another testing
- Stop both firewalld and docker again
- Start firewalld, only the LIBVIRT rules appear
- Start docker, both docker and LIBVIRT rules appear
One issue was facing during reboot, if both docker and firewalld are enabled, the server might hung during reboot, maybe this is because root filesystem is on iSCSI disk, but can not confirm.
Above behaivor shows iptables is not supporting firewalld, which directly inserts rules into iptables periodically, which corrupts firewalld rules.
Solution
Run script
This solution disables firewalld and enable docker
systemctl disable firewalld
systemctl enable dockerThen run following command to add iptables rules to enable traffics
iptables -I FORWARD -i br0 -j ACCEPT
iptables -I FORWARD -o br0 -j ACCEPTThis script can be put in /etc/rc.local, which will be executed when during boot up.
Install iptables services
This solution also disables firewalld and enable docker as previous solution, then add two FORWARD rules into default iptables rules /etc/sysconfig/iptablesas below.
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
-A FORWARD -o br0 -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
:OUTPUT ACCEPT [0:0]
#-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#-A INPUT -p icmp -j ACCEPT
#-A INPUT -i lo -j ACCEPT
#-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
#-A INPUT -j REJECT --reject-with icmp-host-prohibited
#-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMITThen both LIBVIRT and docker will add their rules later after system started.
Modify firewalld rules
For this solution, failed last time, I will try it again later.
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPTFeature
If possible, define firewalld rules which cover both LIBVIRT and docker.
References
Configure FirewallD to allow bridged virtual machine network access
Debug firewalld
How to configure iptables on CentOS
Less related topic
Do I need to restore iptable rules everytime on boot?
need iptables rule to accept all incoming traffic