Firewalld conflict between Docker and KVM
After install docker, KVM bridge network can not access anything on network.
To identify the issue came from firewall and created by docker, the following facts had been collected.
- After rebooted server, VM can access network, and restart firewalld without issue
- After start docker service, VM can not access network any more
- Then VM can access network after stop firewalld, but docker can not start container, because iptables is not accessible
No matter how to change iptables rules, and accept all traffics from everywhere, but VM was still isolated.
Following commands were used for troubleshooting
In fact, there is no chain, rule, or passthroughs in firewall-cmd output. But after stop firewalld, the iptables rules became empty.
systemctl restart firewalld firewall-cmd --list-all firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT firewall-cmd --reload firewall-cmd --permanent --direct --get-all-chains firewall-cmd --permanent --direct --get-all-rules firewall-cmd --permanent --direct --get-all-passthroughs firewall-cmd --permanent --direct --remove-passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT firewall-cmd --get-default-zone firewall-cmd --get-active-zone firewall-cmd --get-zones firewall-cmd --get-services firewall-cmd --list-all-zones
iptables -L -v iptables -L -v FORWARD iptables -I FORWARD -i br0 -o br0 -j ACCEPT iptables -I FORWARD -j ACCEPT iptables -I FORWARD 1 -j ACCEPT iptables -d FORWARD 1 iptables-save iptables-restore
Following commands are used to collect info and compare the differences between before and after.
brctl-show ip a netstat -rn
Following possiblities caused this issue or wrong troubleshooting
- The iptables might not be used in the system, but the counters are refreshing.
- Some rules in intables might not appearred in the iptables list
FIREWALLD_ARGS=--debug needs to be added into
-j LOG --log-prefix "rule description" needs to be added into iptables rules which require debugging.
Suggestions from others
Add ACCEPT rules
Run following commands to add ACCEPT rules
#!/bin/sh # If I put bridge0 in trusted zone then firewalld allows anything from # bridge0 on both INPUT and FORWARD chains ! # So, I've put bridge0 back into the default public zone, and this script # adds rules to allow anything to and from bridge0 to be FORWARDed but not INPUT. BRIDGE=bridge0 iptables -I FORWARD -i $BRIDGE -j ACCEPT iptables -I FORWARD -o $BRIDGE -j ACCEPT
After many testings, found that docker is directly adding rules into iptables, not go thru firewalld. This can be noticed using following steps.
- Stop both firewalld and docker, iptables has no rules
- Start docker, iptables has only docker's rules
- Start filewalld, in short period time, LIBVIRT rules appear, after seconds, replaced by docker rules
- Stop both firewalld and docker again
- Start firewalld, only the LIBVIRT rules appear
- Start docker, both docker and LIBVIRT rules appear
One issue was facing during reboot, if both docker and firewalld are enabled, the server might hung during reboot, maybe this is because root filesystem is on iSCSI disk, but can not confirm.
Above behaivor shows iptables is not supporting firewalld, which directly inserts rules into iptables periodically, which corrupts firewalld rules.
This solution disables firewalld and enable docker
systemctl disable firewalld systemctl enable docker
Then run following command to add iptables rules to enable traffics
iptables -I FORWARD -i br0 -j ACCEPT iptables -I FORWARD -o br0 -j ACCEPT
This script can be put in /etc/rc.local, which will be executed when during boot up.
Install iptables services
This solution also disables firewalld and enable docker as previous solution, then add two FORWARD rules into default iptables rules
# sample configuration for iptables service # you can edit this manually or use system-config-firewall # please do not ask us to add additional ports/services to this default configuration *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] -A FORWARD -o br0 -j ACCEPT -A FORWARD -i br0 -j ACCEPT :OUTPUT ACCEPT [0:0] #-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT #-A INPUT -p icmp -j ACCEPT #-A INPUT -i lo -j ACCEPT #-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT #-A INPUT -j REJECT --reject-with icmp-host-prohibited #-A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT
Then both LIBVIRT and docker will add their rules later after system started.
Modify firewalld rules
For this solution, failed last time, I will try it again later.
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT
If possible, define firewalld rules which cover both LIBVIRT and docker.
Configure FirewallD to allow bridged virtual machine network access
How to configure iptables on CentOS
Less related topic
Do I need to restore iptable rules everytime on boot?
need iptables rule to accept all incoming traffic