Category: firewalld

Firewalld conflict between Docker and KVM

Firewalld conflict between Docker and KVM

After install docker, KVM bridge network can not access anything on network.

Identify

To identify the issue came from firewall and created by docker, the following facts had been collected.

  • After rebooted server, VM can access network, and restart firewalld without issue
  • After start docker service, VM can not access network any more
  • Then VM can access network after stop firewalld, but docker can not start container, because iptables is not accessible

Issue

No matter how to change iptables rules, and accept all traffics from everywhere, but VM was still isolated.

Commands used

Following commands were used for troubleshooting

Firewalld

In fact, there is no chain, rule, or passthroughs in firewall-cmd output. But after stop firewalld, the iptables rules became empty.

systemctl restart firewalld
firewall-cmd --list-all
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT
firewall-cmd --reload

firewall-cmd --permanent --direct --get-all-chains
firewall-cmd --permanent --direct --get-all-rules
firewall-cmd --permanent --direct --get-all-passthroughs
firewall-cmd --permanent --direct --remove-passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT

firewall-cmd --get-default-zone
firewall-cmd --get-active-zone
firewall-cmd --get-zones
firewall-cmd --get-services
firewall-cmd --list-all-zones

iptables

iptables -L -v
iptables -L -v FORWARD
iptables -I FORWARD -i br0 -o br0 -j ACCEPT
iptables -I FORWARD -j ACCEPT
iptables -I FORWARD 1 -j ACCEPT
iptables -d FORWARD 1
iptables-save
iptables-restore

others

Following commands are used to collect info and compare the differences between before and after.

brctl-show
ip a
netstat -rn

Potential issues

Following possiblities caused this issue or wrong troubleshooting

  • The iptables might not be used in the system, but the counters are refreshing.
  • Some rules in intables might not appearred in the iptables list

Debugging

For firewald, FIREWALLD_ARGS=--debug needs to be added into /etc/sysconfig/firewalld.

For iptables, -j LOG --log-prefix "rule description" needs to be added into iptables rules which require debugging.

Suggestions from others

Add ACCEPT rules

Run following commands to add ACCEPT rules

#!/bin/sh

# If I put bridge0 in trusted zone then firewalld allows anything from 
# bridge0 on both INPUT and FORWARD chains !
# So, I've put bridge0 back into the default public zone, and this script 
# adds rules to allow anything to and from bridge0 to be FORWARDed but not INPUT.

BRIDGE=bridge0
iptables -I FORWARD -i $BRIDGE -j ACCEPT
iptables -I FORWARD -o $BRIDGE -j ACCEPT

Conclusion

After many testings, found that docker is directly adding rules into iptables, not go thru firewalld. This can be noticed using following steps.

  1. Stop both firewalld and docker, iptables has no rules
  2. Start docker, iptables has only docker's rules
  3. Start filewalld, in short period time, LIBVIRT rules appear, after seconds, replaced by docker rules

Another testing

  1. Stop both firewalld and docker again
  2. Start firewalld, only the LIBVIRT rules appear
  3. Start docker, both docker and LIBVIRT rules appear

One issue was facing during reboot, if both docker and firewalld are enabled, the server might hung during reboot, maybe this is because root filesystem is on iSCSI disk, but can not confirm.

Above behaivor shows iptables is not supporting firewalld, which directly inserts rules into iptables periodically, which corrupts firewalld rules.

Solution

Run script

This solution disables firewalld and enable docker

systemctl disable firewalld
systemctl enable docker

Then run following command to add iptables rules to enable traffics

iptables -I FORWARD -i br0 -j ACCEPT
iptables -I FORWARD -o br0 -j ACCEPT

This script can be put in /etc/rc.local, which will be executed when during boot up.

Install iptables services

This solution also disables firewalld and enable docker as previous solution, then add two FORWARD rules into default iptables rules /etc/sysconfig/iptablesas below.

# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
-A FORWARD -o br0 -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
:OUTPUT ACCEPT [0:0]
#-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#-A INPUT -p icmp -j ACCEPT
#-A INPUT -i lo -j ACCEPT
#-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
#-A INPUT -j REJECT --reject-with icmp-host-prohibited
#-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Then both LIBVIRT and docker will add their rules later after system started.

Modify firewalld rules

For this solution, failed last time, I will try it again later.

firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i bridge0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o bridge0 -j ACCEPT

Feature

If possible, define firewalld rules which cover both LIBVIRT and docker.

References

Configure FirewallD to allow bridged virtual machine network access
Debug firewalld
How to configure iptables on CentOS

Less related topic
Do I need to restore iptable rules everytime on boot?
need iptables rule to accept all incoming traffic

Firewalld Basic

Firewalld Basic

Concept

Some basic concepts for firewalld to understand the commands

  • NIC
    Different NIC can have different zone assigned using nmcli command, if not specified, it is using default zone.
  • Zone
    By default, default zone is called default, this can be changed using firewalld command temporarily.
    To assign the default zoon to the zone that isn't named default, using nmcli command is required.
  • Service
  • Port

Start/Stop

# systemctl start firewalld
# systemctl enable firewalld

Default zone

Default zone is public when option --zone is not specified in command line.

Display the default zone

# firewall-cmd --get-default-zone
public

Display current settings

# firewall-cmd --list-all
public (default, active)
  interfaces: eno16777736
  sources:
  services: dhcpv6-client ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Display all zones defined by default

# firewall-cmd --list-all-zones
block
  interfaces:
  sources:
  services:
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:
  .....
  .....

Display allowed services on a specific zone

# firewall-cmd --list-service --zone=external
ssh

Change default zone

# firewall-cmd --set-default-zone=external
success

Change zone for an interface

Note: it's not changed permanently with "change-interface" even if added "--permanent" option

# firewall-cmd --change-interface=eth1 --zone=external
success
# firewall-cmd --list-all --zone=external
external (active)
  interfaces: eth1
  sources:
  services: ssh
  ports:
  masquerade: yes
  forward-ports:
  icmp-blocks:
  rich rules:

To change permanently, use nmcli like follows

# nmcli c mod eth1 connection.zone external
# firewall-cmd --get-active-zone
external
  interfaces: eth1
public
  interfaces: eth0

Services

Display services

# firewall-cmd --get-services
amanda-client bacula bacula-client dhcp dhcpv6 dhcpv6-client dns ftp high-availability http https imaps ipp ipp-client ipsec kerberos kpasswd ldap ldaps libvirt libvirt-tls mdns mountd ms-wbt mysql nfs ntp openvpn pmcd pmproxy pmwebapi pmwebapis pop3s postgresql proxy-dhcp radius rpc-bind samba samba-client smtp ssh telnet tftp tftp-client transmission-client vnc-server wbem-https

Service definition files are XML files in /usr/lib/firewalld/services

# ls /usr/lib/firewalld/services
amanda-client.xml      ipp-client.xml   mysql.xml       rpc-bind.xml
bacula-client.xml      ipp.xml          nfs.xml         samba-client.xml
bacula.xml             ipsec.xml        ntp.xml         samba.xml
dhcpv6-client.xml      kerberos.xml     openvpn.xml     smtp.xml
dhcpv6.xml             kpasswd.xml      pmcd.xml        ssh.xml
dhcp.xml               ldaps.xml        pmproxy.xml     telnet.xml
dns.xml                ldap.xml         pmwebapis.xml   tftp-client.xml
ftp.xml                libvirt-tls.xml  pmwebapi.xml    tftp.xml
high-availability.xml  libvirt.xml      pop3s.xml       transmission-client.xml
https.xml              mdns.xml         postgresql.xml  vnc-server.xml
http.xml               mountd.xml       proxy-dhcp.xml  wbem-https.xml
imaps.xml              ms-wbt.xml       radius.xml

Add or remove services temporarily.

# firewall-cmd --add-service=http
success
# firewall-cmd --list-service
dhcpv6-client http ssh
...
...
# firewall-cmd --remove-service=http
success
# firewall-cmd --list-service
dhcpv6-client ssh

Add or remove services permanently

Note: Reload the Firewalld is required to enable the change

# firewall-cmd --add-service=http --permanent
success
# firewall-cmd --reload
success
# firewall-cmd --list-service
dhcpv6-client http ssh

Ports

Add or remove ports temporarily.

# firewall-cmd --add-port=465/tcp
success
# firewall-cmd --list-port
465/tcp
# firewall-cmd --remove-port=465/tcp
success
# firewall-cmd --list-port

Add or remove ports permanently

# firewall-cmd --add-port=465/tcp --permanent
success
# firewall-cmd --reload
success
# firewall-cmd --list-port
465/tcp

ICMP

Add or remove ICMP types.

# firewall-cmd --add-icmp-block=echo-request
success
# firewall-cmd --list-icmp-blocks
echo-request
# firewall-cmd --remove-icmp-block=echo-request
success
# firewall-cmd --list-icmp-blocks

Display ICMP types

# firewall-cmd --get-icmptypes
destination-unreachable echo-reply echo-request parameter-problem redirect
router-advertisement router-solicitation source-quench time-exceeded 

References

Firewalld : Basic Operation