Tag: network

Lost network after PVE rebooted

Posted on by Bian Xi Leave a Comment on Lost network after PVE rebooted

Lost network after PVE rebooted

Error

After reboot of PVE, network interfaces detected, but no link activated, ip address command shows all physical interfaces are down, and interfaces LED lights are shut off when loading OS.

Getting permission denied error when run ifup command, when using python3 /usr/sbin/ifup -a command, getting error as another instance of this application is already running

After using strace python3 /usr/sbin/ifup -a command, found that the command tried to access folder /run/network, but it doesn't exist.

Solution

Create folder /run/network after rebooted, then run command python3 /usr/sbin/ifup -a to bring up network manually.

Note: This is only a temporary solution, because the folder /run/network will disappear. Will troubleshoot again when got time.

References

Learning – Docker Swarm Network Drivers

Posted on by Bian Xi Leave a Comment on Learning – Docker Swarm Network Drivers

Learning - Docker Swarm Network Drivers

Bridge

The default network driver. Needs to map the port to host in order to access port of container.

Host

Removes network isolation between the container and the Docker host, and uses the host's networking directly. So the containers can not have port conflicting with other containers and also host.

The IP will be the same as host.

None

Disables all networking for containers. Usually used in conjunction with a custom network drive.

Overlay

Connect multiple Docker daemons together and enable swarm services to communicate with each other daemons.

Using this overlay network, the container on different hosts can communicate with each other.

Macvlan

Allow you to assign a MAC address to a container, making it appears as a physical device on the network. The Docker daemon routes traffic to container by their MAC addresses.

This allows container has different IP address on the host network.

Split Home Network

Posted on by Bian Xi Leave a Comment on Split Home Network

Split Home Network

Note: Tested using KVM network only

Let's say, you want to have a network which is different than the one ISP setup for you.

Why

  • You may want to have a guest network, which outside your home network for guests.
  • You may want to have a network with full functional network devices, firewall, DNS server, DHCP server, etc.
  • You may want to monitor the network inbound and outbound traffic.
  • You may want to setup PXE, iPXE, BOOTP environment, configure DHCP options such as TFTP server, etc.
  • You may want the VMs running on KVM routable network provide services to all machines, including Internet.
  • You may want to have more IPs, want to have IP segments.
  • You may want to have a more secure environment. In fact, if your router was setup by ISP, then they got full control of it, not you.

So, can you setup your own router to replace ISP route? Yes, you can, but you need to know whether you are allowed to do so by ISP. For my case, they asked me sign new agreement to say, service level drop and no technique support provided.

Before split

  • You have a network which connect to ISP via the ISP route, which have DHCP and DNS service.

Requirement

  • The ISP route can add route.
  • One device or router with two network interfaces, and they are or can be on separate LAN.

Simple solution

If you have another router, let's say RouterA

  • Connect WAN port of RouterA to ISP route LAN port
  • Configure ISP LAN IP on RouterA's WAN port, such as 192.168.1.100,
  • Make sure RouterA is in router mode, with it's own LAN, such as 192.168.100.0/24
  • Add a route in ISP Route, network 192.168.100.0/24 gateway 192.168.1.10.

Done.

Difficult solution

Replace RouterA with your own home build router in the simple solution.

Troubleshooting ping drop packet with same interval

Posted on by Bian Xi Leave a Comment on Troubleshooting ping drop packet with same interval

Troubleshooting ping drop packet wit same interval

The issue appear between 10G Qnap switch and the TPlink router. TPLink has a 2.5GB ethernet, which connects to 10G ethernet of Qnap switch. Sometimes, ping drop package, they have almost same interval!

% ping 192.168.1.254 
PING 192.168.1.254 (192.168.1.254): 56 data bytes
64 bytes from 192.168.1.254: icmp_seq=0 ttl=64 time=0.464 ms
Request timeout for icmp_seq 1
64 bytes from 192.168.1.254: icmp_seq=2 ttl=64 time=0.431 ms
64 bytes from 192.168.1.254: icmp_seq=3 ttl=64 time=0.399 ms
64 bytes from 192.168.1.254: icmp_seq=4 ttl=64 time=0.302 ms
64 bytes from 192.168.1.254: icmp_seq=5 ttl=64 time=0.356 ms
64 bytes from 192.168.1.254: icmp_seq=6 ttl=64 time=0.461 ms
64 bytes from 192.168.1.254: icmp_seq=7 ttl=64 time=0.495 ms
64 bytes from 192.168.1.254: icmp_seq=8 ttl=64 time=0.450 ms
64 bytes from 192.168.1.254: icmp_seq=9 ttl=64 time=0.573 ms
64 bytes from 192.168.1.254: icmp_seq=10 ttl=64 time=0.282 ms
64 bytes from 192.168.1.254: icmp_seq=11 ttl=64 time=0.374 ms
64 bytes from 192.168.1.254: icmp_seq=12 ttl=64 time=0.604 ms
64 bytes from 192.168.1.254: icmp_seq=13 ttl=64 time=0.438 ms
64 bytes from 192.168.1.254: icmp_seq=14 ttl=64 time=0.418 ms
64 bytes from 192.168.1.254: icmp_seq=15 ttl=64 time=0.446 ms
64 bytes from 192.168.1.254: icmp_seq=16 ttl=64 time=0.570 ms
64 bytes from 192.168.1.254: icmp_seq=17 ttl=64 time=0.753 ms
64 bytes from 192.168.1.254: icmp_seq=18 ttl=64 time=0.456 ms
64 bytes from 192.168.1.254: icmp_seq=19 ttl=64 time=0.530 ms
64 bytes from 192.168.1.254: icmp_seq=20 ttl=64 time=0.531 ms
64 bytes from 192.168.1.254: icmp_seq=21 ttl=64 time=0.480 ms
64 bytes from 192.168.1.254: icmp_seq=22 ttl=64 time=0.498 ms
64 bytes from 192.168.1.254: icmp_seq=23 ttl=64 time=0.498 ms
64 bytes from 192.168.1.254: icmp_seq=24 ttl=64 time=0.465 ms
Request timeout for icmp_seq 25
64 bytes from 192.168.1.254: icmp_seq=26 ttl=64 time=0.493 ms
64 bytes from 192.168.1.254: icmp_seq=27 ttl=64 time=0.520 ms
64 bytes from 192.168.1.254: icmp_seq=28 ttl=64 time=0.462 ms
64 bytes from 192.168.1.254: icmp_seq=29 ttl=64 time=0.459 ms
64 bytes from 192.168.1.254: icmp_seq=30 ttl=64 time=0.535 ms
64 bytes from 192.168.1.254: icmp_seq=31 ttl=64 time=0.468 ms
64 bytes from 192.168.1.254: icmp_seq=32 ttl=64 time=0.505 ms
64 bytes from 192.168.1.254: icmp_seq=33 ttl=64 time=0.539 ms
64 bytes from 192.168.1.254: icmp_seq=34 ttl=64 time=0.515 ms
64 bytes from 192.168.1.254: icmp_seq=35 ttl=64 time=0.504 ms
64 bytes from 192.168.1.254: icmp_seq=36 ttl=64 time=0.519 ms
64 bytes from 192.168.1.254: icmp_seq=37 ttl=64 time=0.415 ms
64 bytes from 192.168.1.254: icmp_seq=38 ttl=64 time=0.415 ms
64 bytes from 192.168.1.254: icmp_seq=39 ttl=64 time=0.384 ms
64 bytes from 192.168.1.254: icmp_seq=40 ttl=64 time=0.443 ms
64 bytes from 192.168.1.254: icmp_seq=41 ttl=64 time=0.456 ms
64 bytes from 192.168.1.254: icmp_seq=42 ttl=64 time=0.349 ms
64 bytes from 192.168.1.254: icmp_seq=43 ttl=64 time=0.345 ms
64 bytes from 192.168.1.254: icmp_seq=44 ttl=64 time=0.272 ms
64 bytes from 192.168.1.254: icmp_seq=45 ttl=64 time=0.456 ms
64 bytes from 192.168.1.254: icmp_seq=46 ttl=64 time=0.523 ms
64 bytes from 192.168.1.254: icmp_seq=47 ttl=64 time=0.553 ms
64 bytes from 192.168.1.254: icmp_seq=48 ttl=64 time=0.389 ms
Request timeout for icmp_seq 49
64 bytes from 192.168.1.254: icmp_seq=50 ttl=64 time=0.417 ms
64 bytes from 192.168.1.254: icmp_seq=51 ttl=64 time=0.433 ms
64 bytes from 192.168.1.254: icmp_seq=52 ttl=64 time=0.467 ms
64 bytes from 192.168.1.254: icmp_seq=53 ttl=64 time=0.417 ms
^C
--- 192.168.1.254 ping statistics ---
54 packets transmitted, 51 packets received, 5.6% packet loss
round-trip min/avg/max/stddev = 0.272/0.461/0.753/0.083 ms
%

Possible issue

After a month, I found that in Qnap web console, the flow control on the switch port, always flicking, sometimes enable, sometimes disable. Due to this behavior, I think could be the issue with the connection between them could try to re-established again and again.

Then I disabled flow-control from switch side, because I can not find the port settings in TPlink router.

Flow control

Enable flow control is to reduce packet dropping, but auto-negotiate can cause issue. Most of time both ends of ethernet can leave to auto-negotiate, but prefer to set one side manual if possible, especially two side has different highest speed.

References

Flow Control

Network filesystem timeout settings

Posted on by Bian Xi Leave a Comment on Network filesystem timeout settings

Network filesystem timeout settings

Network disruptions are always happening, network filesystems on different OS have different behaviors.

NFS

During Synology disk migration and SSD cache reconfiguration, my Fedora 34 on iSCSI mounted NFS disk kept hanging, I checked the default NFS mount options, then found that it was using hard option with out intr as below,

192.168.1.10:/volume1/kvm on /kvm type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.9,local_lock=none,addr=192.168.1.10)

I think maybe this is giving the factor of hanging.

iSCSI

After I changed NFS setting to soft, I suddenly realized that my iSCSI used by Fedora OS might not able to handle interupt as well, not sure whether iSCSI got similar options.

Samba on MacOS

My MacOS also got issue on samba filesystem, always disconnected after communication dropped, but my Windows machine has no such issue.

References

What are the differences between hard mount and soft mount?

Boot from small USB drive with iscsi root filesystem

Posted on by Bian Xi Leave a Comment on Boot from small USB drive with iscsi root filesystem

Boot from small USB drive with iscsi root filesystem

Boot from small size USB drive only holding boot partitions, rest of filesystems are on iscsi drives. Tested in EFI boot in Fedora 34.

Requirement

  • /boot partition can be 256M, can be very small, but better bigger
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sdb2                          428M  190M  212M  48% /boot
  • /boot/efi is an almost static very small filesystem, can be very small
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sdb1                          512M   31M  482M   6% /boot/efi

grub configuration

Define iscsi login info

GRUB_CMDLINE_LINUX="netroot=iscsi:<user>:<password>@<ip>::3260::<iqn> rd.iscsi.initiator=<client iqn> rhgb quiet ...

Define network interface with static ip 192.168.1.2, gateway 192.168.1.254, nameserver 192.168.1.1, interface enp0s10.

ip=192.168.1.2::192.168.1.254:255.255.255.0::enp0s10:off nameserver=192.168.1.1

Define network with bridge interface br0 on network interface enp0s10

ip=192.168.1.2::192.168.1.254:255.255.255.0::br0:off nameserver=192.168.1.1 ifname=enp0s10:xx:xx:xx:xx:xx:xx bridge=br0:enp0s10"

Update grub using following command

grub2-mkconfig -o /boot/grub2/grub.cfg