Missing iSCSI module in Ubuntu 20.10

Recently, I have upgraded Ubuntu 20.04 to 20.10, then iscsi_tcp module is missing, which caused iscsi LUN could not be loaded.

I checked /var/log/syslog, showed zram (reported by init-zram-swapping) and autofs (reported by containerd) module were missing too.

Error

Looking at error message shown during iscsi discovery, the command was looking for file iscsi_tcp.ko. Which can not be found in the system.

Update

The latest missing package can be installed using following command.

apt install linux-modules-extra-raspi

apt search

When performing apt search, shows three packages should include such file, but when listing all files in those packages, there is no iscsi_tcp.ko file at all.

# apt search iscsi_tcp.ko
Sorting... Done
Full Text Search... Done
libopeniscsiusr/impish,now 2.1.4-0ubuntu1 arm64 [installed]
  iSCSI userspace library

libopeniscsiusr-dev/impish 2.1.4-0ubuntu1 all
  iSCSI userspace library headers

open-iscsi/impish,now 2.1.4-0ubuntu1 arm64 [installed]
  iSCSI initiator tools

Reinstall OS

Then I decided to install a new fresh OS, but got the same result.

Search in ubuntu website

When searching packages content file iscsi_tcp.ko from ubuntu website, found there is no module package called linux-modules-extra for kernel 5.13.0-1009-raspi, the latest is 5.13.0-1008-raspi, which is for 20.04.

https://packages.ubuntu.com/search?suite=impish§ion=all&arch=any&keywords=iscsi_tcp.ko&searchon=contents

In other words, 20.10 doesn't have iscsi_tcp module, at all.

# apt install linux-modules-extra
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-modules-extra
# 

Search in ubuntu repo

Then I go to ubuntu packages repo site below

http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/

...
/lib/modules/5.13.0-1008-oracle/kernel/drivers/scsi/libiscsi_tcp.ko     linux-modules-5.13.0-1008-oracle [arm64, amd64]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/iscsi_tcp.ko  linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/libiscsi_tcp.ko   linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/iscsi_tcp.ko     linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/libiscsi_tcp.ko  linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-19-generic-64k/kernel/drivers/scsi/iscsi_tcp.ko     linux-modules-5.13.0-19-generic-64k [arm64] 
...

found that there are linux-modules-extra package for both 1009 and 1010.

...
[ ] linux-modules-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb    2021-10-29 10:29    23M
[ ] linux-modules-extra-5.13.0-1008-raspi-nolpae_5.13.0-1008.9_armhf.deb    2021-09-29 20:44    21M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_arm64.deb   2021-09-29 20:43    23M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_armhf.deb   2021-09-29 20:44    21M
[ ] linux-modules-extra-5.13.0-1009-raspi-nolpae_5.13.0-1009.10_armhf.deb   2021-10-28 08:33    21M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb  2021-10-28 08:33    23M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_armhf.deb  2021-10-28 08:33    21M
[ ] linux-modules-extra-5.13.0-1010-raspi-nolpae_5.13.0-1010.11_armhf.deb   2021-10-29 10:29    21M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_arm64.deb  2021-10-29 10:29    23M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb  2021-10-29 10:29    21M
[ ] linux-raspi-headers-5.4.0-1008_5.4.0-1008.8_arm64.deb   2020-04-10 11:18    11M
...

Install linux-modules-extra

Then I decided to do download the package and check the contents

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb
dpkg -c linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb | grep iscsi

found the iscsi_tcp.ko is in the package

Install package via dpkg

dpkg -i linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb

Testing

Tested using iscsiadm, shows the problem fixed

iscsiadm --mode discovery --op update --type sendtargets --portal 192.168.1.17

In fact, this also fixed some other similar issues, such as, zram model missing, etc.

Reconfigure btrfs filesystem

In this system, the btrfs filesystem which was mirrored to iscsi LUN using bcache, the iscsi LUN was removed using following command during troubleshooting

btrfs balance start -f -sconvert=single -mconvert=single -dconvert=single /app
btrfs device remove /dev/bcache0 /app

Now use following command to reconfigure it back.

btrfs device add -f /dev/bcache0 /app
btrfs balance start -dconvert=raid1 -mconvert=raid1 /app

Configure module auto-loading

In order to load modules automatically, add iscsi_tcpline in /etc/modules. This may also avoid apt autoremove command removes linux-modules-extra package.

Possible cause

I think the issue was caused by spliting the kernel module package in to two in 1008, because linux-modules-extra only exists in the version 1008 in package searching list. I think ubnutu forgot this when releasing 1009. At this time, apt hasn't auto updated system to 1010.

References

Ubuntu Raspberry Package Repo
Ubuntu Package Searching
How to list files of a Debian package without install
How to install specific Ubuntu packages, with exact version?