Tag: linux

Escape Percent-signs(%) in crontab

Escape Percent-signs (%) in crontab

In order to input % character as command parameter in cron task, it needs to be escaped using backslash ().

man (5) crontab:

Percent-signs (%) in the command, unless escaped with backslash (\), 
will be changed into newline characters, and all data after the 
first % will be sent to the command as standard input.

Missing iSCSI module in Ubuntu 20.10

Missing iSCSI module in Ubuntu 20.10

Recently, I have upgraded Ubuntu 20.04 to 20.10, then iscsi_tcp module is missing, which caused iscsi LUN could not be loaded.

I checked /var/log/syslog, showed zram (reported by init-zram-swapping) and autofs (reported by containerd) module were missing too.


Looking at error message shown during iscsi discovery, the command was looking for file iscsi_tcp.ko. Which can not be found in the system.


The latest missing package can be installed using following command.

apt install linux-modules-extra-raspi

apt search

When performing apt search, shows three packages should include such file, but when listing all files in those packages, there is no iscsi_tcp.ko file at all.

# apt search iscsi_tcp.ko
Sorting... Done
Full Text Search... Done
libopeniscsiusr/impish,now 2.1.4-0ubuntu1 arm64 [installed]
  iSCSI userspace library

libopeniscsiusr-dev/impish 2.1.4-0ubuntu1 all
  iSCSI userspace library headers

open-iscsi/impish,now 2.1.4-0ubuntu1 arm64 [installed]
  iSCSI initiator tools

Reinstall OS

Then I decided to install a new fresh OS, but got the same result.

Search in ubuntu website

When searching packages content file iscsi_tcp.ko from ubuntu website, found there is no module package called linux-modules-extra for kernel 5.13.0-1009-raspi, the latest is 5.13.0-1008-raspi, which is for 20.04.


In other words, 20.10 doesn't have iscsi_tcp module, at all.

# apt install linux-modules-extra
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-modules-extra

Search in ubuntu repo

Then I go to ubuntu packages repo site below


/lib/modules/5.13.0-1008-oracle/kernel/drivers/scsi/libiscsi_tcp.ko     linux-modules-5.13.0-1008-oracle [arm64, amd64]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/iscsi_tcp.ko  linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi-nolpae/kernel/drivers/scsi/libiscsi_tcp.ko   linux-modules-extra-5.13.0-1008-raspi-nolpae [armhf]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/iscsi_tcp.ko     linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-1008-raspi/kernel/drivers/scsi/libiscsi_tcp.ko  linux-modules-extra-5.13.0-1008-raspi [armhf, arm64]
/lib/modules/5.13.0-19-generic-64k/kernel/drivers/scsi/iscsi_tcp.ko     linux-modules-5.13.0-19-generic-64k [arm64] 

found that there are linux-modules-extra package for both 1009 and 1010.

[ ] linux-modules-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb    2021-10-29 10:29    23M
[ ] linux-modules-extra-5.13.0-1008-raspi-nolpae_5.13.0-1008.9_armhf.deb    2021-09-29 20:44    21M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_arm64.deb   2021-09-29 20:43    23M
[ ] linux-modules-extra-5.13.0-1008-raspi_5.13.0-1008.9_armhf.deb   2021-09-29 20:44    21M
[ ] linux-modules-extra-5.13.0-1009-raspi-nolpae_5.13.0-1009.10_armhf.deb   2021-10-28 08:33    21M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb  2021-10-28 08:33    23M
[ ] linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_armhf.deb  2021-10-28 08:33    21M
[ ] linux-modules-extra-5.13.0-1010-raspi-nolpae_5.13.0-1010.11_armhf.deb   2021-10-29 10:29    21M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_arm64.deb  2021-10-29 10:29    23M
[ ] linux-modules-extra-5.13.0-1010-raspi_5.13.0-1010.11_armhf.deb  2021-10-29 10:29    21M
[ ] linux-raspi-headers-5.4.0-1008_5.4.0-1008.8_arm64.deb   2020-04-10 11:18    11M

Install linux-modules-extra

Then I decided to do download the package and check the contents

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-raspi/linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb
dpkg -c linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb | grep iscsi

found the iscsi_tcp.ko is in the package

Install package via dpkg

dpkg -i linux-modules-extra-5.13.0-1009-raspi_5.13.0-1009.10_arm64.deb


Tested using iscsiadm, shows the problem fixed

iscsiadm --mode discovery --op update --type sendtargets --portal

In fact, this also fixed some other similar issues, such as, zram model missing, etc.

Reconfigure btrfs filesystem

In this system, the btrfs filesystem which was mirrored to iscsi LUN using bcache, the iscsi LUN was removed using following command during troubleshooting

btrfs balance start -f -sconvert=single -mconvert=single -dconvert=single /app
btrfs device remove /dev/bcache0 /app

Now use following command to reconfigure it back.

btrfs device add -f /dev/bcache0 /app
btrfs balance start -dconvert=raid1 -mconvert=raid1 /app

Configure module auto-loading

In order to load modules automatically, add iscsi_tcpline in /etc/modules. This may also avoid apt autoremove command removes linux-modules-extra package.

Possible cause

I think the issue was caused by spliting the kernel module package in to two in 1008, because linux-modules-extra only exists in the version 1008 in package searching list. I think ubnutu forgot this when releasing 1009. At this time, apt hasn't auto updated system to 1010.


Ubuntu Raspberry Package Repo
Ubuntu Package Searching
How to list files of a Debian package without install
How to install specific Ubuntu packages, with exact version?

Preset Tmux Windows

Preset Tmux Windows

To preset Tmux windows, following script can be used. It also issues ssh command to host which has same name as window name.


WINDOWS="window_name1 window_name2 window_name3"

for each in $WINDOWS
  if ! tmux has-session -t 0:$each; then
    tmux new-window -n $each ssh $each

tmux attach


Restore tmux session after reboot
Check If Window With a Specific Name Exists. If It Does Attach to it; Otherwise Create it and Run Command?

Tmux vs Screen

Tmux vs Screen

The unix command screen and tmux have similar goal, but tmux is more complex than screen.

Quick Tips

In order to start using tmux without worrying about forgetting keys, remember follows one after another.

  • Help using key Ctrl+b ?
  • Detach using key Ctrl+b d
  • Resume using command tmux a
  • Create window using command Ctrl+b c
  • Navigate windows using key Ctrl+b w, then arrow keys
  • Rename window using key Ctrl+b ,
  • Status bar has window name and <id>
  • Switch window using key Ctrl+b <id>


screen tmux
Startup screen tmux
Starting Named Session screen -S session_name tmux new -s session_name
List running session screen -ls tmux ls
Reattach screen -r
Reattach by id screen -r <id> tmux attach-session -t <id>


screen tmux
Help Ctrl+a ? Ctrl+b ?
Create a new shell (without window) Ctrl+a c
Switch to next shell Ctrl+a space
Switch to previous shell Ctrl+a backspace
Create a new window (with shell) Ctrl+b c
List all shells Ctrl+a "
Choose window from a list Ctrl+b w
Switch to N'th shell Ctrl+a <n>
Switch to N'th window Ctrl+b <n>
Rename the current window Ctrl+a A Ctrl+b ,
Split current region horizontally into two regions Ctrl+a S Ctrl+b %
Split current region vertically into two regions Ctrl+a | Ctrl+b "
Switch the input focus to the next region Ctrl+a tab Ctrl+b o
Toggle between the current and previous windows Ctrl+a Ctrl+a Ctrl+b ;
Close all regions but the current one Ctrl+a Q
Close the current region Ctrl+a X Ctrl+b x
Detach from session Ctrl+a d Ctrl+b d
Sent Ctrl+a to region Ctrl+a a



Customization can be done in /etc/screenrc and ~/.screenrc. Sample of contents is shown below.

# Turn off the welcome message
startup_message off

# Disable visual bell
vbell off

# Set scrollback buffer to 10000
defscrollback 10000

# Customize the status line
hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %m-%d %{W}%c %{g}]'


Customization can be done in ~/.tmux.conf. Sample of contents is shown below.

# Improve colors
set -g default-terminal 'screen-256color'

# Set scrollback buffer to 10000
set -g history-limit 10000

# Customize the status line
set -g status-fg  green
set -g status-bg  black


How To Use Linux Screen
Getting started with Tmux
How to split the terminal into more than one "view"?

SSH Proxy

SSH Proxy

When access remote servers, proxy server/jump host could be a good choice.

Two servers definition


This is the proxy server, normally can be in the form of user@proxy_host.


This is the target server, has the form of target_user@target_host.

Jump host

This is the most simplest method which uses remote ssh command to login to target server. If just wants to use one SSH server to access another SSH server, run following command

ssh -t <jump_host> ssh <target>

Here -t is to create a pseudo-terminal, otherwise, the remote server could not display message because no tty can be used.

Proxy Jump

This is using SSH building proxy function.

ssh -o "ProxyJump <jump_host>" <target>

Proxy Command

This is to run an external command to build communication. For example, use nc command, and pass two parameters, %h is the target host, %p is the port.

ssh -o "ProxyCommand ssh <jump_host> nc %h %p" <target>

Use ssh client configuration


Instead of using command, ProxyJump and ProxyCommand can be set in ssh client configuration file.

Host <connection_name>
  HostName <target_host>
  User <target_user>
  ProxyCommand ssh <jump_host> nc %h %p

For one server

Host target
  User user1
  ProxyCommand ssh user2@proxy nc %h %p

For all servers

Host * !proxy
  HostName %h
  User target_user
  # ProxyCommand ssh proxy_user@proxy nc %h %p


This is the same as previous example, except it is a global setting.


4 ways to SSH & SCP via proxy (jump) server in Linux