Author: Bian Xi

ZFS cache and log

Posted on by Bian Xi Leave a Comment on ZFS cache and log

ZFS cache and log

There are two kinds of cache, read cache and write cache.

Read cache

Called ARC and L2ARC.

ARC (Adaptive Replacement Cache)

In memory, caching the information that would require in the near future, while discarding the ones that will be needed furthest ahead in time.

This can be set using kernel/module parameter, such as zfs_arc_max.

L2ARC (Level 2 ARC)

In cache device, extension of ARC. Can be created using following command

zpool add tank cache ada3

Note: tank is the pool name, ada3 is the block device used for caching

Write cache

Called ZIL (ZFS Intent Log).

Asynchronous

By default, ZFS will cache write data in memory before write to disk, this is called asynchronous mode.

Synchronous

Synchronous will make sure data written to disk before continue, this can be set using following command

zfs set sync=always mypool/dataset1

ZFS Intent Log (ZIL)

This is the temporary space to store data before write into main disks, this can be used to speed up write operation. The write operation is considered as completed once data written into ZIL device, which is called SLOG (Separate Intent Log) devices, can be defined as follow

zpool add tank log ada3

Note: tank is the pool name, ada3 is the block device used for slog

If worrying SLOG device faulty, it can be mirrored too.

zpool add tank log mirror ada3 ada4

References

Configuring ZFS Cache for High Speed IO
ZFS Performance with Databases (Cached)

Snapshot and Copy on Write

Posted on by Bian Xi Leave a Comment on Snapshot and Copy on Write

Snapshot and Copy on Write

Two type of snapshots, they are using different ways.

Keep original data

No write to original data, all new data will be in delta file.

VMware

All new data will be in delta file, the original disk file will not be changed. This is very usefull especially in VDI environment, all VDI servers will base on same images and no impact to original disk file.

When deleting a snapshot, the snapshot files are consolidated and written to the parent snapshot disk. If parent is base disk, and all the data from the delta disk will be merged with the virtual machine base disk.

QEMU / KVM: COW mode

COW mode is available on some formats of virtual machine disk as QCOW2. When using the COW mode, no changes are applied to the disk image. All changes are recorded in a separate file preserving the original image. Several COW files can point to the same image to test several configurations simultaneously without jeopardizing the basic system.

QEMU / KVM allows to incorporate changes from a COW file to the original image

Overwrite original disk

Write latest data into original disk, and the original data move to delta disk.

RedHat LVM snapshot

The data in snapshot volume is original data.

ZFS and btrfs

Due to native copy-on-write feature, file usage reference structure always points to new data, and the old data is saved in old reference structure.

Compare

Original Disk Pros Cons
Keep * Support multiple childs without too much performance overhead Deleting snapshot takes time
 When disk full, no more write can be done
Overwrite Less overhead - only when first time writing data on new location
 Reverting snapshot takes time		 * Deleting snapshot fast
Native COW No overhead
 Fast dropping snapshot
Fast reverting snapshot
 No impact to service when snapshot full, but snapshot corrupt		 Unable to delete file when disk full
 Disk fragmented easily
* Unable to cache disk write operation

References

Deleting Snapshots
QEMU / KVM: Using the Copy-On-Write mode
Why would I want to disable Copy-On-Write while creating QEMU Images?

Error of txg_sync blocked for more than 120 seconds

Posted on by Bian Xi Leave a Comment on Error of txg_sync blocked for more than 120 seconds

Error of txg_sync blocked for more than 120 seconds

Following error was appearing in my dmesg monitoring screen.

txg_sync blocked for more than 120 seconds --> excessive load

If I'm not wrong, it could be caused by slow harddisk speed, because the TrueNAS zfs cache is about 61GB, can take longer time to flush back to hard disk.

Same as other filesystem, zfs has writeback caching (aka write-behind caching), which will flush data back to hard disk in specific interval. zfs has synchronous and asynchronous mode, they are a bit different that readonly, writethrough and writeback mode.

Except above, zfs has different behaviors on copy on write (COW) as below.

  • Always write to new block due to copy on write
  • Big file for random writing, such as VM disk file, can be fragmented
  • Can not reduce the write operation even if keep writing same block

Therefore, copy on write should be disabled for VM images. But if so, snapshot function could be lost.

Reference

Read-Through, Write-Through, Write-Behind Caching and Refresh-Ahead

Change max arc size on TrueNAS SCALE

Posted on by Bian Xi Leave a Comment on Change max arc size on TrueNAS SCALE

Change max arc size on TrueNAS SCALE

After upgrade memory to 64GB, the memory usage is less than 32GB even run two VMs together. To utilize all memory, increase zfs cache size is one of the solution can be done.

c_max

The max arc size is defined as a module parameter, which can be viewed by following command

truenas# grep c_max /proc/spl/kstat/zfs/arcstats
c_max                           4    62277025792
truenas# cat /sys/module/zfs/parameters/zfs_arc_max
62277025792
truenas#

To justify this value, following command can be used, but it is not a persistent way.

echo 60129542144 > /sys/module/zfs/parameters/zfs_arc_max

Suggestion from others

Many suggestions can be found, some of them maybe workable, for example

Create module option file

echo "options zfs zfs_arc_max=34359738368" > /etc/modprobe.d/zfs.conf

But they may not suitable for a NAS OS which can not be backed up using configuration backup provided by NAS OS.

  • The upgrade of OS can simply overwrite or delete the file
  • The file can be lost during OS rebuilting process.

Update sysctl (not workable)

Suggestion is update vfs.zfs.arc_max using sysctl, along with disable autotune, but it is only workable for kernel parameters, but no zfs parameters could be found, the zfs is loaded as module.

Implemenation

The parameter needs to be modified using TrueNAS web interface, to ensure that it will be saved during configuration export via System Settings => General => Manage Configuration => Download File.

So, following command is added into System Settings => Advanced => Init/Shutdown Scripts with When set to Post Init

echo 60129542144 > /sys/module/zfs/parameters/zfs_arc_max

Verification

Verify the setting as below.

arc_summary | grep size

Note: The number is in bytes

Reduce the number

In order to reduce the number without reboot, following command needs to be executed to reduce the cache immediately

echo 3 > /proc/sys/vm/drop_caches

References

Why I cannot modify "vfs.zfs.arc_max" in WebUI?
QEMU / KVM: Using the Copy-On-Write mode

Ubuntu grub-efi-amd64-signed error after do-release-upgrade

Posted on by Bian Xi Leave a Comment on Ubuntu grub-efi-amd64-signed error after do-release-upgrade

Ubuntu grub-efi-amd64-signed error after do-release-upgrade

Following error occurred whenever run apt upgrade after perform do-release-upgrade

dpkg: error processing package grub-efi-amd64-signed (–configure):
installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32

Solution

Reinstall all grub group packages using following commands

sudo apt-get purge grub\*
sudo apt-get install grub-efi
sudo apt-get autoremove
sudo update-grub

Options restrict in one filesystem

Posted on by Bian Xi Leave a Comment on Options restrict in one filesystem

Options restrict in one filesystem

There are quite number of tasks may want to be executed in one filesystem, this is important during troubleshooting, especially for root directory (/).

find

Restrict find command only looking entries within one filesystem, use option -xdev

find /usr -xdev ...

du

Restrict du command only calculate for one filesystem, use option -x

du -cshx /

tar

Restrict tar command only archive files in one filesystem, use option --one-file-system

tar --one-file-system -czvf /tmp/root.tgz /

Memory test hung after upgrade to 64GB RAM

Posted on by Bian Xi Leave a Comment on Memory test hung after upgrade to 64GB RAM

Memory test hung after upgrade to 64GB RAM

Update: The problem was fixed as stated in Memory test hung after upgrade to 64GB RAM (Follow-up)

Took many hours to troubleshoot RAM test hanging issue.

Testing software

MemTest86

The test was hung at 86% during pass 1.

Memtest86+

The bootable USB created was not bootable.

Ubuntu Live CD

The testing was hung just started.

Posible issue

Hardware issue

Should not be RAM issue, tested them by separating them into two sets with 32GB each, using MemTest86, both sets can pass.

Maybe slot issue

The RAM must be in slots in correct order, otherwise, the RAM can not be detected. In fact, this mother board is quite sensitive to the RAM position.

E8036_P9X79_DELUXE

Maybe RAM hot

The RAMs could be too hot, the newly added RAMs make the gabs between RAMs are too small, the fans are also not strong enough, this could cause high temperature. I tried to adjust the fan speed by using the controls on casing, but not effective.

But RAM should be able to take high temperature, and the type of RAM I bought has cooling case.

Software issue

Maybe conflict with VGA

The max RAM size that MemTest86 can support, is 64GB, which means it could have bug too. On the other hand, the hunging shows nothing on the screen, black screen, but keyboard light was still responsive, and the VGA light on mother board was turned on. Maybe the software wrote the area that video card used caused such issue.

Conclusion

Although the testing was hung, I decided still use them.

In order to use them fully in TrueNAS, I adjusted the zfs_arc_max to 60GB, and run two VMs, one Windows at 4GB, one ubuntu at 6GB, till the memory free less than 4GB.

Result

The TrueNAS looks working fine, although restarted once, log didn't show the cause related to memory, and there was no memory issue in dmesg monitoring screen.

Increase upload file size limit for WordPress and NGNIX

Posted on by Bian Xi Leave a Comment on Increase upload file size limit for WordPress and NGNIX

Increase upload file size limit for WordPress and NGNIX

There are various ways to do, but the workable way is, updating .htaccess in WordPress and NGNIX configuration file.

Issue

First, tried the way by changing function.php in theme, but no luck. Then updated .htaccess file, it worked.

Then the client gets the error “Request Entity Too Large” (413). This error reported by NGINX.

WordPress

Add following lines in .htaccess file in html directory

php_value upload_max_filesize 64M
php_value post_max_size 64M
php_value max_execution_time 300
php_value max_input_time 300

Then the upload page in WordPress should be shown as below

Maximum upload file size: 64 MB.

Alternative

These options are PHP options, which can be applied to php.ini as well as below

upload_max_filesize = 64M
post_max_size = 64M
max_execution_time = 300

NGINX

Add the following line to http, server or location context in nginx.conf or conf.d/default.conf

client_max_body_size 64M;

Then reload NGINX configure.

# /usr/local/nginx/sbin/nginx -s reload

This will fix the client error “Request Entity Too Large” (413).

Remove ubuntu zfs snapshots

Posted on by Bian Xi Leave a Comment on Remove ubuntu zfs snapshots

Remove ubuntu zfs snapshots

There are so many snapshots when using zfs in ubuntu.

Issue

When tried to do release update, got following error

# do-release-update
...
...
Not enough free disk space 

The upgrade has aborted. The upgrade needs a total of 256 M free 
space on disk '/boot'. Please free at least an additional 91.4 M of 
disk space on '/boot'. You can remove old kernels using 'sudo apt 
autoremove' and you could also set COMPRESS=xz in 
/etc/initramfs-tools/initramfs.conf to reduce the size of your 
initramfs. 
...

This error messsage was occurred many times before, but those systems had very small /boot partition or many old kernels kept. If it is the first case, total repartitioning and moving root filesystem are required.

Space on /boot

Examing disk space for bpool, found that zfs reported 675MB used in bpool, but actual usage is only 242MB.

root@ubuntu:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   960M   675M   285M        -         -    30%    70%  1.00x    ONLINE  -
rpool  17.5G  7.99G  9.51G        -         -    21%    45%  1.00x    ONLINE  -
root@ubuntu:~# zfs list bpool
NAME    USED  AVAIL     REFER  MOUNTPOINT
bpool   675M   157M       96K  /boot
root@ubuntu:~# du -cshx /boot
242M    /boot
242M    total
root@ubuntu:~#

Then found many snapshots both in bpool and data pool

root@ubuntu:~# zfs list -t snapshot | head
NAME                                                               USED  AVAIL     REFER  MOUNTPOINT
bpool/BOOT/ubuntu_e8m8h0@autozsys_ywm1ok                             0B      -      238M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_ms74md                             0B      -      238M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_ugu9z7                            80K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_r3xqau                            72K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_nkagbh                             0B      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_xdbwsy                             0B      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_zrt7vi                            72K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_jbmnwk                            72K      -      242M  -
bpool/BOOT/ubuntu_e8m8h0@autozsys_0e5p2e                            64K      -      242M  -
root@ubuntu:~# 
root@ubuntu:~# zfs list -t snapshot | wc
    301    1505   27701

Too many! Not sure how many snapshots ubuntu likes to create

Removing snapshots

List all snapshots for /boot

root@ubuntu:~# df /boot
Filesystem               1K-blocks   Used Available Use% Mounted on
bpool/BOOT/ubuntu_e8m8h0    408192 247808    160384  61% /boot
root@ubuntu:~# zfs list -H -o name -t snapshot bpool/BOOT/ubuntu_e8m8h0
bpool/BOOT/ubuntu_e8m8h0@autozsys_ywm1ok
bpool/BOOT/ubuntu_e8m8h0@autozsys_ms74md
bpool/BOOT/ubuntu_e8m8h0@autozsys_ugu9z7
bpool/BOOT/ubuntu_e8m8h0@autozsys_r3xqau
bpool/BOOT/ubuntu_e8m8h0@autozsys_nkagbh
bpool/BOOT/ubuntu_e8m8h0@autozsys_xdbwsy
bpool/BOOT/ubuntu_e8m8h0@autozsys_zrt7vi
bpool/BOOT/ubuntu_e8m8h0@autozsys_jbmnwk
bpool/BOOT/ubuntu_e8m8h0@autozsys_0e5p2e
bpool/BOOT/ubuntu_e8m8h0@autozsys_b17dwn
bpool/BOOT/ubuntu_e8m8h0@autozsys_uad1rb
bpool/BOOT/ubuntu_e8m8h0@autozsys_mxhvc9
bpool/BOOT/ubuntu_e8m8h0@autozsys_9athz8
bpool/BOOT/ubuntu_e8m8h0@autozsys_61umv1
bpool/BOOT/ubuntu_e8m8h0@autozsys_1q65cz
root@ubuntu:~#

Then remove them

zfs list -H -o name -t snapshot bpool/BOOT/ubuntu_e8m8h0 | xargs -n 1 zfs destroy

Now, it is ok to upgrade

root@ubuntu:~# zfs list -o space bpool
NAME   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
bpool   589M   243M        0B     96K             0B       243M
root@ubuntu:~#