Day: January 5, 2022

TrueNAS k3s-server uses more than 10% CPU

TrueNAS k3s-server uses more than 10% CPU

The k3s supposed to be the lightweight k8s, but it used more than 10% when no container running. The problem was reported when TrueNAS running as VM.

This issue also caused high disk utilization, which caused whole system slow, huge IO wait.

Unset Pool

There is a zpool called ix-applications created for k3s. To stop k3s process, needs to unset pool in Apps => Settings => Unset Pool.

Result

After Unset Pool, the CPU utilization dropped from 70% to 5%.

References

k3s-server uses 10% CPU for no reason

Memory test hung after upgrade to 64GB RAM (Follow-up)

Memory test hung after upgrade to 64GB RAM (Follow-up)

Note: This is follow up action of Memory test hung after upgrade to 64GB RAM

This machine was installed TrueNAS, but reboot quite often, about few times a week. After that, I changed it to Proxmox, then one of the active VM kept hanging.

I decided to re-look into the memory test issue.

32GB testing

If only install 32GB memory, the test was OK at beginning. But after full day of testing, start to have 1 failure in one pass, not many but not acceptable.

64GB testing

If install 48GB or 64GB memory, the test caused system hanging in black screen.

Reduce memory speed

After reduce the memory speed to 1066 for 64GB memory, looks OK in Pass 1, but Pass 2 got issue again.

Getting worse

After full day of testing, the test result was getting worse. Previously hung at 80%+, then hung at 40%+. Then looks like problem caused by other factors, such as temperature, etc.

Then I noticed that the power supply is very hot and no noise. Then I think the issue could be the system was not getting enough power.

Change power supply

After take out the power supply, found that power supply fan was not turning. Normally, this issue can not be detected, because the fan towards the bottom of casing.

I got one new power supply, after installed new power supply, the system become colder.

Memory test

After changed the power supply, the memory passed when setting at speed of 1333, which is also the speed detected by motherboard. Even the motherboard auto overclocked CPU speed to 4000+ (CPU speed should be 3600), no error too.

Speed of the old set of RAMs

The new set of RAMs can pass testing at 1600 speed, but the old set got error less than 1 minute.

Although the spec of the old and new RAMs are all the same, brand, speed, etc, but the system detected the new RAMs are 1600, but old set are 1333, no matter in which memory slot. I'm not sure whether the lower speed is caused by aging or because cheated by seller.

Speed of testing among 1600, 1333, and 1066

During the testing, I noticed that the testing speeds are quite different among 1600, 1333 and 1066 if they are under same CPU speed.

References

MemTest86