performance issue on Intel(R) Xeon(R) Gold 5320 with ubuntu 20.04

[charleschetty]

I’d nearly forgotten about this project, but recently some friends took an interest in the tool. So I ran a new round of benchmarks and discovered a significant performance degradation on an Intel® Xeon® Gold 5320 under Ubuntu 20.04.

CrabFetch exhibits the same issue.

hyperfine --runs 100 -N --warmup=20 "./fastfetch -c /home/zjt/ccfetch/benchmark/2.jsonc" "ccfetch" "/home/zjt/CrabFetch/target/release/crab-fetch -c /home/zjt/CrabFetch/target/release/1.toml"
Benchmark 1: ./fastfetch -c /home/zjt/ccfetch/benchmark/2.jsonc
  Time (mean ± σ):       3.8 ms ±   0.2 ms    [User: 1.0 ms, System: 2.8 ms]
  Range (min … max):     3.7 ms …   4.7 ms    100 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ccfetch
  Time (mean ± σ):      11.4 ms ±   0.9 ms    [User: 6.4 ms, System: 4.9 ms]
  Range (min … max):    11.1 ms …  18.4 ms    100 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 3: /home/zjt/CrabFetch/target/release/crab-fetch -c /home/zjt/CrabFetch/target/release/1.toml
  Time (mean ± σ):      12.7 ms ±   0.5 ms    [User: 8.1 ms, System: 4.5 ms]
  Range (min … max):    12.5 ms …  15.9 ms    100 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  ./fastfetch -c /home/zjt/ccfetch/benchmark/2.jsonc ran
    2.97 ± 0.26 times faster than ccfetch
    3.31 ± 0.20 times faster than /home/zjt/CrabFetch/target/release/crab-fetch -c /home/zjt/CrabFetch/target/release/1.toml

However, on Intel(R) Core(TM) i7-6700 CPU with ArchLinux ,

hyperfine --runs 100 -N --warmup=20 "./fastfetch -c /home/charles/Documents/github/ccfetch/benchmark/2.jsonc" "/home/charles/Documents/github/ccfetch/target/release/ccfetch" /"home/charles/Sourcefile/CrabFetch/target/release/crab-fetch -c /home/charles/Sourcefile/CrabFetch/target/release/1.toml"
Benchmark 1: ./fastfetch -c /home/charles/Documents/github/ccfetch/benchmark/2.jsonc
  Time (mean ± σ):       6.8 ms ±   0.3 ms    [User: 2.2 ms, System: 4.5 ms]
  Range (min … max):     6.4 ms …   7.5 ms    100 runs

Benchmark 2: /home/charles/Documents/github/ccfetch/target/release/ccfetch
  Time (mean ± σ):       5.0 ms ±   0.2 ms    [User: 3.3 ms, System: 1.5 ms]
  Range (min … max):     4.7 ms …   5.6 ms    100 runs

Benchmark 3: /home/charles/Sourcefile/CrabFetch/target/release/crab-fetch -c /home/charles/Sourcefile/CrabFetch/target/release/1.toml
  Time (mean ± σ):       7.5 ms ±   0.4 ms    [User: 5.1 ms, System: 2.3 ms]
  Range (min … max):     6.9 ms …   9.3 ms    100 runs

Summary
  /home/charles/Documents/github/ccfetch/target/release/ccfetch ran
    1.37 ± 0.07 times faster than ./fastfetch -c /home/charles/Documents/github/ccfetch/benchmark/2.jsonc
    1.51 ± 0.10 times faster than /home/charles/Sourcefile/CrabFetch/target/release/crab-fetch -c /home/charles/Sourcefile/CrabFetch/target/release/1.toml

I’ve been tied up with some things lately, but I may come back later to look into the issue。

[Sisyphus1813]

I'm not gonna pretend I know the answer to this question but this might be related:

https://stackoverflow.com/questions/74949701/rust-poor-runtime-performance-on-ubuntu-with-core-i9-compared-to-macos-with-cor

My best guess is this is compiler optimizations related. Let me know if you come up with a definitive answer!

[charleschetty]

I took a look at it roughly, and I think there are three aspects which lead to the issue.

DRM VS PCI

In

ccfetch/src/tools/pci.rs

Line 51 in 32caba2

pub fn read_pci_devices_and_find_gpu() -> io::Result<Vec<(String, String)>> {

I use /sys/bus/pci/devices to get the vender id and device id. However, the fastfetch uses /sys/class/drm.

const char* ffDetectGPUImpl(const FFGPUOptions* options, FFlist* gpus)
{
    #ifdef FF_HAVE_DIRECTX_HEADERS
        const char* ffGPUDetectByDirectX(const FFGPUOptions* options, FFlist* gpus);
        if (ffGPUDetectByDirectX(options, gpus) == NULL)
            return NULL;
    #endif

    if (options->detectionMethod == FF_GPU_DETECTION_METHOD_AUTO)
    {
        if (drmDetectGPUs(options, gpus) == NULL && gpus->length > 0)
            return NULL;
    }
    return pciDetectGPUs(options, gpus);
}

My personal PC has only 16 subdirectories in /sys/bus/pci/devices, while the cloud server contains 286 subdirectories.
In contrast, my PC has 8 subdirectories in /sys/class/drm, whereas the server has 16 subdirectories.

I rewrote it using the DRM method and conducted a test: the DRM method took 77.438 µs, while the PCI method took 2.971324 ms.

Specific methods to obtain GPU information.

I simply use pci.ids to get those information, but for the fastfetch:

  if (gpu->vendor.chars == FF_GPU_VENDOR_NAME_AMD) {
 //....
    if (drmKey && options->driverSpecific)
      ok = drmDetectAmdSpecific(options, gpu, drmKey, buffer) == NULL;
// ... 
    }
  } else if (gpu->vendor.chars == FF_GPU_VENDOR_NAME_INTEL) {
    pciDetectIntelSpecific(options, gpu, deviceDir, buffer, drmKey);
// ...
  } else {
    ffGPUDetectDriverSpecific(options, gpu,
                              (FFGpuDriverPciBusId){
                                  .domain = pciDomain,
                                  .bus = pciBus,
                                  .device = pciDevice,
                                  .func = pciFunc,
                              });
  }

For the NVIDIA devices, they use libnvml to directly get the result, but I obtain information by traversing the entire PCI ID file, which has 40000+ lines.

For the cloud server , there are three gpu devices,

󰾲  Matrox Electronics Systems Ltd. Integrater
󰾲  NVIDIA GeForce RTX 3090 [Discrete]
󰾲  NVIDIA GeForce RTX 3090 [Discrete]

The detection time for the Matrox Electronics Systems Ltd. Integrater is 0.826 ms. Since there are no specific methods in fastfetch, it also uses pci.ids to generate the information.
In contrast, the detection time for the NVIDIA RTX 3090 is 0.028 ms.

For ccfetch, 610.949µs for Matrox Electronics Systems and 1.386698ms for RTX 3039

My implementation for obtaining CPU information is lacking

My implementation took 1.71644 ms, while Fastfetch's took only 0.085 ms.

Similarly, I traverse the entire /proc/cpuinfo to retrieve the logical cores, whereas Fastfetch uses the get_nprocs_conf function from libc.

The cpuinfo file contains 2,808 lines on the Ubunutu server."

What's next?

It seems that implementing different methods to accommodate various GPUs requires a significant amount of time. And it will obviously require other external libraries.

So, maybe I can just start with NVIDIA devices, as for other devices, maybe I'll address them when I encounter them.

[Sisyphus1813]

This could be worth looking into.

https://github.com/gfx-rs/gfx/

[charleschetty]

My implementation took 1.71644 ms, while Fastfetch's took only 0.085 ms.

Well ,

   let cpuinfo = fs::read_to_string("/proc/cpuinfo").map_err(|e| e.to_string())?;

takes 1.590776ms , but if I copy cpuinfo to my local dir,

   let cpuinfo = fs::read_to_string("/home/zjt/cpuinfo").map_err(|e| e.to_string())?;

takes only 84.667µs ....

[charleschetty]

In fastfetch,

    FF_STRBUF_AUTO_DESTROY cpuinfo = ffStrbufCreateA(PROC_FILE_BUFFSIZ);
    if (!ffReadFileBuffer(FF_CPUINFO_PATH, &cpuinfo) || cpuinfo.length == 0)
        return "ffReadFileBuffer(\"" FF_CPUINFO_PATH "\") failed";

takes 1.85ms,

https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html

/proc is very special in that it is also a virtual filesystem. It's sometimes referred to as a process information pseudo-file system. It doesn't contain 'real' files but runtime system information

It appears that each time a user reads /proc/cpuinfo, the kernel generates a fresh, updated version.

I think I can implement a cache anyway, since the CPU information doesn’t change very often.

[charleschetty]

Specific methods to obtain GPU information.

I was mistaken. In fact, the options->driverSpecific flag defaults to false, and calling into libnvml takes roughly 30 ms.

#include "nvml.h"
int main(){
    nvmlReturn_t result;
    unsigned int deviceCount, i;

    nvmlInit_v2();
    nvmlDeviceGetCount_v2(&deviceCount);
    for (i = 0; i < deviceCount; i++) {
        nvmlDevice_t device;
        char name[NVML_DEVICE_NAME_BUFFER_SIZE];
        unsigned int temp;
        nvmlDeviceGetHandleByIndex_v2(i, &device);
        nvmlDeviceGetName(device, name, sizeof(name));
    }
    nvmlShutdown();
}

fastfetch uses a pre-generated pciids file to accelerate PCI device lookups.

gen-pciids.py

void ffGPUFillVendorAndName(uint8_t subclass, uint16_t vendor, uint16_t device, FFGPUResult* gpu)
{
    #if FF_HAVE_EMBEDDED_PCIIDS
    bool ok = loadPciidsInc(subclass, vendor, device, gpu);
    if (ok) return;
    #endif
    return parsePciIdsFile(loadPciIds(), subclass, vendor, device, gpu);
}

The loadPciidsInc function is much faster the parsePciIdsFile routine I’m using.
Perhaps I can learn from their approach.

[charleschetty]

I think I can implement a cache anyway, since the CPU information doesn’t change very often.

Well, fastfetch doesn't cache the cpuinfo, instead they cache the packages count result...

    FF_STRBUF_AUTO_DESTROY cacheDir = ffStrbufCreate();
    FF_STRBUF_AUTO_DESTROY cacheContent = ffStrbufCreate();

    uint32_t num_elements;
    if (ffPackagesReadCache(&cacheDir, &cacheContent, baseDir->chars, packageId, &num_elements))
    {
        ffStrbufSubstrBefore(baseDir, baseDirLength);
        return num_elements;
    }

    num_elements = getNumStringsImpl(baseDir->chars, needle);
    ffStrbufSubstrBefore(baseDir, baseDirLength);

    ffPackagesWriteCache(&cacheDir, &cacheContent, num_elements);

    return num_elements;

[Sisyphus1813]

This seems counter intuitive as package count is rather dynamic. Alternatively, we could cache CPU & GPU info to offload the performance cost of those systems calls. I'm unsure how much of a difference it's going to make though assuming we would still have to retrieve the info from memory.

[charleschetty]

results now

With both the /proc/cpuinfo cache and the dpkg cache in place, here are the results on a server running Ubuntu 20.04 with an Intel Xeon Gold 5320:

hyperfine --runs=1000 --shell=none --warmup=20 "./fastfetch -c 2.jsonc" "/home/zjt/CrabFetch/target/release/crab-fetch -c /home/zjt/CrabFetch/target/release/1.toml" "ccfetch"
Benchmark 1: ./fastfetch -c 2.jsonc
  Time (mean ± σ):       3.5 ms ±   0.2 ms    [User: 0.9 ms, System: 2.6 ms]
  Range (min … max):     3.3 ms …   6.4 ms    1000 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: /home/zjt/CrabFetch/target/release/crab-fetch -c /home/zjt/CrabFetch/target/release/1.toml
  Time (mean ± σ):      12.5 ms ±   0.6 ms    [User: 8.1 ms, System: 4.3 ms]
  Range (min … max):    12.3 ms …  23.3 ms    1000 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 3: ccfetch
  Time (mean ± σ):       2.9 ms ±   0.1 ms    [User: 2.2 ms, System: 0.6 ms]
  Range (min … max):     2.8 ms …   3.7 ms    1000 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  ccfetch ran
    1.22 ± 0.07 times faster than ./fastfetch -c 2.jsonc
    4.37 ± 0.23 times faster than /home/zjt/CrabFetch/target/release/crab-fetch -c /home/zjt/CrabFetch/target/release/1.toml

I haven’t tackled the gpu methods; I’ll find time to address it later.

fastfetch uses a pre-generated pciids file to accelerate PCI device lookups.

package cache

This seems counter intuitive as package count is rather dynamic

Fastfetch stores the last-modified timestamp of /var/lib/dpkg/status in its cache and compares it to the file’s current timestamp on each run.

I'm unsure how much of a difference it's going to make though assuming we would still have to retrieve the info from memory.

In my tests, traversing /var/lib/dpkg/status to count packages takes about 2.1 ms, whereas the cache take 12.681µs.
The /var/lib/dpkg/status file contains a total of 35,991 lines.

[Sisyphus1813]

In case you're curious, I've tested these recent upgrades on my system.
Before:

  Time (mean ± σ):      11.1 ms ±   0.9 ms    [User: 8.3 ms, System: 2.7 ms]
  Range (min … max):     9.4 ms …  14.0 ms    203 runs

After:

  Time (mean ± σ):      10.5 ms ±   1.0 ms    [User: 8.3 ms, System: 2.1 ms]
  Range (min … max):     8.9 ms …  14.3 ms    233 runs

[charleschetty]

My results are quite similar

Benchmark 1: /home/charles/Documents/github/ccfetch/target/release/ccfetch
  Time (mean ± σ):       4.6 ms ±   0.1 ms    [User: 3.4 ms, System: 1.0 ms]
  Range (min … max):     4.5 ms …   6.1 ms    1000 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: /home/charles/Documents/github/ccfetch-old/target/release/ccfetch
  Time (mean ± σ):       5.2 ms ±   0.1 ms    [User: 3.7 ms, System: 1.3 ms]
  Range (min … max):     5.0 ms …   6.6 ms    1000 runs

ccfetch-old uses 32caba2

[charleschetty]

Alternatively, we could cache CPU & GPU info to offload the performance cost of those systems calls.

I use the system’s boot time as a cache invalidation key. Because replacing the CPU or GPU always requires a reboot, the boot time reliably indicates when the hardware may have changed.

The runtime on the server has now improved from 2.9 ms down to 849.3 µs.

[Sisyphus1813]

Similar performance improvements on my end.
Before:

  Time (mean ± σ):      10.6 ms ±   1.2 ms    [User: 8.2 ms, System: 2.3 ms]
  Range (min … max):     8.9 ms …  15.0 ms    276 runs

After:

  Time (mean ± σ):       7.6 ms ±   1.0 ms    [User: 5.5 ms, System: 2.0 ms]
  Range (min … max):     6.2 ms …  11.9 ms    237 runs