i586: crash> ps *** stack smashing detected ***: terminated

[vt-alt]

While testing build we got crash to crash on i586.

crash 9.0.0-alt1
Copyright (C) 2002-2025  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

crash: read error: kernel virtual address: c1c2df28  type: "kernel_config_data"
WARNING: cannot determine hardirq_ctx addresses
WARNING: cannot determine softirq_ctx addresses
      KERNEL: /usr/lib/debug/lib/modules/6.12.28-6.12-alt0.c10f.2/vmlinux
    DUMPFILE: /proc/kcore
        CPUS: 8
        DATE: Mon Jun  9 17:15:58 UTC 2025
      UPTIME: 00:00:08
LOAD AVERAGE: 0.08, 0.02, 0.01
       TASKS: 140
    NODENAME: (none)
     RELEASE: 6.12.28-6.12-alt0.c10f.2
     VERSION: #1 SMP PREEMPT_DYNAMIC Tue May 13 15:33:47 UTC 2025
     MACHINE: i686  (2450 Mhz)
      MEMORY: 2 GB
         PID: 168
     COMMAND: "crash"
        TASK: c3cc8000  [THREAD_INFO: c3cc8000]
         CPU: 2
       STATE: TASK_RUNNING (ACTIVE)

crash> ps
*** stack smashing detected ***: terminated

Compiled with GCC 10.3.1. I am not saying the difference is related to GCC version, but on other box with GCC 14.2.1 (and Linux 6.14.10) there is no failure.

[vt-alt]

After further investigations it seems related to compilation with LTO. If compiled with LTO (-flto=auto) there is no failure, if compiled without LTO (which also happens in GCC 10 case) it fails (even for GCC 14).

This maybe not related but there is additional compilation warning when compiled without LTO:

 In file included from /usr/include/string.h:548,
                  from defs.h:32,
                  from symbols.c:18:
 In function 'strcpy',
     inlined from 'parse_for_member_extended' at symbols.c:9335:4:
 /usr/include/bits/string_fortified.h:81:10: warning: '__builtin___strcpy_chk' offset 0 is out of the bounds [0, 0] [-Warray-bounds=]
    81 |   return __builtin___strcpy_chk (__dest, __src, __glibc_objsize (__dest));
       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[vt-alt]

Under gdb:

crash> ps
[Detaching after fork from child process 203]
*** stack smashing detected ***: terminated

Thread 1 "crash" received signal SIGABRT, Aborted.
0xb782e415 in __pthread_kill_implementation (threadid=threadid@entry=3078151552, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:43
43            int ret = INTERNAL_SYSCALL_CALL (tgkill, __getpid (), tid, signo);
(gdb) bt
#0  0xb782e415 in __pthread_kill_implementation (threadid=threadid@entry=3078151552, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:43
#1  0xb782e48b in __pthread_kill_internal (threadid=3078151552, signo=6) at pthread_kill.c:78
#2  0xb77d7181 in __GI_raise (sig=6) at ../sysdeps/posix/raise.c:26
#3  0xb77be252 in __GI_abort () at abort.c:79
#4  0xb77bf29d in __libc_message_impl (fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:134
#5  0xb78bea7f in __GI___fortify_fail (msg=0xb7939830 "stack smashing detected") at fortify_fail.c:24
#6  0xb78bf76b in __stack_chk_fail () at stack_chk_fail.c:24
#7  0x00d66404 in __stack_chk_fail_local () at stack_chk_fail_local.c:45
#8  0x00ac4a5b in percpu_counter_sum_positive (fbc=3288384692) at tools.c:6970
#9  0x00ad9d4e in get_task_mem_usage (task=3271065600, tm=0xbfffd8d8) at memory.c:4916
#10 get_task_mem_usage (task=<optimized out>, tm=<optimized out>) at memory.c:4840
#11 0x00b1f946 in show_ps_data (flag=flag@entry=8, tc=<optimized out>, tc@entry=0x174b6b0, psi=psi@entry=0x1253040 <psinfo>) at task.c:3838
#12 0x00b22173 in show_ps (flag=8, psi=0x1253040 <psinfo>) at task.c:3905
#13 0x00aad8cc in exec_command () at main.c:893
#14 0x00aadb2e in main_loop () at main.c:840
#15 0x008ab477 in captured_main (data=0xbffff2a0) at main.c:1374
#16 gdb_main (args=0xbffff2a0) at main.c:1407
#17 0x008ab4eb in gdb_main_entry (argc=2, argv=0xbffff4b4) at main.c:1434
#18 0x00b5a6da in gdb_main_loop (argc=<optimized out>, argv=<optimized out>) at gdb_interface.c:81
#19 0x005bdcd4 in main (argc=<optimized out>, argv=<optimized out>) at main.c:721
(gdb)
(gdb) f 8
#8  0x00ac4a5b in percpu_counter_sum_positive (fbc=3288384692) at tools.c:6970
6970    }
(gdb) list
6965                            sizeof(int), "percpu_counter.counters count", FAULT_ON_ERROR);
6966                    ret += count;
6967            }
6968
6969            return (ret < 0) ? 0 : ret;
6970    }
6971
6972    ulong
6973    get_subsys_private(char *kset_name, char *target_name)
6974    {
(gdb) p kt
$1 = (struct kernel_table *) 0x1228de0 <kernel_table>
(gdb) p kt->cpus
$2 = 8
(gdb) p ret
$3 = 380
(gdb) p/x addr
$8 = 0xc2977f84

[vt-alt]

I may attach full log if needed, but there is tail of crash -d99 --no_scroll:

> ps
...
>       0       0   7  c343d400  RU   0.0        0        0  [swapper/7]
<readmem: c2f88000, KVADDR, "fill_task_struct", 5376, (ROE|Q), 3ba9060>
<read_proc_kcore: addr: c2f88000 paddr: 2f88000 cnt: 4096>
<read_proc_kcore: addr: c2f89000 paddr: 2f89000 cnt: 1280>
<readmem: c4444300, KVADDR, "fill_mm_struct", 704, (ROE|Q), 394ebc0>
<read_proc_kcore: addr: c4444300 paddr: 4444300 cnt: 704>
<readmem: c44444b8, KVADDR, "percpu_counter.count", 8, (FOE), bf7fe1a8>
<read_proc_kcore: addr: c44444b8 paddr: 44444b8 cnt: 8>
<readmem: c44444c8, KVADDR, "percpu_counter.counters", 4, (FOE), bf7fe1a4>
<read_proc_kcore: addr: c44444c8 paddr: 44444c8 cnt: 4>
<readmem: f5928f8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f5928f8c paddr: 35928f8c cnt: 4>
<readmem: f594bf8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f594bf8c paddr: 3594bf8c cnt: 4>
<readmem: f596ef8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f596ef8c paddr: 3596ef8c cnt: 4>
<readmem: f5991f8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f5991f8c paddr: 35991f8c cnt: 4>
<readmem: f59b4f8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f59b4f8c paddr: 359b4f8c cnt: 4>
<readmem: f59d7f8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f59d7f8c paddr: 359d7f8c cnt: 4>
<readmem: f59faf8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f59faf8c paddr: 359faf8c cnt: 4>
<readmem: f5a1df8c, KVADDR, "percpu_counter.counters count", 4, (FOE), bf7fe1a0>
<read_proc_kcore: addr: f5a1df8c paddr: 35a1df8c cnt: 4>
*** stack smashing detected ***: terminated
Aborted

So this seems related to this code:

        long ret;
...
        readmem(fbc + OFFSET(percpu_counter_count), KVADDR, &ret,
                sizeof(long long), "percpu_counter.count", FAULT_ON_ERROR);

causing stack overflow. With this change applied failure didn't happen:

--- a/tools.c
+++ b/tools.c
@@ -6946,7 +6946,7 @@ percpu_counter_sum_positive(ulong fbc)
 {
         int i, count;
         ulong addr;
-        long ret;
+        long long ret;

         if (INVALID_MEMBER(percpu_counter_count))
                 return 0;

[vt-alt]

I'd suggest you add __attribute__((access(write_only, 3, 4))) to readmem definition. With this gcc would detect the overflow at compile time (even with LTO build):

gcc -c -g -DX86 -D_FILE_OFFSET_BITS=64 -DGDB_16_2 -pipe -frecord-gcc-switches -Wall -fdiagnostics-color=always -g -O2  -march=i586 -mtune=generic -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64     tools.c
tools.c: In function 'percpu_counter_sum_positive':
tools.c:6954:9: warning: 'readmem' writing 8 bytes into a region of size 4 overflows the destination [-Wstringop-overflow=]
 6954 |         readmem(fbc + OFFSET(percpu_counter_count), KVADDR, &ret,
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 6955 |                 sizeof(long long), "percpu_counter.count", FAULT_ON_ERROR);
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools.c:6949:14: note: destination object 'ret' of size 4
 6949 |         long ret;
      |              ^~~
In file included from tools.c:18:
defs.h:5868:5: note: in a call to function 'readmem' declared with attribute 'access (write_only, 3, 4)'
 5868 | int readmem(ulonglong, int, void *, long, char *, ulong) __attribute__((access(write_only, 3, 4)));
      |     ^~~~~~~

[liutgnu]

Hi @vt-alt ,

Thanks for reporting the x86 issues on crash. To be honest, the 32-bit x86 crash hasn't been tested for some time, since nowadays, most people have turned to x86_64. So it is not surprised that 32-bit x86 crash code is outdated and faulty. Frankly to maintainers, the 32-bit fix isn't a priority and we need to focus on more important bugs/features. If you or anyone are interested in the 32-bit crash supporting, any PRs are welcomed. You can post your patch to the mailing list devel@lists.crash-utility.osci.io, normally we don't accept patches from github directly. Thanks!