Missing Native Crash Stacktrace on Android 13

[SakshamSahgal]

Description:

On Android 13 Sometimes, native crashes (e.g., Fatal signal 11 (SIGSEGV)) do not produce a backtrace in logcat or tombstones. The same crash on Android 10 shows a full stacktrace and register dump.

Android 13 Logs -

--------- beginning of crash
05-13 23:58:01.828  8864  9134 F libc    : Fatal signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0 in tid 9134 (pool-34-thread-), pid 8864 (oft.office.word)
05-13 23:58:01.830  8864  8864 D PERFHOST: CodeMarker: 51567
05-13 23:58:01.831  8864  8864 D K2_ULS axbjw OfficeDispatchQueue_CallbackContext:  DQCallbackContextInvoke {"ETW_EventId": 56, "ETW_Keywords": 16, "callbackContext": "0x00007FFF25CE8500", "queue": "0x00007364ED628470", "callback": "0x000073643D85D710", "isIdle": false}
05-13 23:58:01.831  8864  8864 D K2_ULS axbjw OfficeDispatchQueue_CallbackContext:  DQCallbackContextInvoke {"ETW_EventId": 56, "ETW_Keywords": 16, "callbackContext": "0x00007FFF25CE83D0", "queue": "0x00007364DD636AE8", "callback": "0x000073644D75D250", "isIdle": false}
05-13 23:58:01.833  8864  8864 D PERFHOST: CodeMarker: 51566
05-13 23:58:01.835  8864  8864 I K2_ULS bjtct Activity:  ActivityStarted {"Name": "Office.AppDocs.AppDocs.Operation.NotifyEventListener", "CV": "yT9QQthp4EGo8xMndlpOmA.339.28.2.1.13.1.2048.29.2.1.8.1.8.29.2.1.8.1.8.29.2.1.8.1.8.29.2.1.8.1.9.5.12"}
05-13 23:58:01.835  8864  8864 D 0:126885105730808: :Mso logging: [bda784f8][bjtct][Activity][Medium][ActivityStarted {"Name": "Office.AppDocs.AppDocs.Operation.NotifyEventListener", "CV": "yT9QQthp4EGo8xMndlpOmA.339.28.2.1.13.1.2048.29.2.1.8.1.8.29.2.1.8.1.8.29.2.1.8.1.8.29.2.1.8.1.9.5.12"}] 
05-13 23:58:01.836  8864  8864 D 0:126885105730808: :
05-13 23:58:01.836  8864  8864 D 0:126885105730808:  
05-13 23:58:01.836  8864  8864 V K2_ULS a24qy Logging Liblet:  The following traces were triggered while servicing this thread's previous trace. {"cTracesDeferred": 2}
05-13 23:58:01.837  8864  8864 I K2_ULS a8hc1 Word InProc Automation:  [Comment] No more space for testlogs.
05-13 23:58:01.839  8864  8864 D 0:126885105730808: :Mso logging: [bda784f8][a8hc1][Word InProc Automation][Medium][[Comment] No more space for testlogs.] 
05-13 23:58:01.839  8864  8864 D 0:126885105730808: :
05-13 23:58:01.839  8864  8864 D 0:126885105730808:  

Android 10 Logs -

05-30 15:46:15.620 23042 23042 F DEBUG   : 
05-30 15:46:15.620 23042 23042 F DEBUG   : backtrace:
05-30 15:46:15.620 23042 23042 F DEBUG   :       #00 pc 0000000000076dfd  /apex/com.android.runtime/lib64/bionic/libc.so (je_large_dalloc+29) (BuildId: b5c6019a3b4ea61b5e9a2f56319b584e)
05-30 15:46:15.620 23042 23042 F DEBUG   :       #01 pc 0000000000052292  /apex/com.android.runtime/lib64/bionic/libc.so (je_free+1794) (BuildId: b5c6019a3b4ea61b5e9a2f56319b584e)
05-30 15:46:15.620 23042 23042 F DEBUG   :       #02 pc 000000000048e7e9  /data/app/com.microsoft.office.word-z3KNTWWmX--e0IZDV4IrJQ==/base.apk!libmso20android.so (offset 0x6764000) (Mso::Memory::Free(void*)+12) (BuildId: 196331c0fc9f82fe02cac60fb99a0bcf0864da2d)
05-30 15:46:15.620 23042 23042 F DEBUG   :       #03 pc 0000000001059a14  /data/app/com.microsoft.office.word-z3KNTWWmX--e0IZDV4IrJQ==/base.apk!libmsoandroidtest.so (offset 0x8bc4000) (BuildId: 3ee07b7f03c07112e4c3f6076c6e41f585a1b0a5)
05-30 15:46:15.620 23042 23042 F DEBUG   :       #04 pc 00000000010506aa  /data/app/com.microsoft.office.word-z3KNTWWmX--e0IZDV4IrJQ==/base.apk!libmsoandroidtest.so (offset 0x8bc4000) (BuildId: 3ee07b7f03c07112e4c3f6076c6e41f585a1b0a5)
05-30 15:46:15.620 23042 23042 F DEBUG   :       #05 pc 0000000000feb068  /data/app/com.microsoft.office.word-z3KNTWWmX--e0IZDV4IrJQ==/base.apk!libmsoandroidtest.so (offset 0x8bc4000) (BuildId: 3ee07b7f03c07112e4c3f6076c6e41f585a1b0a5)
05-30 15:46:15.621 23042 23042 F DEBUG   :       #06 pc 0000000000feb616  /data/app/com.microsoft.office.word-z3KNTWWmX--e0IZDV4IrJQ==/base.apk!libmsoandroidtest.so (offset 0x8bc4000) (BuildId: 3ee07b7f03c07112e4c3f6076c6e41f585a1b0a5)

Attaching Complete Logs For both Android Versions :

Android_10_and_13_Logs.zip

How can we enable full native crash dumps?

[enh-google]

if you look later in your log for messages from the same pid/tid pair, you can see the problem:

05-13 23:58:02.172   288   288 I tombstoned: received crash request for pid 9134 
05-13 23:58:02.173   288   288 F tombstoned: failed to create temporary tombstone in /data/tombstones: No such file or directory
05-13 23:58:02.173   288   288 E tombstoned: received fatal signal 6

i thought we had cts tests that ensured no OEM fucked up a device in that way, but maybe not. what device(s) have you seen this problem on?

@cferris1000 in case he's aware of any broken devices.

[SakshamSahgal]

We are seeing This on a Nexus 7 API 33 Emulator.

[cferris1000]

I'm not familiar with any device that doesn't create /data/tombstones properly. And, CTS does verify that the directory exists.

It's possible your emulator has not created the directory, or even created it but not with the correct permissions.

[enh-google]

yeah, you should be able to check that with adb shell. (can you adb root an emulator? it's been years since i used one...)

[jmgao]

It wouldn't be the directory not having been created, that's referenced by a directory fd that's opened on tombstoned startup (with a PLOG(FATAL) if we failed to open it).

I'm not sure what would happen if the directory was deleted while we're holding an fd to it, though?

[SakshamSahgal]

@enh-google I’m able to use su on the emulator, so I do have superuser privileges.

[SakshamSahgal]

@jmgao @enh-google , do you need any more info ?

[SakshamSahgal]

@enh-google @jmgao @cferris1000 , what can be a workaround for this Issue ? as we are blocked on this one, this issue was intermittent in Android 10 but it is deterministic for many of our test failures in Android 13.

[cferris1000]

Unfortunately, I don't know what could be causing this. Are you still having the same errors?

Is it possible that /data is full and what you are seeing is not a permissions problem, but some kind of disk full issue. That seems unlikely though.

[SakshamSahgal]

Here are the full Android Runtime Logs I found :

	Line   500: 08-07 03:47:57.958  8050  8050 D AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 2000 <<<<<<
	Line   501: 08-07 03:47:57.961  8050  8050 I AndroidRuntime: Using default boot image
	Line   502: 08-07 03:47:57.961  8050  8050 I AndroidRuntime: Leaving lock profiling enabled
	Line   508: 08-07 03:47:58.028  8050  8050 D AndroidRuntime: Calling main entry com.android.commands.content.Content
	Line   509: 08-07 03:47:58.032  8050  8050 D AndroidRuntime: Shutting down VM
	Line  2126: 08-07 03:48:21.209   594   674 E AndroidRuntime: *** FATAL EXCEPTION IN SYSTEM PROCESS: NativeTombstoneManager:tombstoneWatcher
	Line  2127: 08-07 03:48:21.209   594   674 E AndroidRuntime: java.lang.NullPointerException
	Line  2128: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at java.io.File.<init>(File.java:363)
	Line  2129: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at com.android.server.os.NativeTombstoneManager$TombstoneWatcher.lambda$onEvent$0(NativeTombstoneManager.java:564)
	Line  2130: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at com.android.server.os.NativeTombstoneManager$TombstoneWatcher.$r8$lambda$brzM_6e7cyhhJeR_ISUCuvFg3_s(Unknown Source:0)
	Line  2131: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at com.android.server.os.NativeTombstoneManager$TombstoneWatcher$$ExternalSyntheticLambda0.run(Unknown Source:4)
	Line  2132: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at android.os.Handler.handleCallback(Handler.java:942)
	Line  2133: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:99)
	Line  2134: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at android.os.Looper.loopOnce(Looper.java:201)
	Line  2135: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:288)
	Line  2136: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at android.os.HandlerThread.run(HandlerThread.java:67)
	Line  2137: 08-07 03:48:21.209   594   674 E AndroidRuntime: 	at com.android.server.ServiceThread.run(ServiceThread.java:44)
	Line  2193: 08-07 03:48:21.366  8457  8457 D AndroidRuntime: Shutting down VM
	Line  2196: 08-07 03:48:21.370  8457  8457 E AndroidRuntime: FATAL EXCEPTION: main
	Line  2197: 08-07 03:48:21.370  8457  8457 E AndroidRuntime: Process: com.microsoft.office.excel, PID: 8457
	Line  2198: 08-07 03:48:21.370  8457  8457 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2213: 08-07 03:48:21.438  1203  1246 E AndroidRuntime: FATAL EXCEPTION: launcher-loader
	Line  2214: 08-07 03:48:21.438  1203  1246 E AndroidRuntime: Process: com.google.android.apps.nexuslauncher, PID: 1203
	Line  2215: 08-07 03:48:21.438  1203  1246 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2235: 08-07 03:48:21.568   876   876 D AndroidRuntime: Shutting down VM
	Line  2236: 08-07 03:48:21.568   876   876 E AndroidRuntime: FATAL EXCEPTION: main
	Line  2237: 08-07 03:48:21.568   876   876 E AndroidRuntime: Process: com.android.systemui, PID: 876
	Line  2238: 08-07 03:48:21.568   876   876 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2281: 08-07 03:48:21.666  8453  8453 D AndroidRuntime: Shutting down VM
	Line  2282: 08-07 03:48:21.668  8453  8453 E AndroidRuntime: FATAL EXCEPTION: main
	Line  2283: 08-07 03:48:21.668  8453  8453 E AndroidRuntime: Process: com.microsoft.office.RioApp, PID: 8453
	Line  2284: 08-07 03:48:21.668  8453  8453 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2322: 08-07 03:48:21.698  8462  8462 D AndroidRuntime: Shutting down VM
	Line  2323: 08-07 03:48:21.698  8462  8462 E AndroidRuntime: FATAL EXCEPTION: main
	Line  2324: 08-07 03:48:21.698  8462  8462 E AndroidRuntime: Process: com.microsoft.office.officehubrow, PID: 8462
	Line  2325: 08-07 03:48:21.698  8462  8462 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2518: 08-07 03:48:21.912  8477  8477 D AndroidRuntime: Shutting down VM
	Line  2519: 08-07 03:48:21.912  8477  8477 E AndroidRuntime: FATAL EXCEPTION: main
	Line  2520: 08-07 03:48:21.912  8477  8477 E AndroidRuntime: Process: com.microsoft.office.powerpoint, PID: 8477
	Line  2521: 08-07 03:48:21.912  8477  8477 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2586: 08-07 03:48:22.131  4133  4185 E AndroidRuntime: FATAL EXCEPTION: GoogleApiHandler
	Line  2587: 08-07 03:48:22.131  4133  4185 E AndroidRuntime: Process: com.google.android.gms, PID: 4133
	Line  2588: 08-07 03:48:22.131  4133  4185 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2592: 08-07 03:48:22.132  4133  8592 E AndroidRuntime: FATAL EXCEPTION: [com.google.android.gms.chimera.container.intentoperation.GmsIntentOperationChimeraService-Executor] idle
	Line  2593: 08-07 03:48:22.132  4133  8592 E AndroidRuntime: Process: com.google.android.gms, PID: 4133
	Line  2594: 08-07 03:48:22.132  4133  8592 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2606: 08-07 03:48:22.135   895  8580 E AndroidRuntime: FATAL EXCEPTION: RoleControllerService
	Line  2607: 08-07 03:48:22.135   895  8580 E AndroidRuntime: Process: com.google.android.permissioncontroller, PID: 895
	Line  2608: 08-07 03:48:22.135   895  8580 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2615: 08-07 03:48:22.138  3974  4015 E AndroidRuntime: FATAL EXCEPTION: highpool[0]
	Line  2616: 08-07 03:48:22.138  3974  4015 E AndroidRuntime: Process: com.google.android.gms.persistent, PID: 3974
	Line  2617: 08-07 03:48:22.138  3974  4015 E AndroidRuntime: DeadSystemException: The system died; earlier logs will point to the root cause
	Line  2870: 08-07 03:48:22.454  8622  8622 D AndroidRuntime: >>>>>> START com.android.internal.os.ZygoteInit uid 0 <<<<<<
	Line  2872: 08-07 03:48:22.458  8622  8622 I AndroidRuntime: Using default boot image
	Line  2873: 08-07 03:48:22.459  8622  8622 I AndroidRuntime: Leaving lock profiling enabled
	Line  7400: 08-07 03:48:28.555  8926  8926 I AndroidRuntime: VM exiting with result code 0, cleanup skipped.

From what I infer :

1. System crash due to missing /data/tombstones directory.
2. This led to a fatal system process crash, causing DeadSystemException in all running apps.
3. No native tombstone logs were generated for debugging due to this issue.

@cferris1000 Do these logs help in identifying the possible cause?

[enh-google]

looks like that device is missing https://android-review.googlesource.com/c/platform/frameworks/base/+/2758848