Android Debugging

Tombstones in Android.

What is tombstone?

Tombstone is a crash occurred in native c/c++ code of AOSP.

Why crash occurs in Native c/c++ Code?

When a process’ native code runs into a fatal error, the Android OS sends a fatal signal to the process, then dumps the process and finally terminates it.

Situations

1. App have some native code and that code is generating some exception.
2. Background service or App is interacting with native side and stuck somewhere
3. Framework itself interacting with native side and get crashed

Visible Symptoms:

1. A foreground application disappears unexpectedly, without “ForceClose” or “ANR” dialog. 
2. If tombstone happened in system server process, the system server is restarted. Black screen is visible for couple of seconds; 
3. No visible symptoms if the application is in background.

Invisible Symptoms:

1. The “debuggerd” process creates a “/data/tombstones/tombstone_0X” (X: 0-9) file.
2. Dumps the crashing thread’s backtrace in the “main” log. The log tag is “DEBUG”, the keyword is “Build fingerprint”.
3. For “userdebug” and “eng” builds, a full core dump file can also be created in the “/sdcard/app_dump” folder.
4. Debuggerd will keep up to 10 tombstones, cycling through the numbers 00 to 09 and overwriting existing tombstones as necessary.

The tombstone informs about-

1. Crashed process and PIDs 
2. Terminated signal and fault address 
3. CPU registers 
4. Call Stack 
5. Stack content of each call
6. Build fingerprint

Tombstone ExampleMain log:

10-17 16:10:09.085 10167 10167 I DEBUG : Build fingerprint: ‘verizon/spyder_vzw/cdma_spyder:2.3.5/6.5.1_51/7:userdebug/test-keys’10-17 16:10:09.085 10167 10167 I DEBUG : pid: 29911, tid: 29936 >>> com.android.browser <<<10-17 16:10:09.085 10167 10167 I DEBUG : signal 7 (SIGBUS), code 128 (?), fault addr 00000000/data/tombstones/tombstone_0X:Build fingerprint: ‘verizon/spyder_vzw/cdma_spyder:2.3.5/6.5.1_51/7:userdebug/test-keys’pid: 29911, tid: 29936 >>> com.android.browser <<<signal 7 (SIGBUS)*, code 128 (?), fault addr 00000000 tls: 452cdf00r0 0000dd58 r1 012e5bbc r2 00000000 r3 00000000r4 012e5b5c r5 afd42584 r6 00000003 r7 012e5b8cr8 0000017f r9 ffffffff 10 00000002 fp ffffffffip 012e5b62 sp 452bcea8 lr 012e5b68 pc afd2f374 cpsr 80000030d0 207373616c63202c d1 3b09094431090969d2 0a7972697078656d d3 4d30330909090967d4 40dcb000002c6e50 d5 0000000049000000d6 000000050006e580 d7 0125fe8042c80000d8 0000000000000000 d9 0000000000000000d10 0000000000000000 d11 0000000000000000

*Unix Signal

How to collect necessary logsTombstone file in /data/tombstones via ADB

adb pull /data/tombstones .

Full Coredump file in /sdcard/app_dump

adb pull /sdcard/app_dump .

AP logs (Both of the following approaches are OK)

Manually collected logs (not available in “user” build)

adb shell dmesg >kernel_log.txt

adb shell su ^^ /proc/kmsg >> kernel_log.txt

adb logcat –b main –b system –b radio –b events >logcat_logs.txt

Aplogd collected logs (not available in “user” build)

adb pull /sdcard/logger .

adb pull /data/logger .

Bugreport

adb bugreport >bugreport.txt

How to analyze tombstone’s necessary preconditions:
Get all AP logs (main/system/event/kernel/radio log) and Bugreport . Use the commands mention above to get all the logs from the phone.

1. Download symbol files , Arm, addr2line and gdb tool 
2. Install Android NDK http://developer.android.com/sdk/ndk/index.html ndk-stack, addr2line, gdb tools can be found in NDK  Or 
3. Find them from AOSP <working-dir> prebuilt/linux-86/toolchain folder 
4. Get tombstone and full core dump file if it’s generated 
5. Download the symbols.tar from jenkins/local build( if tombstone occurred in local build), for the corresponding build on which tombstone happened. 
6. Decode each line in the backtrace using addr2line tool. 

addr2line -C -f -e <path of symbols/system/lib>/<library_name> <address from backtraces>
Working Example
backtrace:#00 pc 00044320 /system/lib/libc.so (tgkill+12)#01 pc 00041f21 /system/lib/libc.so (pthread_kill+32)#02 pc 0001ba4f /system/lib/libc.so (raise+10)#03 pc 00018bf1 /system/lib/libc.so (__libc_android_abort+34)#04 pc 000167b0 /system/lib/libc.so (abort+4)#05 pc 0001a663 /system/lib/libc.so (__libc_fatal+16)#06 pc 0001a67b /system/lib/libc.so (__fortify_chk_fail+18)#07 pc 00016b4c /system/lib/libc.so (__memcpy_chk_fail+16)#08 pc 00000849 /data/app/com.exe.jazz-1/lib/arm/libhellojava_jni.so (Java_com_exe_jazz_Hellojava_crash+36)#09 pc 000ea8a9 /system/lib/libart.so (art_quick_generic_jni_trampoline+40)#10 pc 000e61b1 /system/lib/libart.so (art_quick_invoke_stub_internal+64)#11 pc 003eb4e3 /system/lib/libart.so (art_quick_invoke_stub+170)#12 pc 007fc9ec [stack]stack:ff8827dc 00000000ff8827e0 00000000ff8827e4 00000000Command:<AOSP-working-dir>/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin$ ./arm-linux-androideabi-addr2line -C -f -e
<working-dir>/out/target/product/falcon/symbol/lib/libhellojava_jni.so00000849Output:Java_com_sks_softsolution_Hellojava_crash??:?
For more detail of addr2line<aosp-working-dir>/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin$ arm-linux-androideabi-addr2line -help

Usage: arm-linux-androideabi-addr2line [option(s)] [addr(s)]
Convert addresses into line number/file name pairs.
If no addresses are specified on the command line, they will be read from stdin
The options are:

• @<file> Read options from <file> 
• -a –addresses Show addresses 
• -b –target=<bfdname> Set the binary file format 
• -e –exe=<executable> Set the input file name (default is a.out) 
• -i –inlines Unwind inlined functions 
• -j –section=<name> Read section-relative offsets instead of addresses 
• -p –pretty-print Make the output easier to read for humans 
• -s –basenames Strip directory names 
• -f –functions Show function names 
• -C –demangle[=style] Demangle function names 
• -h –help Display this information 
• -v –version Display the program’s version

Thanks !!! Hope you enjoyed reading !!!