Android NDK Segfault crash related to `sentry_value_set_by_key_n` about sentry-native HOT 6 OPEN

Cc @kahest: While my hopes are low that we will be able to repro this quickly, this is another concrete driver for #962

from sentry-native.

Thanks for the response @supervacuus!

Whenever you add breadcrumbs to your Android application, a call will be made to nativeAddBreadcrumb() because breadcrumbs will be synced to the Native SDK.

You might do this directly via any Interface providing an addBreadcrumb() method or indirectly via an integration that automatically adds them, for instance, during an activity state or navigation change.

Ahh thats good to know. We are calling addBreadcumbs pretty often in our code, basically anytime we log something we are calling addBreadcrumbs. So potentially some log message that occasionally occurs may be causing this bug to pop up.

I understand you currently cannot reproduce this locally, but any additional context (like a logcat output or a tombstone) will be helpful for me. Do you get these reports via Sentry? If so, could you provide a link to one of our employees, [email protected], for one such event?

Yes, we get these reports in Sentry. My workplace has a Slack channel with Sentry reps. I will share the events in that channel with them, and hopefully they will be able to forward them to you. I'll keep you posted with that correspondance.

Does this issue appear only with recent updates?

The first crash was reported on February 16, at that time we were on Sentry Android version 6.29.0 and Sentry android gradle version 3.12.0

from sentry-native.

Hi @aaronjosephsonos, thanks for the report.

There are no places in our code that are directly calling nativeAddBreadcrumb, so thinking something under the hood in Sentry library is making the call.

Whenever you add breadcrumbs to your Android application, a call will be made to nativeAddBreadcrumb() because breadcrumbs will be synced to the Native SDK.

You might do this directly via any Interface providing an addBreadcrumb() method or indirectly via an integration that automatically adds them, for instance, during an activity state or navigation change.

Still, there seems to be a Segfault bug in sentry native code sentry_value.c's file. Thanks in advance for taking a look, and please let me know if there is additional information I can help provide.

I understand you currently cannot reproduce this locally, but any additional context (like a logcat output or a tombstone) will be helpful for me. Do you get these reports via Sentry? If so, could you provide a link to one of our employees, [email protected], for one such event?

This is the second report from Android, in which a freshly created object has either uninitialized members or refers to memory it doesn't own. AFAICR, this also appeared while adding a timestamp to a breadcrumb object (i.e., the first key to be added). Back then, I thought we might hit an issue with the page_allocator because it happened in the signal handler, but this seems to be happening outside any raised signal.

In any case, I will investigate.

Our project has seen the crash approximately 43 times in the past 30 days across 32 users.

Does this issue appear only with recent updates?

from sentry-native.

So potentially some log message that occasionally occurs may be causing this bug to pop up.

I hope it is not a particular log message (although that would make the case easy). The breadcrumbs are a pretty beaten path, especially on mobile, and if the error were in that code, I would be surprised that we're not getting many more reports. But it is hard to say at this point.

I will share the events in that channel with them, and hopefully they will be able to forward them to you. I'll keep you posted with that correspondence.

Thanks!

from sentry-native.

Our internal crash reporting indicates that this was related/caused by a specific app release and most likely not by an SDK issue - we're keeping this open for a bit for further input

from sentry-native.

We also see this issue reported in Google Play. There aren't that many cases, and they seem random. The stack trace points to addBreadcrumb just as described above.

  #00  pc 0x00000000000550d0  /apex/com.android.runtime/lib64/bionic/libc.so (__strlen_aarch64+16)
  #01  pc 0x0000000000069040  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk!libsentry.so (BuildId: b8159197a216f654a35b91c96e5c254da1061566)
  #02  pc 0x000000000006cc10  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk!libsentry.so (sentry_value_set_by_key_n+92) (BuildId: b8159197a216f654a35b91c96e5c254da1061566)
  #03  pc 0x0000000000002368  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk!libsentry-android.so (Java_io_sentry_android_ndk_NativeScope_nativeAddBreadcrumb+560) (BuildId: b145cbf70adb9cf3532c318f02caa53f22523296)
  #04  pc 0x0000000000074eb4  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (art_jni_trampoline+196)
  #05  pc 0x00000000001854e0  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.android.ndk.NativeScope.addBreadcrumb+96)
  #06  pc 0x00000000001a3cf0  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.android.ndk.NdkScopeObserver.addBreadcrumb+800)
  #07  pc 0x0000000000159f14  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Scope.addBreadcrumb+612)
  #08  pc 0x000000000014c578  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Hub.addBreadcrumb+568)
  #09  pc 0x000000000014c308  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Hub.addBreadcrumb+232)
  #10  pc 0x0000000000103b7c  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/oat/arm64/base.odex (io.sentry.Sentry.addBreadcrumb+332)
  #11  pc 0x000000000033b680  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640)
  #12  pc 0x00000000003815d0  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeWithJValues<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, jvalue const*)+652)
  #13  pc 0x000000000056bb64  /apex/com.android.art/lib64/libart.so (art::JNI<false>::CallStaticVoidMethodA(_JNIEnv*, _jclass*, _jmethodID*, jvalue const*)+164)
  #14  pc 0x000000000002e680  /data/app/~~MxMknbSw2B1InjMS8lHCIg==/com.springfin.docutrack-NIPgtRgS3PujIDchTt8h8A==/split_config.arm64_v8a.apk

from sentry-native.