Segfault and data race between OPENSSL_sk_{num, insert, value} in X509 store/lookup about openssl HOT 3 OPEN

Its going to be very difficult to tell what exactly is going on here, without having some significant interrogation of the sources for the python libraries you are using. I say that because, looking at your backtrace:

#0  X509_LOOKUP_by_subject_ex (ctx=0x0, type=type@entry=X509_LU_X509, name=name@entry=0x7fdad0017370, ret=ret@entry=0x7fdadee44550,
    libctx=0x0, propq=0x0) at crypto/x509/x509_lu.c:96
#1  0x00007fdae562f67e in X509_STORE_CTX_get_by_subject (vs=vs@entry=0x7fdad005e500, type=type@entry=X509_LU_X509,
    name=name@entry=0x7fdad0017370, ret=ret@entry=0x7fdad00ee420) at crypto/x509/x509_lu.c:333
#2  0x00007fdae5633982 in X509_STORE_CTX_get1_issuer (issuer=0x7fdadee44660, ctx=0x7fdad005e500, x=0x7fdad001cae0)
    at crypto/x509/x509_lu.c:736
#3  0x00007fdae563c4a8 in get1_trusted_issuer (cert=0x7fdad001cae0, ctx=0x7fdad005e500, issuer=0x7fdadee44660)
    at crypto/x509/x509_vfy.c:2982
#4  build_chain (ctx=0x7fdad005e500) at crypto/x509/x509_vfy.c:3103

In stack frame (2) we see X509_STORE_CTX_get1_issuer is being called, which was passed ctx=0x7fdad005e500, which, so far so good.

In stack frame (1) We see X509_STORE_CTX_get_by_subject is being called, with the first argument being vs=vs@entry=0x7fdad005e500. If I remember my gdb syntax properly, the vs@entry emission is gdbs way of opportunistically saying "when we entered the function, the value of this parameter was 0x7fdad005e500". The address matches the ctx pointer from frame (2), so thats good. The significance of the @entry notation however is that gdb detected an unexpected change to the value.

in stack frame (0) we see this alteration was realized, based on the fact that ctx is now NULL.

Looking at the source for these three stack frames, we can see that ctx is passed from one to the other without alteration, so something external to those functions is impacting its value

My guess is that, given the python code you've posted, that something in the boto3 client is sharing library contexts between threads, and one client is exiting, possibly deallocating the context while another thread is still using it. Of course, thats just a guess, we would have to go digging through the boto3 client library and its subordonates to figure out exactly whats going on.

from openssl.