Segmentation errors for gfal-copy's GridFTP plugin on RHEL8 about gfal2 HOT 7 OPEN

Hello Xavier,

Repeating my message from cern-fts/gfal2-util#6 , given this is about GridFTP, it's not something we want to deeply investigate. The Gfal2 plan is to abandon GridFTP support.

Maybe I can help you get a stacktrace. Given the gfal2-util clients are shell wrappers, maybe you can try to do the copy via Python3 directly:

import gfal2

ctx = gfal2.creat_context()
params = ctx.transfer_parameters()

ctx.filecopy(params, "src", "dst")

Maybe this way, you'll have better chances of producing a coredump.
Once the coredump is available, we can look whether it's something evident in Gfal2 or coming from the underlying GridFTP library. If it's in Gfal2, I'm interested whether this could affect the other plugins (e.g.: http, xrootd).

Cheers,
Mihai

from gfal2.

Thank you for the suggestion, Mihai. I'll give it a try.

from gfal2.

This small Python script was running for about 90 minutes without breaking due to a segmentation error...

import gfal2

ctx = gfal2.creat_context()
params = ctx.transfer_parameters()

ctx.filecopy(params, 'file:////bin/bash', 'gsiftp://ppsgridftp-kit.gridka.de:2811//pnfs/gridka.de/dteam/disk-only/force-core-dump.tmp')
while True:
  ctx.filecopy(params, 'gsiftp://ppsgridftp-kit.gridka.de:2811//pnfs/gridka.de/dteam/disk-only/force-core-dump.tmp', 'file:////dev/null')

Anyway, our regular workflow (miraculously) produced three core dump files, of which I've attached the human readable information produced by coredumpctl (gfal2.coredumpctl.txt). If you need the actual core dump files, I can provide them. If you need me to do something specific with the core dump files, please tell me.

Best regards,
Xavier.

from gfal2.

Hello,

It does look like a double free for the same GSI security context:

                Stack trace of thread 3717519:
                #0  0x00007f19af2bfe31 __libc_free (libc.so.6)
                #1  0x00007f19b05648ec ASN1_OBJECT_free (libcrypto.so.1.1)
                #2  0x00007f19b0575198 asn1_primitive_free.localalias.0 (libcrypto.so.1.1)
                #3  0x00007f19b0575618 asn1_template_free (libcrypto.so.1.1)
                #4  0x00007f19b05752d6 asn1_item_embed_free (libcrypto.so.1.1)
                #5  0x00007f19b0575618 asn1_template_free (libcrypto.so.1.1)
                #6  0x00007f19b05752d6 asn1_item_embed_free (libcrypto.so.1.1)
                #7  0x00007f19b0575539 ASN1_item_free (libcrypto.so.1.1)
                #8  0x00007f199917d697 globus_gsi_proxy_handle_destroy (libglobus_gsi_proxy_core.so.0)
                #9  0x00007f199ac7db8d gss_delete_sec_context (libglobus_gssapi_gsi.so.4)

Could you provide the coredump for gfal-rm -vvv -t 20 gsiftp://atlasgridftp-kit.gridka.de:2811/[..] ?

Cheers,
Mihai

from gfal2.

Here you go
core.python3.1000.1b33bd67a03e41faa9fc307a01ea236f.3717501.1703065389000000.lz4.txt. Please remove/ignore the .txt suffix when using the file. I had to append it because GitHub doesn't allow me to upload .lz4.

from gfal2.

Hi @mpatrascoiu,

did the coredump help you in any way?

Ciao,
Xavier.

from gfal2.

Hi @mpatrascoiu,

the core dumps keep happening for our setup; around 10 times per day, which is less than 1 % of all tests, but still...

Of course, you are free to simply reject the issue on the basis that nobody cares about either gsiftp or RHEL-8 anymore. Simply close this issue and I'll stop bothering you, since we've implemented retries in case of segfault errors in our tests.

Kind regards,
Xavier.

from gfal2.