"pthread_create failed: Resource temporarily unavailable" during Grafana dashboard load about zed HOT 3 OPEN

I happened to just now notice some slow performance when accessing that Shasta S3 bucket. On closer inspection it appears compaction has not run successfully on the Autoperf pool since April, even though the zed manage process has been running this whole time. That makes me think that the extra-slow performance puling so many tiny objects may go a long way toward aggravating this problem. If we end up fixing whatever's wrong there such that it compacts again, this crash might magically go away, but if so I could probably create the repro conditions again by manually creating a fragmented, non-compacted pool.

from zed.

tl;dr

I've attached a shell script repro.sh.gz that reliably triggers the repro for me with GA Zed tagged v1.17.0.

Before running, start:

zed -lake s3://brimdataio-lakes-us-east-2/shasta.lake.brimdata.io serve

Details

I made this script by using Wireshark to sniff the queries to the Zed lake API that were being issued by Grafana when loading the dashboard. There's actually more queries in the pcap than I transcribed (i.e., I only made it as big as I needed to trigger the problem) so if this doesn't work for others on the first try (geographic locations that see lower latency to the S3 bucket in Ohio might fare better than me) just say the word and I can add more.

from zed.

Per the previous comment, the Zed services on Shasta have been updated to GA v1.17.0, compaction ran successfully, and indeed the problem no longer reproduces when the repro script is run against the pool data in this compacted form. However, anticipating this, I saved off the pool beforehand in its fragmented state, so to repro the problem, now do:

zed -lake s3://brimdataio-lakes-us-east-2/shasta-fragmented/shasta.lake.brimdata.io serve

...and the repro script still triggers the crash just as before. I just confirmed this with Zed commit 54a5a15.

from zed.