[BUG] Shuffle read latency is too high when automatic Broadcastjoin is triggered about incubator-celeborn HOT 8 CLOSED

I think it should be caused by too many connections when the partition is too large. I also found a lot of connection creation process from the log as follows

from incubator-celeborn.

If the shuffle.partitions can be reduced, this problem can be solved, but in complex tasks, both large and small tasks exist. If the setting is too small, it is not very suitable for large tasks

from incubator-celeborn.

Does increasing celeborn.network.timeout help? The default value is 240s.
https://celeborn.apache.org/docs/0.2.1-incubating/configuration/#network

Note: you should set it in Spark configuration with the additional prefix spark., then it should be spark.celeborn.network.timeout

from incubator-celeborn.

Is it caused by not disabling spark.sql.adaptive.localShuffleReader.enabled?

from incubator-celeborn.

Is it caused by not disabling spark.sql.adaptive.localShuffleReader.enabled?

@wForget Right, @thomasg19930417 could you turn off spark.sql.adaptive.localShuffleReader.enabled and test again?

from incubator-celeborn.

After testing, it is indeed caused by this parameter, thank you for your reply @pan3793 @wForget @waitinfuture

from incubator-celeborn.

After testing, it is indeed caused by this parameter, thank you for your reply @pan3793 @wForget @waitinfuture

We do added comment about this config, maybe we should highlight it more :)

from incubator-celeborn.

After testing, it is indeed caused by this parameter, thank you for your reply @pan3793 @wForget @waitinfuture

We do added comment about this config, maybe we should highlight it more :)

This parameter was commented out by my mistake, maybe it was automatically commented out when the shell was copied.

from incubator-celeborn.