[BUG] Filechannel close takes a long time. about incubator-celeborn HOT 3 CLOSED

Hi, can you share your disk metrics like disk bandwidth or "%iowait" data?
I think this means you need more disks, can you try to add more disks to "rss.worker.base.dirs" ?
@packageman

from incubator-celeborn.

The IO metrics command I am using is iostat -x 2

• IO metrics when writing files using FileChannel(ESSD 100G PL1 disk):

  

• IO metrics when writing files using DataOutputStream(ESSD 100G PL1 disk):

  

When using FileChannel, %iowait is almost 100%, and when using DataOutputStream, only 32% max, and it takes less time to complete the writing of DataOutputStream.

I upgraded the disk to ESSD 500G PL2 and re-tested the FileChannel, and the IO metrics are as follows:

%iowait has dropped to about %20, w/s and wKB/s have been improved, and the writing time has been shortened a lot.

Through the above tests, it is found that the writing speed can be improved by improving disk performance or changing FileChannel to DataOutputStream(which may bring additional CPU and memory overhead for data copying).

from incubator-celeborn.

Thanks for your concentration. You are always welcome to discuss your idea here.

I think you might misunderstand the meaning of %iowait.
IOWait (usually labeled %wa in top) is a sub-category of idle (%idle is usually expressed as all idle except defined subcategories), meaning the CPU is not doing anything. Therefore, as long as there is another process that the CPU could be processing, it will do so. Additionally, idle, user, system, iowait, etc are measurements for the CPU. In other words, you can think of iowait as the idle caused by waiting for io.
So high iowait value means efficient CPU usage and, higher disk performance means lower iowait.

FileChannel vs DataOutputStream, there is some existing reference.
http://web.archive.org/web/20120815094827/http://geekomatic.ch/2008/09.html
http://web.archive.org/web/20120815094827/http://geekomatic.ch/2008/09.html

from incubator-celeborn.