request for supporting for HDFS file system about chainermn HOT 4 CLOSED

hi @sharpstill
Thank you for your request. We will consider this carefully because it is better to be a Chainer's feature, rather than ChainerMN's. In addition, we have limited resource and we don't have much experience on Hadoop/HDFS unfortunately.

Thanks!

from chainermn.

@keisukefukuda Thank you very much, HDFS can setup in only 2 machines to experiment.
By the way, tensorflow has already support HDFS and Yahoo release a tensorflow On spark version.
Maybe this will help you.

from chainermn.

I wonder in what @sharpstill meant by supporting HDFS file system. Even today it is possible to just access data in HDFS; by creating your own Dataset subclass that has list of files or chunks of HDFS file(s), and iterating over them at each get_example() call. See our example on local file system and Chainer's documentation on dataset . With these tools it's already possible to make your job directly read contents from HDFS. Also see Wes McKinney's article on Python HDFS clients comparison for HDFS Python clients choice.

For further support such as data locality awareness and data access optimization, it needs various design decision and we'll be working on it.

from chainermn.

So we're testing some use cases on HDFS but still not feeling the need to special code for HDFS, as we can make own Dataset accessor like this, using libhdfs and PyArrow. For the time being I'm closing this issue, but feel free to reopen for further discussion.

from chainermn.