google / space Goto Github PK

The features of loading a TFDS datasets (append_array_record) and Parquet files (append_parquet) don't copy/rewrite the source files. As the consequence, a Space's dataset will be split across two locations: the original files and the new Space storage directory.

To support an option that first copies or moves the source files to the Space storage directory. Note that such copy should be still faster than writing files in the normal append methods.

The Ray runner still has many local operations, only offload some steps that are trivial to parallelize. Ideally all data and metadata processing (include commit) of Ray runner should run in the Ray cluster instead of the client.

The error should be because Ray dataset does not support Ray client, as mentioned in ray-project/ray#41333. Working on a fix.