KeyError: 'dir0' in partitioned dataset about fastparquet HOT 3 CLOSED

Just to note that this appears to work if

• you use the pandas or fastparquet API, not spark
• you use the actual root of the dataset, write_path rather than tempdir

So it's a bug in dask, or one could argue not a bug at all.

from fastparquet.

Thanks for getting back so quickly - I was not sure if I should open the issue here or in the dask repo.

You know the codebase much better than me to conclude if this is a bug or not - at least it is an inconsistency that was annoying me to create this issue ;)

from fastparquet.

I would say it is not strictly a bug. The behaviour is a result of giving the wrong directory path, one level higher than the actual root of the dataset. Specifically, when you partition on write, you get directories like "Report_date=value/" inside which the data files sit; this is the "hive" convention. When you then analyse a path like "spark/Report_date=value/part1.parq", it has a mix of this "hive" style information and non-hive part, and the exception is related to this apparent inconsistency.

I think this can safely be closed.

from fastparquet.