Loading List of List of Strings leads to nans about fastparquet HOT 6 OPEN

Fastparquet only support one-level nested structures (struct, list or map or primitive types) https://fastparquet.readthedocs.io/en/latest/details.html#reading-nested-schema
This is because pandas without arrow doesn't really support anything else either.

from fastparquet.

Thanks, that is indeed in the docs, so not a bug unfortunately..

@martindurant What do you mean it's not supported by pandas? After parquet is read (with arrow by default), I get a nice numpy array of strings there which I can access, map through etc.

Also, wouldn't it be better to hard-crash on parse failure as opposed to silently producing nans? It took me ages to find this since its a just one column out of many I have in the format. Maybe I should file a separate issue on that though..

from fastparquet.

After parquet is read (with arrow by default)

Either you get a native arrow column with a set of arrays in the background or a numpy array ob objects. fastpython could in theory do the latter, but it would be a lot of work and be really slow loopy python code.

from fastparquet.

Sounds good! So what do you think about hard-crashing instead of silently ignoring data in this case? I don't think ignoring a specific column would fly in production systems..

from fastparquet.

A warning may be OK, but I don't think we want to force users to list specific columns they want to read just to avoid those they can't

from fastparquet.

It's probably more frequent to either want all the data (in that case, a user would expect things to be read correctly) or already specify what you want anyway. I haven't seen a library where data is ignored silently just for the syntactic convenience. Also, "ignore_columns" could be an option. But that's just an opinion. Thank you a lot for the very fast library!

from fastparquet.