pndaproject / platform-data-mgmnt Goto Github PK

Hi,

Is it possible to get hdfs-cleaner working/packaged with non-pnda HDFS?
We're running HDP and currently cleaning old files using a NFS mount (via HDFS NFSGateway). And we use find to delete old files. It's slow and buggy.
I really haven't found any good solution to delete old files in HDFS. I would like to give your cleaner a try.

Updating a dataset configuration from the console-ui does not work if dataset is in /user/pnda/datasets/topics.

log output:

WARNING 404 PUT /api/v1/datasets/$DATASET-NAME (10.42.X.X) 0.55ms

If data-service starts without a successful connection to hbase, It will not connect afterwards.

A continuous WARNING log is output:

WARNING Failed to read table from hbase error('HDBDataStore' object has no attribute 'conn_pool')

Happybase is retrieving data as binary strings. We should encode/decode to python strings.

Updating a dataset configuration from the console-ui does not work if dataset is in /user/pnda/datasets/topics.

log output:

WARNING 404 PUT /api/v1/datasets/$DATASET-NAME (10.42.X.X) 0.55ms