vtn21 / human Goto Github PK

The AMASS loader (from npz to TFRecord) is at a nice stage, but it needs a counterpart to load data from the TFRecord files, checking how to recover all fields.

The model is currently training with raw inputs as labels, however these must be filtered using the selection vector. The network has no way of driving non-selected inputs away from zero, thus unsolvable errors will occur.

This will (probably) result in lower errors when performing training and validation.

The load_amass.py script is creating corrupted TFRecord files. This might be caused by the multiple processes accessing a single file.

Try making one process for each file (test, valid, train: 3 total) and checking the files again.

Re-implement the HuMAn model using Keras sub-classing, to allow a customized "call" function. This enables to use the model in auto-regressive setup. The TF Functional API has a fixed generated "call" method, disallowing customization.

Something similar to this TensorFlow tutorial.

Training loop works properly, however complains not finding CUPTI. Try to sort this out by replacing the conda cudatoolkit from the official NVIDIA installation.

After some time training, GPU runs out of memory. See if CUPTI/CuDNN helps. Or, make the model smaller.

Instead of running Normalization.adapt every time, processing the whole training dataset (from disk), save means and variances in npz file.

The tf.data.TFRecordDataset is able to load multiple TFRecord files into a single dataset. This might be the best solution to writing files using multiprocessing (each process writes a single file).

Modify the loader scripts to operate accordingly.