swapneelm / deepjetcore Goto Github PK

The internal format that DeepJet writes to while converting from ROOT Ntuples to Numpy Arrays needs to be done away with. The new workflow should write directly into TFRecords.

TFRecords can then be read into a tf.data.Dataset and manipulated (batched, shuffled, custom-function applied to all elements, reshaped, etc.).

For reference on converting data from ROOT Ntuples to TFRecords, refer to the [Batch] Data Preprocessing Notebook in TrackingNtuples.

DeepJet is now completely pip-installable (this includes auto-compiling the custom C++ extensions that earlier required a manual make install).

This needs to be tested thoroughly because it still results in an ImportError for the c_meanNormZeroPad.so shared library when it is not manually compiled.

Further, this depends on the path where the compiled shared libraries (DeepJetCore/compiled/src and others in the compiled folder) are installed.

It is designed to be installed in the conda environment thus lies in the python/site-packages directory when pip install DeepJetCore is run, but this needs to be tested on different setups (LXPLUS and FlatIron for instance) just to confirm in independent environments that DeepJet can be installed successfully and automatically without manual git-clone of DeepJetCore (note that a manual git-clone of DeepJet will still be required to run the sample models).

Need to test the old (Python 2, DL4Jets:master) and new version (Python 3, SwapneelM:python-package) on the GPUs at the FlatIron Institute to see if we can port DeepJet to other systems (and get faster performance).

Documentation is available, but we need a complete set of docs relating to what each function (or at least submodule i.e. DeepJetCore.preprocessing, DeepJetCore.evaluation, etc.) is meant for and the parameters that go in and out of each function within these submodules.

Bare minimum: Explain how DeepJetCore is structured and the idea behind each submodule if not docstrings for each function.

See also the process of generating documentation (and a PDF file containing all docs) based on the ReadTheDocs theme.

You want to start off by building your own version of the docs using the make html or other commands available in this Makefile.

Note that this is on the python-package branch, not master.