theochem / cgrid Goto Github PK

Cellcutoff needs CI testing and conda packaging first.

The build and runtime dependencies can be generated and installed as follows:

conda render tools/conda.recipe > tools/conda.recipe/rendered.yaml
python -c "import yaml; f= open('tools/conda.recipe/rendered.yaml'); data = yaml.load(f); f.close(); deps = data['requirements']['build'] + data['requirements']['run']; print(' '.join(dep.split()[0] for dep in deps))" | xargs conda install --use-local

It may be better to put all of that into one convenient python script. (Needs to be discussed.)

It looks like the install in .travis.yml 106-118 isn't necessary in a tagged release. Since we use conda build on tagged releases, everything should be installed in its own build root. Installing the dependencies otherwise should only be a waste of time.

- conda render tools/conda.recipe > tools/conda.recipe/rendered.yaml
- conda render ${PYDIR}/tools/conda.recipe > ${PYDIR}/tools/conda.recipe/rendered.yaml
- CONDA_DEPENDENCIES=$(python -c "from yaml import load;
    req1 = load(open('tools/conda.recipe/rendered.yaml'))['requirements'];
    req2 = load(open('${PYDIR}/tools/conda.recipe/rendered.yaml'))['requirements'];
    deps = req1['build'] + req1['run'] + req2['build'] + req2['run'];
    print(' '.join(set(dep.split()[0] for dep in deps if not dep.startswith('${CONDA_PKG_NAME_CPP}'))))")

- conda install -q python=${MYCONDAPY} cppcheck doxygen anaconda-client ${CONDA_DEPENDENCIES}

Rename qcgrid to cgrid everywhere

@matt-chan We could use .travis.yml as its own template file, reducing the number of files to keep an eye on. The main downside is that it requires a file structure as follows:

- "Some header yaml part that contains variables and other things that tend to change between projects"
# EVERYTHON BELOW THIS LINE MAY BE OVERWRITTEN BY THIEF!!!
# stolen from: url    <--- optional line
- "Template yaml"

We can write a python thief script that takes one argument: a URL pointing to the file to steal, e.g. https://raw.githubusercontent.com/theochem/cellcutoff/master/.travis.yml or a stolen file to be updated .travis.yml We may even add a safety check that a file to steal should not be stolen from somewhere else, just to maintain sanity.

In either case, the file to steal and the one to replace (if present) must contain the above structure. If not, the thief script should stop without making changes and print an appropriate error message.

1. URL:
   1. If the file does not exist locally, steal it entirely and add the # stole from: url line.
   2. If the file does exist, replace everything below the separator line. Also add or update the # stolen from: url line.
2. File (must exist): look up the # stolen from: url line and update from the url on that line, as in case 1.ii. Only works if # stolen from is present.

In any case, even with the current templating method, it would be good to make the .travis.yml completely self-contained.

This can be copied over from HORTON to large extent. In addition to the pruned grids in HORTON, it would be useful to also implement SG1 (https://doi.org/10.1016/0009-2614(93)80125-9) to facilitate comparisons with Gaussian and QChem.

We can't link successfully unless all the dependencies are using the same compiler/libstdc++. This is particularly a problem because old versions of cellcutoffs were built with gcc4.8/libstdc++(gcc) and new versions of conda python and osx qcgrids is now built with clang/libc++(llvm). The symbols aren't the same between the two anymore, so builds are failing.

Depends on theochem/cellcutoff#8 (The same approach can be followed.)

The following should be safe/needed for caching:

• miniconda/LICENSE.txt
• miniconda/bin
• miniconda/conda-meta (meta info on installed packages)
• miniconda/envs
• miniconda/etc
• miniconda/gcc (not sure why it exists)
• miniconda/include
• miniconda/lib
• miniconda/lib64
• miniconda/libexec
• miniconda/sbin
• miniconda/share
• miniconda/ssl
• ...

Not good for caching:

• miniconda/conda-bld
• miniconda/locks
• miniconda/pkgs
• miniconda/var

There may be others.

This may not be a good idea because of things not to cache. See also broadinstitute/viral-ngs#290

This is how to enable it: https://docs.travis-ci.com/user/caching/#Arbitrary-directories

Other things to consider for caching in Travis:

• CCache: https://docs.travis-ci.com/user/caching/#ccache-cache (has to be enabled in the build, probably disabled when building packages for deployment, just to be sure they are clean)
• Pip packages: https://docs.travis-ci.com/user/caching/#pip-cache (may not work as we install pip packages into the conda env)

Looks broken. Need to fix later.

The PR builds fail because the include paths aren't added by conda unless it's during a conda-build.

We need to modify the cmakelists.txt with find_path() to look at the conda prefix. I'm assigning it via the CMAKE_PREFIX_PATH environmental variable.

This can be done more easily after it starts working for cellcutoff

It gets linked dynamically, so gtest libraries are needed at runtime.

Tagged commits:

• Build packages with good optimization flags
• Build for all Python versions and OSes of interest.

Other commits and PRs:

• Build packages with development/debug/coverage flags
• Build just for one Py3.6 on Linux

Ideally, we'd have only three different .travis.yml files in the organization:

• For pure python projects
• For python+cython projects
• For c++ projects with a python/cython wrapper

These .travis.yml files can be made configurable by sourcing shell scripts that set environment variables. At least two such shell scripts should be present:

1. Environment variables that can be made visible.
2. Encrypted tokens and passwords. Because these sit outside the .travis.yml file, the secret file has to be encrypted as a whole. More details can be found here: https://docs.travis-ci.com/user/encrypting-files/ Extra care has to be taken to not reveal the contents of the secret file in the Travis log.

With this setup, we should select three projects (one of each kind) where improvements to the .travis.yml file are tested before they are applied to other projects. For now the following seems to be a good choice:

• Pure python: derivcheck
• Python+Cython: python-cython-ci-example
• C++ with wrapper: qcgrids

(We can always update later. This coiche has to be a tradeoff between speed of testing new ideas and complexity of the project, such that all features of the .travis.yml file get tested.)

We have something reasonably good in HORTON. It would however be good to see if we can make a better 1D integration and interpolation, e.g. using https://github.com/scipy/scipy/blob/master/scipy/special/orthogonal.py

This can be done in the same way as in the hold horton.grid, by providing a list of GridFunc objects, in the canonical order of the corresponding spherical harmonics. Recursion relations for computing the spherical harmonics efficiently can be found in horton 2.x

The other travis build types have doc builds. We removed it for testing, but we should add it back in soon.

We should have a C++ CubicSpline class with a Python wrapper, which is derived from the ScalarFunction class, such that it can be used for an efficient evaluation of spherical functions and (spherical harmonic)x(radial function) within a cutoff. Basic features that should be supported:

• Arbitrary positions of the spline grid points should be supported. Don't work with transformations of the x-axis.
• First and Second derivatives of the spline, no inverses.
• Constructor should work with either function values only (and solve for derivatives), or function values and derivatives on the grid.

Another linking problem. At least it's on linux this time...

I don't know what's going on here. I think there's undefined symbols in some of these programs (although the error suggests there's some libraries compiled with debug symbols, there aren't. I checked with readelf).

The error is at ln 1621-1649. https://travis-ci.org/theochem/qcgrids/jobs/302337015

I'm not sure if we're getting this issue because we're using gtest in conda as opposed to gtest from the github repo. Cellcutoff works fine (and it uses gtest from github). Another possibility is that the gtest package in conda is borked, but that's somewhat unlikely. I did take a look at the meta.yml from the package though, and it's always using gcc/g++, and the versions aren't specified. There's a possibility that we're running into issues because anaconda gtest is linked with an old version of libstdc++. Anaconda was supposed to update their builds, but I guess not all of them switched over (Python did already, which was why we had to do all these rebuilds in the first place).