auto-differentiation / xad Goto Github PK

Allow rolling back multiple adjoints at once in the tape, e.g. for functions with multiple outputs.

The idea is to have an adjoint matrix instead of a vector. Multiple adjoints can be set at once, and then the library's computeAdjoints() rolls all of them back jointly. This would allow to calculate derivatives of functions with multiple outputs more efficiently, possibly using compiler vectorisation as well.

Description

Occasionally, especially in GitHub actions Windows builds with VC++ 14.1, the post-build test discovery times out and makes the build fail. We have to re-run the failed jobs, sometimes more than once, to get it to pass. With a growing amount of tests in XAD, this will only get worse with time and should be fixed.

To Reproduce

Build with Visual Studio 2017 in Debug mode on a slow machine. Test discovery might cause the build to fail. See also here for an example.

Expected behaviour

A clean and working build

Environment:

• OS: Windows
• Version: Server 2019
• Compiler: Visual Studio 2017 (VC++ 14.1)
• CMake Configuration: default, Debug build

Additional context

This can likely be fixed by increasing timeouts during Google Test discovery in the CMake settings.

It would be beneficial for debugging to add an AD mode without expression templates, which makes debugging and compiler error messages simple. Note that this wouldn't be recommended for performance-critical production code.

Add a mode that allows to calculate multiple forward-mode derivatives at once, by tracking multiple derivatives in the active data type.

The current data type has value and derivative, which are both scalars. This would allow to have an array for derivative, where multiple ones are stored and updated with the calculations. Note that the number of derivatives should be configurable by the user at compile-time.

In some applications, the exact same calculations need to be carried out for different inputs repeatedly. Examples are Monte-Carlo simulation where the execution path (branches, etc.) is independent of the input data point.

Re-recording the tape on every path is redundant in this case, and it may be beneficial to record only one path on tape and then generate compiled code from this recording just in time. This code should produce both the value and the derivatives, and optimisations can be applied to increase performance. The overhead of JIT compilation should be compensated for larger numbers of repeated executions.

It would be good to add such a feature to XAD as an option.

When debugging code using XAD's types in Visual Studio, the slot_ member is shown directly while the a_ member (with the value) requires unfolding the base class component. This is especially inconvenient while trying to inspect vectors of values, as shown here:

Visual Studio Natvis files for custom types allow to customise the view for the XAD types. This would allow showing the value directly, and have the tape slot hidden in the raw view for those that are interested. This would provide a consistent experience with native doubles that users are used to.

Note: This only applies to Visual Studio. It should be done for both forward and adjoint modes, as well as higher orders, to be generally useful.

Describe the bug
Reproduction of simple working examples leads to a compiler error:

c++ unitialized_fill_n is not a member of std

To Reproduce
Attempt to build a simple CLI app following code derived from the base example - unfortunately I cannot provide a complete example.

The main app creates a solver class with parameter structs defined using an AD type (either xad::fwd<double>::active_type or xad::adj<double>::active_type). All struct fields initialize derivatives as per examples in the docs and, in adjoint mode, create a tape with recording starting and stopping either side of primary code logic.

Occurs during both forward and adjoint modes.

Required Fix

The bug appears to be fixed by adding #include <memory> to the includes in ChunkContainer.hpp as per C++ reference docs for std::uninitialized_fill_n.

Environment (please complete the following information):

• Tag v1.2.0
• Windows 10
• MSVC++ v14.34.31933
• Visual Studio 2022
• CMake import via CPM.cmake

Additional context

I think there is a decent chance I have missed something here, so please let me know if this is a genuine issue. In the meantime, I can run off a fork with the fix described above in place.

Hi to all. I've been thinking of using XAD in a project I'm implementing, and I notice that if a tape is not instantiated and one calls the setDerivative method, the program crashes. Is there a way to prevent the user from doing so? If not, what about doing something like this:

XAD_INLINE Scalar& derivative()
    {
        auto t = tape_type::getActive();
        // register ourselves if not already done
        if(!t) throw std::exception("Tape not defined");
        if (slot_ == INVALID_SLOT)
        {
            slot_ = t->registerVariable();
            t->pushLhs(slot_);
        }
        return t->derivative(slot_);
    }

What do you think?

Regards,

Add higher-level functions to compute full Jacobians or Hessians, possibly in a multi-threaded fashion,
which drive the underlying tape operations accordingly.

At the moment, our CI/CD pipelines only test Intel-Mac and the SIMD compilation flags are only for the Intel Platform.

We should add support for Apple Silicon (ARM-based), with the following steps:

• Update the SIMD flags to add the appropriate flag for ARM and select it by default on that architecture
• Add a CI/CD pipeline for Apple arm64 architecture
• Test with the latest MacOS / latest Apple Clang release
• Add Apple silicon builds to the Python wheels pipeline
• If possible, also add linux-arm64

XAD should support Eigen data types and related linear algebra operations, calculating values and derivatives efficiently.

Ideally, simply using an XAD type within Eigen vectors or matrices should work out of the box and be efficient. Given that both Eigen and XAD are using expression templates, it may require some traits and template specialisations to make this work seamlessly - and efficiently.

Implement bindings to expose XAD functionality in Python preferably via pybind. It should also be possible to allow the use of NumPy (the fundamental package for scientific computing with Python) with the XAD module in Python, though this might be done as a second step.

The goal is to include the bindings into this repository directly, with the following suggested approach:

• Add a folder bindings/python to hold the build and wrapper code
• Add a CMake option XAD_ENABLE_PYTHON to enable the build (disabled by default)
• Create a typical Python package structure in the folder:
  • Add bindings/python/xad folder with CMakeLists.txt to build Pybind11 module _xad as described in the pybind docs
  • Add __init__.py in bindings/python/xad to import all symbols from the C++ module import * from _xad
  • Add a pyproject.toml file in bindings/python with all the necessary information, dependencies, and package information to make a Python wheel, including test dependencies.
  • Add bindings/python/tests to hold pytest based tests
• Export the first-order adjoint type (xad::AReal<double>) along with the overloaded operators for all basic arithmetics
• Add an xad.Math module, holding all the math functions (like sin, cos, etc) for the newly defined type
• Add the tape Tape class with all its methods
• Add full tests to calculate derivatives of various test programs

All of this should be done driven by pytest tests, and we recommend following the test-driven design approach.

After this, we can look into forward mode and higher orders, possibly using submodules such as xad.adj_1st, xad.fwd_1st, etc., with the same interface. And add numpy support etc.

Useful Resources

• https://pybind11.readthedocs.io/en/stable/compiling.html#building-with-cmake
• https://github.com/pybind/python_example