j-c-cook / cpgfunctionep Goto Github PK

The interp1d function listed below needs modified to hold the final value rather than throwing an error for extrapolation.

The single input multiple data (SIMD) instruction set of OpenMP could be utilized to vectorize the finite line source integration approximation.

There is a segmentation fault that is occurring inside of the Eigen library upon the API call to cpgfunctionEP via the UBHWTCalc input on the Linux CI machines and inside of CLion on my local. This segmentation fault has been around and unnoticed for quite some time. The segmentation fault does not exist if I build energyplus from source, and then pass a weather file and the GSHP-GLHE-CalcGFunctions-cpg.idf test file. The segmentation fault does not exist when I build cpgfunctionEP in my own repository as a standalone.

mkdir build && cd build && cmake .. && make energyplus -j 16
cp ../testfiles/GSHP-GLHE-CalcGFunctions-cpg.idf Products/ && cp ../weather/USA_OK_Oklahoma.City-Will.Rogers.World.AP.723530_TMY3.epw Products/
cd Products/ 
./energyplus -w USA_OK_Oklahoma.City-Will.Rogers.World.AP.723530_TMY3.epw GSHP-GLHE-CalcGFunctions-cpg.idf

The resulting long time-step g-function located in the eplusout.glhe file:

        2.5227814172726757,
        2.7705529489647103,
        3.0181546610457,
        3.2655977297246355,
        3.516846551928096,
        3.7912271892255647,
        4.127136395615098,
        4.56335062650864,
        5.116250608380114,
        5.774475535544771,
        6.509681059546577

I have gone to the j-c-cook/cpgfunctionEP repository and created a test file that performs the same function call to the uniform_borehole_wall_temperature() function. That example is below (Note: I have modified the number of segmnents in the EnergyPlus call to be 1 to reduce the total number of sources so the matrices are more palpable):

//
// Created by jackcook on 8/16/21.
//

#include <cpgfunction/coordinates.h>
#include <cpgfunction/boreholes.h>
#include <cpgfunction/utilities.h>
#include <cpgfunction/gfunction.h>


int main() {
    // -- Definitions --
    // Coordinate geometry
    int Nx = 2;
    int Ny = 2;
    double Bx = 5.;
    double By = 5;

    // -- Configurations --
    // Get x,y coordinates for a rectangle
    std::string shape = "Rectangle";
    std::vector<std::tuple<double, double>> coordinates =
            gt::coordinates::configuration(shape, Nx, Ny, Bx, By);

    // -- Borehole geometry --
    double H = 100;  // height of the borehole (in meters)
    double D = 1;  // burial depth (in meters)
    double r_b = 0.057;  // borehole radius (in meters)

    // -- boreField --
    std::vector<gt::boreholes::Borehole> boreField =
            gt::boreholes::boreField(coordinates, r_b, H, D);

    // Thermal properties
    double alpha = 1.0e-6;      // Ground thermal diffusivity (m2/s)

    // Number of segments per borehole
    int nSegments = 1;

    double duration = 1;
    std::string units = "year";
    double const expansion = 0.5;

    vector<double> time = gt::utilities::time_vector_constant_expansion(
            H, alpha, duration, units, expansion);

    std::vector<double> gFunction =
            gt::gfunction::uniform_borehole_wall_temperature(boreField,
                                                             time,
                                                             alpha,
                                                             nSegments,
                                                             true);
    for (size_t i=0; i<gFunction.size(); i++) {
        std::cout << gFunction[i] << std::endl;
    }

}

Outputs (Note the exact match out to 16 decimal places to the ./energyplus UBHWT g-function above):

2.522781417272676
2.77055294896471
3.0181546610457
3.265597729724635
3.516846551928096
3.791227189225565
4.127136395615098
4.56335062650864
5.116250608380114
5.774475535544771
6.509681059546577

The segmentation fault in EnergyPlus occurs upon the first partial pivot LU decomposition API call. The A and B matrix are shown below:

A (EnergyPlus -> cpgfunctionEP)

    2.52278 1.71313e-14 1.71313e-14 8.60119e-27          -1
1.71313e-14     2.52278 8.60119e-27 1.71313e-14          -1
1.71313e-14 8.60119e-27     2.52278 1.71313e-14          -1
8.60119e-27 1.71313e-14 1.71313e-14     2.52278          -1
        100         100         100         100           0

B (EnergyPlus -> cpgfunctionEP)

 -0
 -0
 -0
 -0
400

A (cpgfunctionEP)

    2.52278 1.71313e-14 1.71313e-14 8.60119e-27          -1
1.71313e-14     2.52278 8.60119e-27 1.71313e-14          -1
1.71313e-14 8.60119e-27     2.52278 1.71313e-14          -1
8.60119e-27 1.71313e-14 1.71313e-14     2.52278          -1
        100         100         100         100           0

B (cpgfunctionEP)

 -0
 -0
 -0
 -0
400

The A and B matrices appear identical. The g-function results of the energyplus executable are identical to cpgfunctionEP. The reasoning for the segmentation fault in Linux on the CI machines when running tests, when running tests (ctest) on my local and when running the input file on CLion are unknown.

Given that this is a timely manner due to the deadline for API changes for the next release to be finalized by this Friday, I am considering replacing the Eigen LU partial pivot decomposition call with a hand written one or finding a standalone.

The tests don't need to be ran by outside users. nlohmann uses something called MainProject.

This library is dependent on boost 1.71.0, Threads, LAPACK, OpenBLAS and gFortran. These third_party dependencies will be made git subtrees or pasted in for the following two reasons:

1. Not every user knows how to install libraries for use in C++. This calculation should be made easily usable by all engineers, not just those whom are computer savvy
2. EnergyPlus developers do not want to have to worry about what versions I need in my build. EnergyPlus should build without needing to install any libraries.

There are 5 total heights computed (24, 48, 96, 192, 384). Each height has a range of total burial depths where the ideal number of segments required to match a converged UIFT solution is stored. A double interpolation will be made available which interpolates over height and total drilling depth.

With the removal of BLAS/LAPACK, speed was lost. The size of the tests are now too large and take too much time to run. Rather than running tests for 100 boreholes, the size should be reduced so that the tests are not running for too long.

On Wednesday, June 23, 2021 Edwin mentioned that Neal Kruis had made use of boost in the third party tool kiva of EnergyPlus. He mentioned that it was not a complete set of the boost library, but rather only the files that are necessary to run the program.

This can surely be done for cpgfunction, given that the only libraries made use of from boost are asio for the thread pool and math for the gaussian quadrature integration.

In an effort to reduce the number of files in this library, the nlohmann json third party library will only make use of the single header include file.

The approximation is given in [1].

[1] Cimmino, M., 2021. An approximation of the finite line source solution to model thermal interactions between geothermal boreholes. International Communications in Heat and Mass Transfer, 127, p.105496.

If the number of time steps described nt is less than 6, then there is a memory error.

This pure C++ implementation of the cpgfunction library is slower. A major performance bottleneck is the LU decomposition to solve for the time varying heat extraction rates. The Eigen library was proven to be unusable (atleast for now) in EnergyPlus due to segment faults being thrown in Linux (see NREL/EnergyPlus#8708 (comment)).

One idea could be to utilize a this matrix_lu library.

There should be an optional argument for the g-Function interface that allows the user to specify the number of threads. The default should be all threads.

Use a double colon in target name to give alias or imported target. (cmake-policy 28)

https://cmake.org/cmake/help/latest/policy/CMP0028.html

The jcc::LinearAlgebra library is currently a git submodule. The cloning of a project with submodules is more complex, therefore it was mentioned that subtrees are the superior option for purposes of interfacing with EnergyPlus.

The g-function API currently accepts the adaptive discretization as a boolean argument. This does not make sense because it also accepts the number of segments as an argument. The adaptive discretization struct and method can compute the ideal number of segments and that can be passed into the UBHWT g-function method.

Some of the g-function results are wonky if the time values are not what were originally tested on.

The minimal LU-decomposition library is going to replace Eigen's LU decomposition call.

https://github.com/j-c-cook/LU-Decomposition

EnergyPlus for some reason needed to pass the height as a copy, so that's what is happening here.

The OpenBLAS library was found to be significantly faster BLAS and LAPACK routines than those found in netlib. The OpenBLAS library has a CBLAS library that can be utilized to deprecate the Fortran dependency for BLAS functions. However, the LAPACK library in OpenBLAS is completely written in Fortran, so any LAPACK function calls need -lgfortran.

(See OpenBLAS dependencies. Would we be within EnergyPlus standard practices if we had a third party library that needed a C compiler? If not, we could tell cmake to consider C files as C++.)

However, even if this is done, the gesv routine of LAPACK is necessary. The speed of OpenBLAS vs. netlib was exponential for this function. There is a C library (LAPACKE) by netlib that could be made use of. The implementation of LAPACK functions are dependent on BLAS routines. Therefore, it could be possible that a combination of CBLAS by OpenBLAS and LAPACKE by netlib could provide similar speed performance.

Before looking into this technique for a removal of Fortran dependence, it would be nice to know answers to the following questions:

• Is EnergyPlus is okay with being dependent on code written in C?
• If they are, does it need to be compiled with g++, or is it assumed that all EnergyPlus developers have a gcc compiler in their path?

The use similarities false is not currently working because it was setup to fill the 3D matrix. The 2D matrix must now be filled.

If we can make use of chocolatey to install BLAS and LAPACK, then CMake could link the libraries to the installed dll's.

This may be an issue discussing that idea:
ms-iot/ROSOnWindows#159

Given that there is a maximum size that can be allocated for std::vector, it is not suitable to make h_ij 1 dimensional. However, the BLAS functions can be more readily used if the packed segment response matrix is indexed time x nSum rather than nSum x time. This is a transpose.

For computing g-functions live time, the best way to keep the number of time steps low may be to accept a total period of time and use Eskilson's original points.

Pass the type of linkage between the linked libraries:

https://cmake.org/pipermail/cmake/2016-May/063400.html

The Eigen library should be able to be installed via choco (on windows), brew (on mac) and apt-get (on Linux).

Windows: https://community.chocolatey.org/packages/eigen
Mac: https://brewinstall.org/Install-eigen-on-Mac-with-Brew/
Linux: https://askubuntu.com/a/936560

CMake find Eigen: https://stackoverflow.com/a/52230189

The EnergyPlus team does not want nlohmann json to be a dependency of the tool. There will never be a need for energy plus to read or write files using cpgfunctionEP. Therefore, the best course of action is to depracate (comment out) the functions that use nlohmann::json.

The g++ MacOS cmath standard library appears to not have std::expint as an available function.

The adaptive discretization scheme needs to interpolate over both height and total drilling depth. Therefore, a bilinear interpolation function is necessary.

For comparison purposes to cpgfunction, it would be nice to know which steps have been slowed down, and which sped up.

There is an issue with the constant time expansion function that gets called here in this example:

At some point the CMakelists file was modified in a way that may access an Eigen installed library. The Eigen library referenced will always be the local library.

The start of issue 26 was going to remove the local Eigen library from the package and just use CMake to find the package. The EnergyPlus dev team pushed back on this and said that nothing should be required on the system for install. Therefore, the scope of the issue was changed. However, the beginning of the changes to the CMake file must have remained, and need to be reverted.

Two additional arguments will be added to the UBHWT g-function call

• adaptive: bool = true
• nSegments: int = 12

The code for setting the number of segments will look something like:

if (adaptive) {
    nSegments = call adaptive_discratization(nbh, height)
}  // if (adaptive)
else {
    nSegments = nSegments
}

The jcc::LinearAlgebra library is currently a git submodule. The cloning of a project with submodules is more complex, therefore it was mentioned that subtrees are the superior option for purposes of interfacing with EnergyPlus.

Here is what Matt did in the E+ repo.

EnergyPlus has "clang-tidy" checks. It is clang-tidy for the instantiation order to be in the same order that the arguments are passed into a struct.

The tests are only built when the CMAKE_PROJECT_NAME is equal to the PROJECT_NAME. Ie. If only cpgfunctionEP is being built, then the tests are ran. If that is not true, then there is nothing.

Now, if that is not true, the Eigen library will be read from EnergyPlus.

The current design tool is making use of the LinearAlgebra project that wraps lapack and blas function calls. However, extensive funding (via a joint project of the U.S. Department of Energy's Office of Science and National Nuclear Security Administration) and work appears to have already been put into making C++ function calls for BLAS and LAPACK.

The C++ API packages are named blaspp and lapackpp. It is believed that these packages can provide the same speed as what is currently being used, and be more robust. The CMakeLists is more robust, and doesn't only search the system for OpenBLAS. Though it is likely that OpenBLAS or ATLAS will provide the best performance of BLAS and LAPACK.