incorrect work of example (as I think) about carma HOT 18 CLOSED

By the way if we want to simply output armadillo matrix without changes:

EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
    arma::Mat<double> mat = carma::arr_to_mat<double>(arr);
    return carma::mat_to_arr(mat);
}

then we also get the wrong output

from carma.

Hi KerimMatlab,

I quickly ran your example against the current master branch and your example works correctly without QtGlobal on MacOS. I'll add a test using this specific style of addition so it can run on windows in the CI/CD pipeline this weekend.

For some reason the memory is being, incorrectly, freed in your case. Have you tried without Qt?
I am not familiar with Qt so it would help eliminate some root causes quickly.

Additionally could you try the dev branch (dev/0.4.0)? This features a fairly comprehensive rewrite of the memory management.

from carma.

Hi @RUrlus,

I include QtGlobal because it enables to run macros that creates my .dll. I think it would be difficult to turn off it (I do not have much experience on programming and CMake).

I tried the dev/0.4.0 branch and the result the same.

There is some important note. Lets suppose that we want to pass numpy array to C++, convert it to Armadillo and then convert back it to numpy. Then We have the code:

#include <armadillo>
#include <carma/carma.h>
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>

#include <QtGlobal>

#define EXAMPLE_EXPORT Q_DECL_EXPORT

EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
    // convert to armadillo matrix without copying.
    arma::Mat<double> mat = carma::arr_to_mat<double>(arr);

    // convert to Numpy array and return
    return carma::mat_to_arr(mat);
}

PYBIND11_MODULE(example, m) {
    m.def(
        "manual_example",
        &manual_example,
        R"pbdoc(
            Example function for manual conversion.

            Parameters
            ----------
            mat : np.array
                input array
        )pbdoc",
        py::arg("arr")
    );
}

In python we have:

import numpy as np
import example
a = np.random.rand(3,2)
b = example.manual_example(a)
a
Out[6]: 
array([[0.1763344 , 0.46757548],
       [0.65567791, 0.46273121],
       [0.54916206, 0.23178091]])
b
Out[7]: 
array([[0.1763344 , 0.65567791],
       [0.54916206, 0.54916206],
       [0.54916206, 0.23178091]])

As you can see there is some regularity in numbers. All the numbers from b contains only numbers from a.
Now I'm going to somehow test numbers when they are in Armadillo matrix (e.g. in C++)

from carma.

@RUrlus I just figured out that Armadillo matrix is fine. Here is the piece of C++ code:

EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
    // convert to armadillo matrix without copying.
    arma::Mat<double> mat = carma::arr_to_mat<double>(arr);

    # we should see this output in Python
    std::cout << mat << std::endl;

    // convert to Numpy array and return
    return carma::mat_to_arr(mat);
}

Now python prints armadillo matrix and we have exactly tha same numbers in Armadillo:

import numpy as np
import example
a = np.random.rand(3,2)
b = example.manual_example(a) 
   0.2922   0.0900    # this is not `b`, it is Armadillo output from C++
   0.1283   0.7299
   0.4878   0.4420
a
Out[6]: 
array([[0.29223221, 0.09004788],
       [0.12834728, 0.72990356],
       [0.48776511, 0.4420319 ]])

This works with both dev/0.4.0 and 0.3.0
Thus I think that something is wrong with carma::mat_to_arr(mat);

from carma.

@RUrlus I just figured out that Armadillo matrix is fine. Here is the piece of C++ code:

EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
    // convert to armadillo matrix without copying.
    arma::Mat<double> mat = carma::arr_to_mat<double>(arr);

    # we should see this output in Python
    std::cout << mat << std::endl;

    // convert to Numpy array and return
    return carma::mat_to_arr(mat);
}

Now python prints armadillo matrix and we have exactly tha same numbers in Armadillo:

import numpy as np
import example
a = np.random.rand(3,2)

@KerimMatlab, something to consider although not the cause of the issue is that you are making a copy unless you set the compile time definition.
Numpy by default create row-major, also called C-contiguous, arrays whereas Armadillo expects column-major (Fortran contiguous) memory.

Could you run the tests on your machine? You can do so with:

1. install pytest: pip3 install pytest or conda install -c anaconda pytest
2. create build directory in the root of the repository and move into it
3. run

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..

4. make install
5. run pytest in the build directory

from carma.

@RUrlus I'm very beginner at Python (I have not used pytests before) and not good in command line operations.
Please help me a little bit more.

1. I successfully installed pytest via conda install -c anaconda pytest
2. I have problems now correct me if I'm wrong: I go to the carma-0.3.0 directory, create there _build dir, go there and write cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 .. ?
   If so I got errors:

(MyEnv38) C:\apps\carma-0.3.0\_build>C:\Qt\Tools\CMake_64\bin\cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.18363.
-- The CXX compiler identification is MSVC 19.27.29111.0
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:101 (add_subdirectory):
  The source directory

    C:/apps/carma-0.3.0/third_party/pybind11

  does not contain a CMakeLists.txt file.


CMake Error at tests/CMakeLists.txt:8 (pybind11_add_module):
  Unknown CMake command "pybind11_add_module".


-- Configuring incomplete, errors occurred!
See also "C:/apps/carma-0.3.0/_build/CMakeFiles/CMakeOutput.log".

These errors most likely connected that I do not have path var to python packages (by the way I'm trying to use Visual Studio 2017)
I prefer using CMakeGUI. I'm going to try it to build test.

from carma.

Sorry I have very little experience outside *NIX OS's when it comes to Python and C++, nor do I have acces to a Windows machine. The best I can do is add some additional tests to our Windows test on travis.

You can also have a look here to see the commands used to build and test on the windows virtual machine: https://travis-ci.com/github/RUrlus/carma/jobs/379189384#L239

from carma.

@KerimMatlab It looks like you don't have the submodules; I should have added:
git submodule update --init

Can you run that and try again?

from carma.

@RUrlus I'm in PyCharm and I just wrote:

(MyEnv38) C:\apps\carma-0.3.0>git init
Initialized empty Git repository in C:/apps/carma-0.3.0/.git/
(MyEnv38) C:\apps\carma-0.3.0>git submodule update --init

Then I write:

(MyEnv38) C:\apps\carma-0.3.0>cd C:\apps\carma-0.3.0\_build
(MyEnv38) C:\apps\carma-0.3.0\_build>C:\Qt\Tools\CMake_64\bin\cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..

And I get the same error:

(MyEnv38) C:\apps\carma-0.3.0\_build>C:\Qt\Tools\CMake_64\bin\cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.18363.
CMake Error at CMakeLists.txt:101 (add_subdirectory):
  The source directory

    C:/apps/carma-0.3.0/third_party/pybind11

  does not contain a CMakeLists.txt file.


CMake Error at tests/CMakeLists.txt:8 (pybind11_add_module):
  Unknown CMake command "pybind11_add_module".


-- Configuring incomplete, errors occurred!
See also "C:/apps/carma-0.3.0/_build/CMakeFiles/CMakeOutput.log".

from carma.

OK, that's not correct. You shouldn't do the git init.
It should be:

• git clone https://github.com/RUrlus/carma.git
• cd carma
• git submodule update --init
• mkdir build
• cd build

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..

• cmake --build . --target INSTALL --config Debug
• pytest

from carma.

@RUrlus worked like a magic 👍
On the las line: pytest I got error (on the picture):

from carma.

Hmm, that's not good. Which branch is this? Arraystore doesn't yet work properly on the dev branch.

Anyway, could you run the following two commands instead of pytest:

• cd tests
• pytest test_arr_to_mat.py test_mat_to_arr.py

from carma.

@RUrlus the version is not written in the folder but I think it is a master as I type git clone https://github.com/RUrlus/carma.git ?

from carma.

@RUrlus I go to the /build/tests and run now:
pytest test_arr_to_mat.py test_mat_to_arr.py
The output:

from carma.

@RUrlus the version is not written in the folder but I think it is a master as I type git clone https://github.com/RUrlus/carma.git ?

Yeah, then it's master.

Strange that the tests pass for the conversions but your example is not working. So maybe Qt is playing a role here.

Although the whole ArrayStore failing is also weird. I'll have a look this weekend.

from carma.

@RUrlus Hi Ralph,

How do you think will this issue be solved? If you are busy that is fine :)

from carma.

@KerimMatlab Hi,

Sorry I've been very busy, new house & job at the same time.
I know the underlying cause of the bug but I don't have an easy solution. I have something in mind but it will take a while.

A work-around is to use Fortran contiguous arrays, np.asarray(arr, order='F') and pass that to C++

from carma.

Resolved in b159447

from carma.