Comments (18)
By the way if we want to simply output armadillo matrix without changes:
EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
arma::Mat<double> mat = carma::arr_to_mat<double>(arr);
return carma::mat_to_arr(mat);
}
then we also get the wrong output
from carma.
Hi KerimMatlab,
I quickly ran your example against the current master branch and your example works correctly without QtGlobal
on MacOS. I'll add a test using this specific style of addition so it can run on windows in the CI/CD pipeline this weekend.
For some reason the memory is being, incorrectly, freed in your case. Have you tried without Qt?
I am not familiar with Qt so it would help eliminate some root causes quickly.
Additionally could you try the dev branch (dev/0.4.0
)? This features a fairly comprehensive rewrite of the memory management.
from carma.
Hi @RUrlus,
I include QtGlobal
because it enables to run macros that creates my .dll
. I think it would be difficult to turn off it (I do not have much experience on programming and CMake).
I tried the dev/0.4.0
branch and the result the same.
There is some important note. Lets suppose that we want to pass numpy array to C++, convert it to Armadillo and then convert back it to numpy. Then We have the code:
#include <armadillo>
#include <carma/carma.h>
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <QtGlobal>
#define EXAMPLE_EXPORT Q_DECL_EXPORT
EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
// convert to armadillo matrix without copying.
arma::Mat<double> mat = carma::arr_to_mat<double>(arr);
// convert to Numpy array and return
return carma::mat_to_arr(mat);
}
PYBIND11_MODULE(example, m) {
m.def(
"manual_example",
&manual_example,
R"pbdoc(
Example function for manual conversion.
Parameters
----------
mat : np.array
input array
)pbdoc",
py::arg("arr")
);
}
In python we have:
import numpy as np
import example
a = np.random.rand(3,2)
b = example.manual_example(a)
a
Out[6]:
array([[0.1763344 , 0.46757548],
[0.65567791, 0.46273121],
[0.54916206, 0.23178091]])
b
Out[7]:
array([[0.1763344 , 0.65567791],
[0.54916206, 0.54916206],
[0.54916206, 0.23178091]])
As you can see there is some regularity in numbers. All the numbers from b
contains only numbers from a
.
Now I'm going to somehow test numbers when they are in Armadillo matrix (e.g. in C++)
from carma.
@RUrlus I just figured out that Armadillo matrix is fine. Here is the piece of C++ code:
EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) {
// convert to armadillo matrix without copying.
arma::Mat<double> mat = carma::arr_to_mat<double>(arr);
# we should see this output in Python
std::cout << mat << std::endl;
// convert to Numpy array and return
return carma::mat_to_arr(mat);
}
Now python prints armadillo matrix and we have exactly tha same numbers in Armadillo:
import numpy as np
import example
a = np.random.rand(3,2)
b = example.manual_example(a)
0.2922 0.0900 # this is not `b`, it is Armadillo output from C++
0.1283 0.7299
0.4878 0.4420
a
Out[6]:
array([[0.29223221, 0.09004788],
[0.12834728, 0.72990356],
[0.48776511, 0.4420319 ]])
This works with both dev/0.4.0
and 0.3.0
Thus I think that something is wrong with carma::mat_to_arr(mat);
from carma.
@RUrlus I just figured out that Armadillo matrix is fine. Here is the piece of C++ code:
EXAMPLE_EXPORT py::array_t<double> manual_example(py::array_t<double> & arr) { // convert to armadillo matrix without copying. arma::Mat<double> mat = carma::arr_to_mat<double>(arr); # we should see this output in Python std::cout << mat << std::endl; // convert to Numpy array and return return carma::mat_to_arr(mat); }
Now python prints armadillo matrix and we have exactly tha same numbers in Armadillo:
import numpy as np import example a = np.random.rand(3,2)
@KerimMatlab, something to consider although not the cause of the issue is that you are making a copy unless you set the compile time definition.
Numpy by default create row-major, also called C-contiguous, arrays whereas Armadillo expects column-major (Fortran contiguous) memory.
Could you run the tests on your machine? You can do so with:
- install pytest:
pip3 install pytest
orconda install -c anaconda pytest
- create build directory in the root of the repository and move into it
- run
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
make install
- run
pytest
in the build directory
from carma.
@RUrlus I'm very beginner at Python (I have not used pytests before) and not good in command line operations.
Please help me a little bit more.
- I successfully installed pytest via
conda install -c anaconda pytest
- I have problems now correct me if I'm wrong: I go to the
carma-0.3.0
directory, create there_build
dir, go there and writecmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
?
If so I got errors:
(MyEnv38) C:\apps\carma-0.3.0\_build>C:\Qt\Tools\CMake_64\bin\cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.18363.
-- The CXX compiler identification is MSVC 19.27.29111.0
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:101 (add_subdirectory):
The source directory
C:/apps/carma-0.3.0/third_party/pybind11
does not contain a CMakeLists.txt file.
CMake Error at tests/CMakeLists.txt:8 (pybind11_add_module):
Unknown CMake command "pybind11_add_module".
-- Configuring incomplete, errors occurred!
See also "C:/apps/carma-0.3.0/_build/CMakeFiles/CMakeOutput.log".
These errors most likely connected that I do not have path
var to python packages (by the way I'm trying to use Visual Studio 2017)
I prefer using CMakeGUI. I'm going to try it to build test.
from carma.
Sorry I have very little experience outside *NIX OS's when it comes to Python and C++, nor do I have acces to a Windows machine. The best I can do is add some additional tests to our Windows test on travis.
You can also have a look here to see the commands used to build and test on the windows virtual machine: https://travis-ci.com/github/RUrlus/carma/jobs/379189384#L239
from carma.
@KerimMatlab It looks like you don't have the submodules; I should have added:
git submodule update --init
Can you run that and try again?
from carma.
@RUrlus I'm in PyCharm and I just wrote:
(MyEnv38) C:\apps\carma-0.3.0>git init
Initialized empty Git repository in C:/apps/carma-0.3.0/.git/
(MyEnv38) C:\apps\carma-0.3.0>git submodule update --init
Then I write:
(MyEnv38) C:\apps\carma-0.3.0>cd C:\apps\carma-0.3.0\_build
(MyEnv38) C:\apps\carma-0.3.0\_build>C:\Qt\Tools\CMake_64\bin\cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
And I get the same error:
(MyEnv38) C:\apps\carma-0.3.0\_build>C:\Qt\Tools\CMake_64\bin\cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.18363.
CMake Error at CMakeLists.txt:101 (add_subdirectory):
The source directory
C:/apps/carma-0.3.0/third_party/pybind11
does not contain a CMakeLists.txt file.
CMake Error at tests/CMakeLists.txt:8 (pybind11_add_module):
Unknown CMake command "pybind11_add_module".
-- Configuring incomplete, errors occurred!
See also "C:/apps/carma-0.3.0/_build/CMakeFiles/CMakeOutput.log".
from carma.
OK, that's not correct. You shouldn't do the git init
.
It should be:
git clone https://github.com/RUrlus/carma.git
cd carma
git submodule update --init
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_TESTS=true -DCMAKE_GENERATOR_PLATFORM=x64 ..
cmake --build . --target INSTALL --config Debug
pytest
from carma.
@RUrlus worked like a magic 👍
On the las line: pytest
I got error (on the picture):
from carma.
Hmm, that's not good. Which branch is this? Arraystore doesn't yet work properly on the dev branch.
Anyway, could you run the following two commands instead of pytest
:
cd tests
pytest test_arr_to_mat.py test_mat_to_arr.py
from carma.
@RUrlus the version is not written in the folder but I think it is a master as I type git clone https://github.com/RUrlus/carma.git
?
from carma.
@RUrlus I go to the /build/tests
and run now:
pytest test_arr_to_mat.py test_mat_to_arr.py
The output:
from carma.
@RUrlus the version is not written in the folder but I think it is a master as I type git clone https://github.com/RUrlus/carma.git ?
Yeah, then it's master.
Strange that the tests pass for the conversions but your example is not working. So maybe Qt is playing a role here.
Although the whole ArrayStore failing is also weird. I'll have a look this weekend.
from carma.
@RUrlus Hi Ralph,
How do you think will this issue be solved? If you are busy that is fine :)
from carma.
@KerimMatlab Hi,
Sorry I've been very busy, new house & job at the same time.
I know the underlying cause of the bug but I don't have an easy solution. I have something in mind but it will take a while.
A work-around is to use Fortran contiguous arrays, np.asarray(arr, order='F')
and pass that to C++
from carma.
Resolved in b159447
from carma.
Related Issues (20)
- Undefined symbols during packaging HOT 4
- `cibuildwheel` manylinux build issue HOT 12
- `carma` changes install location based on installation status HOT 3
- Segmentation Fault on Carma Example code HOT 3
- Carma seem to not support fixed size matrices HOT 1
- LICENSE Question HOT 2
- Is there a way to use carma from source? HOT 5
- CARMA does not correctly borrow the array when used with pybind11 trampoline classes, resulting in referencing garbage collected memory HOT 7
- Citation(s)? HOT 1
- Definition of `ssize_t` in `headers\carma\include\carma(28)` causes CI issue on Windows HOT 4
- Automatic conversion doesn't work for function references with const& return type HOT 6
- Remove unused variables HOT 1
- Can't modify matrix as a side effect (but don't know if this is supposed to be possible) HOT 6
- Help with Armadillo memory bug in pybind11 HOT 13
- Potentially mismatched allocator-deallocator when linking to external shared library HOT 3
- Exception handling broken when linking to carma for Windows+Clang HOT 10
- Crash when assigning one matrix to another HOT 2
- is `cmake v3.16` necessary? HOT 7
- Repo does not pass flake8 HOT 2
- Possible to call `arr_to_mat<float>`? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from carma.