deeplearningais / curfil Goto Github PK

I used curfil earlier this year and I was able to complete the tasks I have.
However, I am trying to install curfil in other machine then I got some errors.
First thing is after running the CTest, I got this message.
I saw some changes but I don't know if this is normal or not.
Start 1: image_test
1/7 Test #1: image_test ....................... Passed 0.78 sec
Start 2: feature_generation_test
2/7 Test #2: feature_generation_test .........._Failed 0.88 sec
Start 3: random_tree_test
3/7 Test #3: random_tree_test ................. Passed 0.00 sec
Start 4: random_tree_image_gpu_test
4/7 Test #4: random_tree_image_gpu_test ......._Failed 0.52 sec
Start 5: image_cache_test
5/7 Test #5: image_cache_test ................._Failed 0.09 sec
Start 6: import_export_test
6/7 Test #6: import_export_test ..............._Failed 0.26 sec
Start 7: random_tree_image_test
7/7 Test #7: random_tree_image_test ...........***Failed 1.34 sec

29% tests passed, 5 tests failed out of 7

Total Test time (real) = 3.91 sec

The following tests FAILED:
2 - feature_generation_test (Failed)
4 - random_tree_image_gpu_test (Failed)
5 - image_cache_test (Failed)
6 - import_export_test (Failed)
7 - random_tree_image_test (Failed)
Errors while running CTest

I ignored this message and installed curfil to run the NYU example.
Then I got the other error.
Could anyone tell me how to fix them. Thanks.
2015-Sep-14 09:59:20.729077 INFO finished loading 795 images. size in memory: 3943.65 MB
2015-Sep-14 09:59:20.729416 INFO max samples per batch: 4661
2015-Sep-14 09:59:20.729439 INFO image cache size: 41 images (252.2 MB)
2015-Sep-14 09:59:20.729460 INFO trees: 3
2015-Sep-14 09:59:20.729474 INFO training trees in parallel: 0
2015-Sep-14 09:59:20.729495 INFO randomSeed: 4711
samplesPerImage: 4537
featureCount: 5729
minSampleCount: 204
maxDepth: 18
boxRadius: 111
regionSize: 3
thresholds: 20
maxImages: 0
imageCacheSize: 41
accelerationMode: gpu
maxSamplesPerBatch: 4661
subsamplingType: classUniform
useCIELab: 1
useDepthFilling: 0
deviceIds: [0]
ignoredColors: [0,0,0]
useDepthImages: 1
horizontalFlipping: 0
2015-Sep-14 09:59:20.729577 INFO learning image tree ensemble. 3 trees with 8 threads
2015-Sep-14 09:59:23.064907 INFO sampling 4 classes. 300510 samples per class with 8 threads from 795 images
2015-Sep-14 09:59:51.552866 INFO sampled 0 pixels of class '0' RGB(0,0,0)
2015-Sep-14 09:59:51.612679 INFO sampled 300510 pixels of class '1' RGB(181,137,0)
2015-Sep-14 09:59:51.669283 INFO sampled 300510 pixels of class '2' RGB(7,54,66)
2015-Sep-14 09:59:51.731785 INFO sampled 300510 pixels of class '3' RGB(147,161,161)
2015-Sep-14 09:59:51.793203 INFO sampled 300510 pixels of class '4' RGB(203,75,22)
2015-Sep-14 09:59:51.799354 INFO sampled 1202040 pixels for 4 classes (300510 samples/class) in 28.7345 s
2015-Sep-14 09:59:51.808080 INFO sorting 1202040 samples
2015-Sep-14 09:59:51.912847 INFO sorted in 104.69 ms
2015-Sep-14 10:00:17.263581 INFO GPU Device 0: GeForce GTX 960
2015-Sep-14 10:00:17.264210 INFO training level 1. nodes: 1
2015-Sep-14 10:00:17.272194 INFO randomFeatureGeneration: skipped 1135664 samples
terminate called after throwing an instance of 'std::runtime_error'
what(): cudaBindTextureToArray(colorTexture, colorTextureData): invalid texture reference
Aborted (core dumped)

when I trained, there are only 89M gpu used, and nothing printed after:
2019-Jun-03 13:17:30.415189 INFO learning image tree ensemble. 3 trees with 20 threads
Is it correct? 7 tests were passed.

Hi,

First of all, thank you for your great work ~~~

I just started using curfil. It seems that the program writes PNGs with the per-class probabilities(all zeros) of the prediction. I think we need to scale the probabilities from 01 to 0255
in file predict.cpp line 325:
probabilityImage.setColor(x, y, c, probability);
--->
probabilityImage.setColor(x, y, c, probability * 255);

The NO_DEFAULT_PATH entry in FindMDBQ.cmake causes a local installation (e.g. ~/local/) of MDBQ to not be found even in CMAKE_PREFIX_PATH is set. Is there a good reason this is there?

Hi, thank you for this great library!

Recently I've compiled curfil in CentOS 6.5, but in the system, there is only gcc 4.4.7 installed. The gcc has no support with c++0x, and it will report an error when compiling curfil.
Could you give me some advice on how to compile this library with gcc 4.4.7 in CentOS 6.5, please?
Thanks!

Hi, in the documentation, the link to Vigra seems to be obsolete. Seems that this is the new link: https://github.com/ukoethe/vigra

I am getting this error, kindly guide me.

795 training images
654 test images
using filled depth images
reading ./dataset/nyu_depth_v2_labeled.mat
processing images
Traceback (most recent call last):
File "../curfil/scripts/NYU/convert.py", line 180, in
Parallel(num_threads, 5)(delayed(convert_image)(i, scenes[i], depth[i, :, :].T, images[i, :, :].T, labels[i, :, :].T) for i in range(len(images)))
File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 514, in init
% (backend, sorted(BACKENDS.keys())))
ValueError: Invalid backend: 5, expected one of ['multiprocessing', 'sequential', 'threading']

I followed the instructions to install Vigra and Intel TBB, but this happens:
$cmake -DCMAKE_BUILD_TYPE=Release ..
-- Found Intel TBB
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find VIGRA (missing: VIGRA_INCLUDE_DIR)
Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
CMakeModules/FindVIGRA.cmake:43 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
src/CMakeLists.txt:30 (FIND_PACKAGE)

-- Configuring incomplete, errors occurred!

Can you please suggest a fix? Or specify in the readme what needs to change in the CMakeLists or other files?

hello @temporaer ,
i installed curfil correctly, and all test has passed when i use ctestcommand, here is the log:

    Start 1: image_test
1/7 Test #1: image_test .......................   Passed    1.11 sec
    Start 2: feature_generation_test
2/7 Test #2: feature_generation_test ..........   Passed    1.98 sec
    Start 3: random_tree_test
3/7 Test #3: random_tree_test .................   Passed    0.02 sec
    Start 4: random_tree_image_gpu_test
4/7 Test #4: random_tree_image_gpu_test .......   Passed    8.56 sec
    Start 5: image_cache_test
5/7 Test #5: image_cache_test .................   Passed    0.48 sec
    Start 6: import_export_test
6/7 Test #6: import_export_test ...............   Passed    1.47 sec
    Start 7: random_tree_image_test
7/7 Test #7: random_tree_image_test ...........   Passed   37.40 sec

100% tests passed, 0 tests failed out of 7

but when i use curfil_train to train the nyu_depth_v2 dataset(has been converted to curfil format), it outputs one run time error,like this:

terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaMalloc3DArray(&colorTextureData, &channelDesc, extent, flags): invalid argument

i just debug the code, and find it will break(just the same error infor) in all 3D-aware function with cuda, like cudaMalloc3DArray,cudaMemcpy3DAsync..., so how to fix this?thanks.
my gpu is K80, cuda 7.0, and i used device 0 for training. i also set SET( CUDA_ARCHITECTURE -gencode arch=compute_37,code=sm_37 ) in src/CMakeLists.txt.

Hi,

I'm having some trouble trying to compile your project under Ubuntu 14.04 following the steps you provide in the documentation.

First, when I execute the cmake command, I receive the following warning:

"CMake Warning (dev) in src/ndarray/src/cuv/CMakeLists.txt:
Policy CMP0022 is not set: INTERFACE_LINK_LIBRARIES defines the link
interface. Run "cmake --help-policy CMP0022" for policy details. Use the
cmake_policy command to set the policy and suppress this warning.

Target "ndarray" has an INTERFACE_LINK_LIBRARIES property. This should be
preferred as the source of the link interface for this library but because
CMP0022 is not set CMake is ignoring the property and using the link
implementation as the link interface instead.

INTERFACE_LINK_LIBRARIES:

/usr/local/cuda-7.0/lib64/libcudart.so;ndarray

Link implementation:

/usr/local/cuda-7.0/lib64/libcudart.so

"

Then, when I try to execute the make command, a error in the linking with the ndarray library appears:

"Linking CXX shared library libndarray.so
Error running link command: No such file or directory
make[2]: *** [src/ndarray/src/cuv/libndarray.so] Error 2
make[1]: *** [src/ndarray/src/cuv/CMakeFiles/ndarray.dir/all] Error 2
make: *** [all] Error 2"

I have the same problems in a different machine with Linux Mint 17 (Linux distribution based in Ubuntu 14.04). Both distributions are running gcc 4.8.

Thanks in advance.

Agustín García.

I seem to be having memory issues with curfil_train. I have a dataset of ~6000 images, and when I run curfil_train, it tries to load all of them, gets somewhere in the 1750-1850 range, then my computer locks up and the process eventually gets killed.

By a back of the envelope calculation, this should be approximately the number of images required to fill up the memory on my computer. Is curfil_train actually trying to load all images simultaneously into memory? If so, this would limit the number of training images to a number that is quite small by modern standards.

I'm probably doing something wrong but I can't seem to get this library to compile. It seems to be an error with ndarray.

When I run CMake I get the following warning:

CMake Warning (dev) at /usr/share/cmake-3.0/Modules/FindCUDA.cmake:1493 (add_library):
Policy CMP0038 is not set: Targets may not link directly to themselves.
Run "cmake --help-policy CMP0038" for policy details.  Use the cmake_policy
command to set the policy and suppress this warning.

Target "ndarray" links to itself.
Call Stack (most recent call first):
src/ndarray/src/cuv/CMakeLists.txt:26 (CUDA_ADD_LIBRARY)
This warning is for project developers.  Use -Wno-dev to suppress it.

When I compile I get the following error:

[  4%] [  8%] [ 12%] Building NVCC (Device) object     src/ndarray/src/cuv/CMakeFiles/ndarray.dir//./ndarray_generated_reference.cu.o
Building NVCC (Device) object src/ndarray/src/cuv/CMakeFiles/ndarray.dir//./ndarray_generated_allocators.cu.o
Building NVCC (Device) object src/ndarray/src/cuv/CMakeFiles/ndarray.dir//./ndarray_generated_memory.cu.o
Linking CXX shared library libndarray.so
Error running link command: No such file or directory
src/ndarray/src/cuv/CMakeFiles/ndarray.dir/build.make:2445: recipe for target     'src/ndarray/src/cuv/libndarray.so' failed
make[2]: *** [src/ndarray/src/cuv/libndarray.so] Error 2
CMakeFiles/Makefile2:285: recipe for target 'src/ndarray/src/cuv/CMakeFiles/ndarray.dir/all' failed
make[1]: *** [src/ndarray/src/cuv/CMakeFiles/ndarray.dir/all] Error 2
Makefile:127: recipe for target 'all' failed  
make: *** [all] Error 2

Just for reference, I ran cmake --help-policy CMP0038 as recommended in the cmake warning and it says:

CMP0038

Targets may not link directly to themselves.

CMake 2.8.12 and lower allowed a build target to link to itself directly with
a target_link_libraries() call. This is an indicator of a bug in
user code.

The OLD behavior for this policy is to ignore targets which list themselves
in their own link implementation. The NEW behavior for this policy is to
report an error if a target attempts to link to itself.

This policy was introduced in CMake version 3.0. CMake version
3.0.2 warns when the policy is not set and uses OLD behavior. Use
the cmake_policy command to set it to OLD or NEW explicitly.

Any idea what I'm doing wrong?

Hello, I'm new using curfil to train and predict on NYU depth v2 dataset. I practice like the wiki said. And I print out the probabilityImage and find that label_1.png and label_2.png are all zeros, label_3.png has little nonzeros , only label_4.png is normal. When I don't use the option "writeProbabilityImages", the per-class accuracy shows ok.
cl 0 cl 1 cl 2 cl 3 cl 4
class 0 RGB(0,0,0 ) : 0.000 0.000 0.000 0.000 0.000
class 1 RGB(181,137,0 ) : 0.000 0.191 0.447 0.285 0.077
class 2 RGB(203,75,22 ) : 0.000 0.079 0.717 0.177 0.028
class 3 RGB(147,161,161) : 0.000 0.026 0.118 0.853 0.003
class 4 RGB(7,54,66 ) : 0.000 0.012 0.072 0.010 0.906
Am I doing wrong in some places?

Hello,
I understand this project has probably only been tested on Linux, but all the dependent libraries are also available on Windows and therefore I created a fork for this task. My environment is Visual Studio 2015 Win64 on Windows 10. I apply it to CMake commands and create VC projects. The main modification in code is to change path strings into type boost::filesystem::path::string_type for the sake that Boost uses self-defined string_type (equivalent to wide chars), instead of string, for path variables. My modified version can be successfully compiled, and I can generate curfil.lib. However, I cannot resolve the numerous linking errors while building curfil_train and curfil_predict. In general, there are only two types LNK2001 and LNK2019:

Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2001	unresolved external symbol "public: virtual void __cdecl cuv::default_allocator::alloc(void * *,unsigned __int64,unsigned __int64,struct cuv::host_memory_space)" (?alloc@default_allocator@cuv@@UEAAXPEAPEAX_K1Uhost_memory_space@2@@Z)	curfil_train	C:\Users\i-fen\Documents\curfil-win64\build\src\curfil\curfil.lib(image.obj)	1	

Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2019	unresolved external symbol "void __cdecl cuv::detail::entry_set<int>(int *,unsigned __int64,int,struct cuv::host_memory_space)" (??$entry_set@H@detail@cuv@@YAXPEAH_KHUhost_memory_space@1@@Z) referenced in function "public: void __cdecl cuv::reference<int,struct cuv::host_memory_space>::operator=<unsigned int>(unsigned int const &)" (??$?4I@?$reference@HUhost_memory_space@cuv@@@cuv@@QEAAXAEBI@Z)	curfil_train	C:\Users\i-fen\Documents\curfil-win64\build\src\curfil\curfil.lib(image.obj)	1	

I don't think it is an OS-related issue, rather an incorrect setup for the linker. Here is the list of input for the linker:

kernel32.lib
user32.lib
gdi32.lib
winspool.lib
shell32.lib
ole32.lib
oleaut32.lib
uuid.lib
comdlg32.lib
advapi32.lib
Debug\curfil.lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64\cudart_static.lib
C:\Program Files\ndarray\lib\static\ndarray.lib
C:\vigra\lib\vigraimpex.lib
C:\Program Files\Intel\TBB\lib\intel64\vc14\tbb.lib
C:\Program Files\Intel\TBB\lib\intel64\vc14\tbbmalloc.lib
C:\Program Files\Intel\TBB\lib\intel64\vc14\tbb_debug.lib
C:\Boost\lib\vc14\x64\libboost_system-vc140-mt-gd-1_64.lib
C:\Boost\lib\vc14\x64\libboost_filesystem-vc140-mt-gd-1_64.lib
C:\Boost\lib\vc14\x64\libboost_iostreams-vc140-mt-gd-1_64.lib
C:\Boost\lib\vc14\x64\libboost_program_options-vc140-mt-gd-1_64.lib
C:\Boost\lib\vc14\x64\libboost_date_time-vc140-mt-gd-1_64.lib
C:\Boost\lib\vc14\x64\libboost_regex-vc140-mt-gd-1_64.lib
pthreadVC2.lib 

I have been stuck for more than a week. I'd heartily appreciate it f anyone has an idea about this or has more knowledge about linking to share.

Cheers,
ifeng

I am trying to run the convert.py file and I get the following error
Traceback (most recent call last):
File "scripts/NYU/convert.py", line 180, in
Parallel(num_threads)(delayed(convert_image)(i, scenes[i], depth[i, :, :].T, images[i, :, :].T, labels[i, :, :].T) for i in range(len(images)))
File "/home/jorgegarciaserna/project/lib/python3.6/site-packages/joblib/parallel.py", line 934, in call
self.retrieve()
File "/home/jorgegarciaserna/project/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/home/jorgegarciaserna/project/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
struct.error: required argument is not an integer

Please guide me for this error,

CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
Could NOT find VIGRA (missing: VIGRA_INCLUDE_DIR)
Call Stack (most recent call first):
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)
CMakeModules/FindVIGRA.cmake:43 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
src/CMakeLists.txt:30 (FIND_PACKAGE)

Hello, I tried to compile this library on my Mac but failed. Would you please tell me the compiler and OS you use to compile these code?

My compiler showed "random_tree.h:157:9: error: 'isnan' was not declared in this scope." I searched over Internet[1] and find out that < cmath > should be paired with std::isnan() rather than isnan(). So I replaced isnan() with std::isnan() but got another error "random_tree.h(157): error: expected an identifier." Simliar errors occurred if I use < math.h > with isnan() instead.

Do you have any suggestions? Thanks.

[1] http://stackoverflow.com/questions/18128899/