Support to make NVIDIA's Caffe fork (for NVIDIA DIGITS) work in Windows about caffe-builder HOT 27 CLOSED

@lunzueta I am always happy to help when I can. Here is what I can suggest from the error messages above:

• First, be careful with caffe headers if you modify them. There is a flaw in the caffe-builder project because it installs every library in the same folder. Because of this the caffe project includes the installation location in its include directories and will sometimes be confused between the installed caffe and the one you are modifying. When making changes delete the caffe folder in install/include.

11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): error C3855: 'std::tuple': template parameter '_Types' is incompatible with the declaration [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): error C2977: 'std::tuple' : too many template arguments [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11> C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\utility(75) : see declaration of 'std::tuple'

I added a preprocessor definition in the root CMakeLists.txt for this. Make the same change.

11>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(6): fatal error C1083: Cannot open include file: 'sys/ioctl.h': No such file or directory [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11> db.cpp

This one I don't remember ever seeing this.

11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\db.cpp(27): error C3861: 'mkdir': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11> math_functions.cpp

You need a define:

#define mkdir _mkdir

11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(346): error C3861: '__builtin_popcount': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(357): error C3861: '__builtin_popcountl': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]

Same here. You need a define for these two. See msvc.hpp.

HTH

from caffe-builder.

Thanks for the hints @willyd. After applying your suggestions, the only compilation error I get now (apart from some linking errors that come in the end which I guess I still can't solve until I fix this), is that related to 'sys/ioctl.h'. I guess you didn't see this before because it seems to be something specific to NVIDIA's fork (in file caffe\src\caffe\parallel.cpp). Any idea about what to do with this? I'll search about what to do for porting this to Windows with the minimal possible changes, but this would be my first time with this file, so any suggestion from somebody more experienced is welcome.

from caffe-builder.

@lunzueta I am not so experienced in porting code from Linux to Windows myself either. So your guess is as good as mine. Google is your friend.

from caffe-builder.

I got a bit stuck with this now. In the parallel.cpp file there are inclusions of two header files which are specific to Linux: sys/ioctl.h and sys/mman.h. If you just skip them and compile the parallel.cpp file (to see which functions rely on those inclusions) you don't get any error but just the following warnings:

1>  parallel.cpp
1>(MY_PATH)\install\include\H5public.h(158): warning C4091: 'typedef ' : ignored on left of '__int64' when no variable is declared
1>(MY_PATH)\DownloadCache\caffe\include\caffe/loss_layers.hpp(16): warning C4305: 'initializing' : truncation from 'double' to 'const float'
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(156): warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(181): warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(98): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(84) : while compiling class template member function 'caffe::GPUParams<float>::GPUParams(boost::shared_ptr<caffe::Solver<Dtype>>,int)'
1>          with
1>          [
1>              Dtype=float
1>          ]
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(512) : see reference to class template instantiation 'caffe::GPUParams<float>' being compiled
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(318): warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(296) : while compiling class template member function 'void caffe::P2PSync<float>::on_start(caffe::Timer *,std::ostringstream *)'
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(513) : see reference to class template instantiation 'caffe::P2PSync<float>' being compiled
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(372): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(342) : while compiling class template member function 'void caffe::P2PSync<float>::on_gradients_ready(caffe::Timer *,std::ostringstream *)'
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(398): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(290): warning C4244: 'argument' : conversion from 'google::protobuf::int64' to 'const unsigned int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(280) : while compiling class template member function 'void caffe::P2PSync<float>::InternalThreadEntry(void)'
1>C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xutility(2132): warning C4996: 'std::_Copy_impl': Function call with parameters that may be unsafe - this call relies on the caller to check that the passed values are correct. To disable this warning, use -D_SCL_SECURE_NO_WARNINGS. See documentation on how to use Visual C++ 'Checked Iterators'
1>          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xutility(2113) : see declaration of 'std::_Copy_impl'
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1293) : see reference to function template instantiation '_OutIt std::copy<const Element*,Element*>(_InIt,_InIt,_OutIt)' being compiled
1>          with
1>          [
1>              _OutIt=google::protobuf::int64 *
1>  ,            Element=google::protobuf::int64
1>  ,            _InIt=const google::protobuf::int64 *
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1292) : while compiling class template member function 'void google::protobuf::internal::ElementCopier<Element,false>::operator ()(Element *,const Element *,int)'
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1285) : see reference to function template instantiation 'void google::protobuf::internal::ElementCopier<Element,false>::operator ()(Element *,const Element *,int)' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1285) : see reference to class template instantiation 'google::protobuf::internal::ElementCopier<Element,false>' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1284) : while compiling class template member function 'void google::protobuf::RepeatedField<google::protobuf::int64>::CopyArray(Element *,const Element *,int)'
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1279) : see reference to function template instantiation 'void google::protobuf::RepeatedField<google::protobuf::int64>::CopyArray(Element *,const Element *,int)' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1278) : while compiling class template member function 'void google::protobuf::RepeatedField<google::protobuf::int64>::MoveArray(Element *,Element *,int)'
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1254) : see reference to function template instantiation 'void google::protobuf::RepeatedField<google::protobuf::int64>::MoveArray(Element *,Element *,int)' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1223) : while compiling class template member function 'void google::protobuf::RepeatedField<google::protobuf::int64>::Reserve(int)'
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1065) : see reference to function template instantiation 'void google::protobuf::RepeatedField<google::protobuf::int64>::Reserve(int)' being compiled
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1010) : while compiling class template member function 'int google::protobuf::RepeatedField<google::protobuf::int64>::size(void) const'
1>          (MY_PATH)\Caffe-prefix\src\Caffe-build\include\caffe/proto/caffe.pb.h(9037) : see reference to function template instantiation 'int google::protobuf::RepeatedField<google::protobuf::int64>::size(void) const' being compiled
1>          (MY_PATH)\Caffe-prefix\src\Caffe-build\include\caffe/proto/caffe.pb.h(686) : see reference to class template instantiation 'google::protobuf::RepeatedField<google::protobuf::int64>' being compiled

From this info, I don't see clearly what to do next, but I guess something has to be done instead of just skipping those inclusions. Don't know...

Anyway, from this point on, if I compile the whole caffe.vcxproj project, I don't get any error either. The caffe-nv.lib file (caffe.lib in your Windows port, @willyd) is generated in Caffe-prefix\src\Caffe-build\lib\Release folder (caffe-nv-d.lib in Caffe-prefix\src\Caffe-build\lib\Debug folder), but those files aren't automatically copied to the install/lib folder, as it happens with caffe.lib and caffe-d.lib in your port. What should be changed to make the builder copy automatically those files to install/lib? Something related to CMake files?

from caffe-builder.

You need to build the INSTALL project. Equivalent of make install on Linux.

from caffe-builder.

Thanks @willyd, that worked for the copy. Anyway, I see that more changes have to be done to make this work. I've done another fork of NVIDIA's fork to have a version control of the changes, and to be able to share this with others: https://github.com/lunzueta/caffe

Anyone interested in collaborating on this, let me know, so that I can give access.

from caffe-builder.

@willyd, I have one question about how to adapt your super builder to make it work with my fork. In the file Caffe.cmake I did the following change:

Instead of this:

buildem_download_package(GIT_REPOSITORY "https://github.com/willyd/caffe.git"
                         GIT_TAG visualstudio
                         SOURCE_DIR Caffe_SOURCE_DIR)

I put this:

buildem_download_package(GIT_REPOSITORY "https://github.com/lunzueta/caffe.git"
                         SOURCE_DIR Caffe_SOURCE_DIR)

I'm not sure what GIT_TAG is for, because I still didn't find a good explanation about it on the Internet. In my repository there's not visualstudio branch, so I just removed that. Would that be ok?

from caffe-builder.

@lunzueta GIT_TAG stands for anything you would do:

git checkout ${GIT_TAG}

so a branch, a commit sha-1 or a tag. You can also remove GIT_TAG altogether so that the default branch (usually master) is checked out.

from caffe-builder.

Ok, thanks. Understood. In my case, for now, it's simpler just to remove it.

from caffe-builder.

At this moment, I'm able to compile NVCaffe from my fork (https://github.com/lunzueta/caffe) in Release mode, except the projects "runtest" and "test.testbin", but I guess I can start testing DIGITS with the compiled executables and libraries. I'll report about these tests when possible.

Anyway, I'd like to be able to compile all projects in Release and also in Debug to finish properly this port. The errors I get are the following:

runtest and test.testbin projects (Release):

2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2784: '_Ty std::max(std::initializer_list<_Elem>,_Pr)' : could not deduce template argument for 'std::initializer_list<_Elem>' from 'const float'
2>          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\algorithm(4108) : see declaration of 'std::max'
2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2780: 'const _Ty &std::max(const _Ty &,const _Ty &,_Pr)' : expects 3 arguments - 2 provided

pycaffe (Debug):

2>LINK : fatal error LNK1104: cannot open file 'python27_d.lib'

The linking error of pycaffe in Debug is strange because python27_d.lib doesn't appear in the project linking properties as a dependence, but python27.lib. In @willyd's equivalent project this project doesn't have this issue in Debug.

Any suggestions?

from caffe-builder.

2>LINK : fatal error LNK1104: cannot open file 'python27_d.lib'

You need to #include boost/python/detail/wrap_python.hpp instead of #include <python.h>. See http://www.boost.org/doc/libs/1_58_0/libs/python/doc/building.html#python-debugging-builds for details.

2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2784: '_Ty std::max(std::initializer_list<_Elem>,_Pr)' : could not deduce template argument for 'std::initializer_list<_Elem>' from 'const float' 2> C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\algorithm(4108) : see declaration of 'std::max' 2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2780: 'const _Ty &std::max(const _Ty &,const _Ty &,_Pr)' : expects 3 arguments - 2 provided

std::max is a template function. It seems MSVC cannot deduce the type it should use. You have to specify it manually like so std::max<Type>(arg1, arg2).

from caffe-builder.

Thanks @willyd, the change about python worked. Now pycaffe compiles correctly in Debug too.

Regarding the template issue, I'm not sure how to apply what you tell me. The only differences I see in test_gradient_check_util.hpp in Nvidia's fork with respect to yours are the following two:

1. In your file:

// First, figure out what blobs we need to check against.
  vector<Blob<Dtype>*> blobs_to_check;
  vector<bool> propagate_down(bottom.size(), check_bottom < 0);
  for (int i = 0; i < layer->blobs().size(); ++i) {
    blobs_to_check.push_back(layer->blobs()[i].get());
  }

In Nvidia's fork:

  // First, figure out what blobs we need to check against, and zero init
  // parameter blobs.
  vector<Blob<Dtype>*> blobs_to_check;
  vector<bool> propagate_down(bottom.size(), check_bottom < 0);
  for (int i = 0; i < layer->blobs().size(); ++i) {
    Blob<Dtype>* blob = layer->blobs()[i].get();
    caffe_set(blob->count(), static_cast<Dtype>(0), blob->mutable_cpu_diff());
    blobs_to_check.push_back(blob);
  }

2. In your file:

  Dtype scale = std::max(
    std::max(fabs(computed_gradient), fabs(estimated_gradient)), 1.);

In Nvidia's fork:

  Dtype scale = std::max<Dtype>(
    std::max<Dtype>(fabs(computed_gradient),
    fabs(estimated_gradient)), 1.);

The rest of the file is exactly the same in both sides. What do you think would be the best to solve this?

from caffe-builder.

@lunzueta

In Nvidia's fork:

Dtype scale = std::max(
std::max(fabs(computed_gradient),
fabs(estimated_gradient)), 1.);

These are my changes not Nvidia's. Weirdly enough github's markdown removes the <Dtype> in the quotation :(.

Something is confusing the compiler type deduction. There are multiple overloads of std::max and this is why you have a cryptic error with an std::initializer_list. My guess is that since Dtype can be of type float or double and that 1. is double by default the compiler cannot deduce if it should use std::max<float> or std::max<double>. To get around this you have to use my version which forces the usage of std::max<Dtype> or try this code instead:

 Dtype scale = std::max(
    std::max(fabs(computed_gradient),
    fabs(estimated_gradient)), Dtype(1.));

from caffe-builder.

Thanks @willyd. This last change worked. Now I can compile all the NVCaffe projects in Release and Debug in VS2013 and x64. I guess this is a good moment to start testing Digits with this and see if everything is ok :)

from caffe-builder.

@willyd, while testing the generated NVCaffe executable for training the MNIST data, I've seen that I get this failure, which I don't get with your fork (your fork does this correctly):

F0913 13:37:23.739025  7344 layer_factory.hpp:79] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Data (known types: )
*** Check failure stack trace: ***

As you can see it doesn't find the known layer types properly. It's not straightforward to figure out where this behavior can come from. While you were doing your Windows port, did you find this kind of behavior at some point (and fixed it)?

from caffe-builder.

@lunzueta You need to integrate all my changes into your fork if you want the build to work. Because of the way VS works I need to use dumpbin to parse the caffe.lib and generate a header file to force the inclusion of some symbols the linker would get rid of otherwise. Do you have any header called caffe/forcelink.hpp in your build tree?

from caffe-builder.

@willyd It's a bit messy to apply your changes in Nvidia's fork, because of the differences between the original Caffe and Nvidia's fork. Initially, I tried to do this through Git (trying to merge Nvidia's fork with yours somehow), but in the end I think that it's better to do this manually, file by file (using Notepad++ and the compare plugin, any alternative ideas are welcome). Probably I missed something on the way, and I'm checking it again (manually). I didn't know this dumpbin that you mention. Maybe I should use it too or adapt something? How should that be? I don't have any header called caffe/forcelink.hpp but I can't see it either in your fork (only one comment inside src\caffe\CMakeLists.txt). Should I have that somewhere?

from caffe-builder.

@lunzueta I use the dumpbin utility to generate a list of #pragma's to force the inclusion of symbols that would be optimized away by the linker if I didn't. This list of #pragma's is generated inside a file called forcelink.hpp. CMake generates this file in this CMakeLists.txt:

# ---[ Visual Studio specifics
#  configure headers for dynamic or static linking on windows
unset(CAFFE_DLLEXPORT)
unset(CAFFE_DLLIMPORT)
unset(CAFFE_INCLUDE_FORCELINK)
if(MSVC)
    if(BUILD_SHARED_LIBS)
        set(CAFFE_DLLEXPORT __declspec(dllexport))
        set(CAFFE_DLLIMPORT __declspec(dllimport))
    else()
        get_filename_component(forcelink_header_name ${forcelink_header} NAME)
        set(CAFFE_INCLUDE_FORCELINK "#      include \"caffe/${forcelink_header_name}\"")
        # add a post build command to generate a header file that will
        # force the linker to include the statically registered layers
        # in a final executable.
        include(${PROJECT_SOURCE_DIR}/cmake/GenerateForceLinkSymbolsHeader.cmake)    
        add_generate_force_link_symbols_header_post_build_command(caffe ${forcelink_header} caffe_force_link)        
        # install the generated file
        install(FILES ${forcelink_header} DESTINATION include/caffe)
    endif()        
endif()

You could also forego this process altogether and use the Visual Studio feature called "Use Library Dependency Inputs" if you prefer this. But this conflicts with the CMake build system that will regenerate the projects overwriting any change you make to them via the IDE. I think this is what @happynear uses in his fork, but he doesn't use CMake.

Should I have that somewhere?

As the file is generated by CMake it should be in the build tree and not in the source tree. It should be in $Caffe_ROOT/caffe where $Caffe_ROOT is the folder containing the caffe.sln.

from caffe-builder.

@willyd Many thanks for the explanation. I also like more the CMake way. Now I see that I have an important difference in that forcelink.hpp file. In your version I see that there are many lines like the following (I only include a few):

#pragma comment(linker, "/include:?g_caffe_force_link_forward_gpu_ThresholdLayer@caffe@@3HA")  // NOLINT
#pragma comment(linker, "/include:?g_caffe_force_link_forward_gpu_TanHLayer@caffe@@3HA")  // NOLINT
#pragma comment(linker, "/include:?g_caffe_force_link_backward_gpu_TanHLayer@caffe@@3HA")  // NOLINT
#pragma comment(linker, "/include:?g_caffe_force_link_forward_gpu_SplitLayer@caffe@@3HA")  // NOLINT
#pragma comment(linker, "/include:?g_caffe_force_link_backward_gpu_SplitLayer@caffe@@3HA")  // NOLINT
#pragma comment(linker, "/include:?g_caffe_force_link_backward_gpu_SoftmaxWithLossLayer@caffe@@3HA")  // NOLINT
#pragma comment(linker, "/include:?g_caffe_force_link_forward_gpu_SoftmaxWithLossLayer@caffe@@3HA")  // NOLINT

And in my forcelink.hpp I don't have any of these. I'll review what I missed, but do you have an explanation of what happened?

from caffe-builder.

@willyd @lukeyeager Finally, I've been able to run Nvidia Digits on Windows 10 :-)

I could do this directly by using the current Windows version of Caffe (the recently created 'windows' branch on BVLC's Caffe repo, which is based on Microsoft's Caffe's fork), the latest version of Nvidia Digits pulled from Github and that's it! I used PyCharm to setup the dependencies and so on. Apart from this I also had to install Graphviz for Windows and set its bin folder in the 'path' environment variable, so that the nets could be visualized in Digits. Then, I was able to generate a dataset, train correctly a classification model (I tried with LeNet), and then test some images correctly too. More tests could be done, but I guess we could say this compatibility issue, in principle, is solved.

from caffe-builder.

@lunzueta DIGITS can be run in windows long ago since this pull request NVIDIA/DIGITS#199 .

from caffe-builder.

@happynear I thought that having NVIDIA's Caffe fork was a requirement, but I've seen that it isn't. Using the latest Windows version of Caffe (already integrated in BVLC's Caffe as a branch, based on Microsoft's Caffe fork), it's more simple to compile and use it, than before. Hopefully, since this Windows version of Caffe appeared it will be easier to get an official Windows version of Digits.

from caffe-builder.

@lunzueta, there is no difference between willyd's or my caffe-windows repo and the official one. I remember that I just copied the caffe.exe to a specified folder, then everything worked well.

from caffe-builder.

@happynear Ok, I see. Looking at comments in some threads here and there, for me (and for more people as I could see in those comments) it wasn't clear enough what should be done to make Nvidia Digits work on Windows, especially because of the use of Nvidia's Caffe fork, instead of BVLC's. Here (NVIDIA/DIGITS#47) I was requested to prepare a brief guide of what I did to make it work on Windows. I wrote a brief guide, but still some doubts arose. Maybe you can also comment there to clarify the doubts. Thanks for your help.

from caffe-builder.

@lunzueta , I have answered the question now. It seems that you haven't came across the folder and version issue?

from caffe-builder.

@happynear I've just tested to install this in another PC, to see if I forgot to include something in the guide explained in the other thread, but I don't think I missed anything apart from what I already explained there. I have it working in both PCs. I don't know what you mean with the folder and version issue. Let's continue the discussion in the other thread :-P

from caffe-builder.

This discussion was very helpful,
thank you very much

from caffe-builder.