facebookresearch / synsin Goto Github PK
View Code? Open in Web Editor NEWView synthesis for the public.
Home Page: http://www.robots.ox.ac.uk/~ow/synsin.html
License: Other
View synthesis for the public.
Home Page: http://www.robots.ox.ac.uk/~ow/synsin.html
License: Other
Dear author,
I am trying to run your demo program , but it seems to lack some key files of the program.
Pytorch3d
Thanks
Could you please give me the code for the network to generate a short video with a perspective movement (such as a circular motion on the x,y axis)? It is similar to the generating program of several animations on the homepage.
Hi,
I understand, from the paper and supplemental, that the ResNet block used in this work has been borrowed from that of BigGAN paper. The same has also been clearly illustrated in Figure 8(a)-(c) in the supplemental(image attached below).
However, the ResNet block definition, as done in blocks.py of model.layers (screenshot of code segment attached), does not seem to match the sequence of operations illustrated in the diagrams.
While in block.py, the order, for channel a, is normalization1-> activation1 -> conv1 -> normalization2 -> activation2 -> conv2 -> down/up/identity, in BigGAN as well in Figure 8 of your supplemental, down/up/identity comes before conv1 and after activation1 (not at the end, as has been coded). This is true of channel b as well.
Would it be possible to clarify which of the two is right? If the code is wrong, it needs to be rectified right away, given many have shown interest in your work :)
when I install the habitat-sim(d383c2011bf1baab2ce7b3cd40aea573ad2ddf71)
finally show that "Running setup.py develop for habitat-sim"
at last show the error that"
ERROR: Command errored out with exit status 1:
command: /home/pl/anaconda3/envs/lookout/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/pl/habitat-sim/setup.py'"'"'; file='"'"'/home/pl/habitat-sim/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/pl/habitat-sim/
Complete output (305 lines):
running develop
running egg_info
writing habitat_sim.egg-info/PKG-INFO
writing dependency_links to habitat_sim.egg-info/dependency_links.txt
writing requirements to habitat_sim.egg-info/requires.txt
writing top-level names to habitat_sim.egg-info/top_level.txt
reading manifest file 'habitat_sim.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'habitat_sim.egg-info/SOURCES.txt'
running build_ext
Cloning into '/home/pl/habitat-sim/src/deps/assimp'...
Cloning into '/home/pl/habitat-sim/src/deps/corrade'...
Cloning into '/home/pl/habitat-sim/src/deps/eigen-git-mirror'...
Cloning into '/home/pl/habitat-sim/src/deps/glfw'...
Cloning into '/home/pl/habitat-sim/src/deps/glog'...
Cloning into '/home/pl/habitat-sim/src/deps/googletest'...
Cloning into '/home/pl/habitat-sim/src/deps/magnum'...
Cloning into '/home/pl/habitat-sim/src/deps/magnum-integration'...
Cloning into '/home/pl/habitat-sim/src/deps/magnum-plugins'...
Cloning into '/home/pl/habitat-sim/src/deps/pybind11'...
Cloning into '/home/pl/habitat-sim/src/deps/rapidjson'...
Cloning into '/home/pl/habitat-sim/src/deps/recastnavigation'...
Cloning into '/home/pl/habitat-sim/src/deps/tinyobjloader'...
Cloning into '/home/pl/habitat-sim/src/deps/tinyply'...
Submodule path 'src/deps/Sophus': checked out 'b475c0a81a197ff8ea6bdcf8df8a2f5a7b1da879'
Submodule path 'src/deps/assimp': checked out '3fbe9095d157ecacf6a9549c9a21bf2ad3110ac6'
Submodule path 'src/deps/corrade': checked out 'ab38ab3d66a76aa8c0d85a9146f210b96645f602'
Submodule 'toolchains' (https://github.com/mosra/toolchains.git) registered for path 'src/deps/corrade/toolchains'
Cloning into '/home/pl/habitat-sim/src/deps/corrade/toolchains'...
Submodule path 'src/deps/corrade/toolchains': checked out '0fb6b9caac486cdb7736af7c10bfc35e34773f36'
Submodule path 'src/deps/eigen-git-mirror': checked out 'd10e1f03d4ef456f78ff6b91d9078b1f13a5975c'
Submodule path 'src/deps/glfw': checked out '53c8c72c676ca97c10aedfe3d0eb4271c5b23dba'
Submodule path 'src/deps/glog': checked out '8d7a107d68c127f3f494bb7807b796c8c5a97a82'
Submodule path 'src/deps/googletest': checked out '28b71e444c41ad93225145e029db4957f15aaae6'
Submodule path 'src/deps/magnum': checked out '9ec10b89d08bf8df6070aa6e814fad0d8fb5219c'
Submodule 'toolchains' (https://github.com/mosra/toolchains.git) registered for path 'src/deps/magnum/toolchains'
Cloning into '/home/pl/habitat-sim/src/deps/magnum/toolchains'...
Submodule path 'src/deps/magnum/toolchains': checked out 'a3e0aff97f7be9b3b69cbc3f981f6765fd0a467a'
Submodule path 'src/deps/magnum-integration': checked out '186396a88086771668c73befe629aedd68e9bbba'
Submodule 'toolchains' (https://github.com/mosra/toolchains.git) registered for path 'src/deps/magnum-integration/toolchains'
Cloning into '/home/pl/habitat-sim/src/deps/magnum-integration/toolchains'...
Submodule path 'src/deps/magnum-integration/toolchains': checked out 'a3e0aff97f7be9b3b69cbc3f981f6765fd0a467a'
Submodule path 'src/deps/magnum-plugins': checked out '335c79204a3acae1ccc3d0026fb7d4bca8bf200d'
Submodule 'toolchains' (https://github.com/mosra/toolchains.git) registered for path 'src/deps/magnum-plugins/toolchains'
Cloning into '/home/pl/habitat-sim/src/deps/magnum-plugins/toolchains'...
Submodule path 'src/deps/magnum-plugins/toolchains': checked out 'a3e0aff97f7be9b3b69cbc3f981f6765fd0a467a'
Submodule path 'src/deps/pybind11': checked out '177713fa4ea49c5594b5568279757f22354fa000'
Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'src/deps/pybind11/tools/clang'
Cloning into '/home/pl/habitat-sim/src/deps/pybind11/tools/clang'...
Submodule path 'src/deps/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
Submodule path 'src/deps/rapidjson': checked out '73063f5002612c6bf64fe24f851cd5cc0d83eef9'
Submodule 'thirdparty/gtest' (https://github.com/google/googletest.git) registered for path 'src/deps/rapidjson/thirdparty/gtest'
Cloning into '/home/pl/habitat-sim/src/deps/rapidjson/thirdparty/gtest'...
Submodule path 'src/deps/rapidjson/thirdparty/gtest': checked out '0a439623f75c029912728d80cb7f1b8b48739ca4'
Submodule path 'src/deps/recastnavigation': checked out '5c7ef6fea3b43abda5b0daa982be265d23d0664c'
Submodule path 'src/deps/tinyobjloader': checked out 'fd06fa49e4e3f4811df4ee566a0ea41832eb0cca'
Submodule path 'src/deps/tinyply': checked out 'a6ca3c89480b89d271b6909e16d8423ef0025336'
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Check for working C compiler: /home/pl/anaconda3/envs/lookout/bin/x86_64-conda_cos6-linux-gnu-cc
-- Check for working C compiler: /home/pl/anaconda3/envs/lookout/bin/x86_64-conda_cos6-linux-gnu-cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/pl/anaconda3/envs/lookout/bin/x86_64-conda_cos6-linux-gnu-c++
-- Check for working CXX compiler: /home/pl/anaconda3/envs/lookout/bin/x86_64-conda_cos6-linux-gnu-c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
CMake Warning at deps/glog/CMakeLists.txt:49 (find_package):
By not providing "Findgflags.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "gflags", but
CMake did not find one.
Could not find a package configuration file provided by "gflags" (requested
version 2.2.0) with any of the following names:
gflagsConfig.cmake
gflags-config.cmake
Add the installation prefix of "gflags" to CMAKE_PREFIX_PATH or set
"gflags_DIR" to a directory containing one of the above files. If "gflags"
provides a separate development package or SDK, be sure it has been
installed.
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for execinfo.h
-- Looking for execinfo.h - found
-- Looking for glob.h
-- Looking for glob.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for libunwind.h
-- Looking for libunwind.h - not found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for pwd.h
-- Looking for pwd.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/syscall.h
-- Looking for sys/syscall.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for sys/utsname.h
-- Looking for sys/utsname.h - found
-- Looking for syscall.h
-- Looking for syscall.h - found
-- Looking for syslog.h
-- Looking for syslog.h - found
-- Looking for ucontext.h
-- Looking for ucontext.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for unwind.h
-- Looking for unwind.h - found
-- Looking for C++ include ext/hash_map
-- Looking for C++ include ext/hash_map - found
-- Looking for C++ include ext/hash_set
-- Looking for C++ include ext/hash_set - found
-- Looking for C++ include ext/slist
-- Looking for C++ include ext/slist - found
-- Looking for C++ include tr1/unordered_map
-- Looking for C++ include tr1/unordered_map - found
-- Looking for C++ include tr1/unordered_set
-- Looking for C++ include tr1/unordered_set - found
-- Looking for C++ include unordered_map
-- Looking for C++ include unordered_map - found
-- Looking for C++ include unordered_set
-- Looking for C++ include unordered_set - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned __int16
-- Check size of unsigned __int16 - failed
-- Check size of u_int16_t
-- Check size of u_int16_t - done
-- Check size of uint16_t
-- Check size of uint16_t - done
-- Looking for dladdr
-- Looking for dladdr - not found
-- Looking for fcntl
-- Looking for fcntl - found
-- Looking for pread
-- Looking for pread - found
-- Looking for pwrite
-- Looking for pwrite - found
-- Looking for sigaction
-- Looking for sigaction - found
-- Looking for sigaltstack
-- Looking for sigaltstack - found
-- Performing Test HAVE_NO_DEPRECATED
-- Performing Test HAVE_NO_DEPRECATED - Success
-- Performing Test HAVE_NO_UNNAMED_TYPE_TEMPLATE_ARGS
-- Performing Test HAVE_NO_UNNAMED_TYPE_TEMPLATE_ARGS - Failed
-- Looking for snprintf
-- Looking for snprintf - found
-- Looking for get_static_proc_name in unwind
-- Looking for get_static_proc_name in unwind - not found
-- Looking for UnDecorateSymbolName in dbghelp
-- Looking for UnDecorateSymbolName in dbghelp - not found
-- Performing Test HAVE___ATTRIBUTE__
-- Performing Test HAVE___ATTRIBUTE__ - Success
-- Performing Test HAVE___ATTRIBUTE__VISIBILITY_DEFAULT
-- Performing Test HAVE___ATTRIBUTE__VISIBILITY_DEFAULT - Success
-- Performing Test HAVE___ATTRIBUTE__VISIBILITY_HIDDEN
-- Performing Test HAVE___ATTRIBUTE__VISIBILITY_HIDDEN - Success
-- Performing Test HAVE___BUILTIN_EXPECT
-- Performing Test HAVE___BUILTIN_EXPECT - Success
-- Performing Test HAVE___SYNC_VAL_COMPARE_AND_SWAP
-- Performing Test HAVE___SYNC_VAL_COMPARE_AND_SWAP - Success
-- Performing Test HAVE_RWLOCK
-- Performing Test HAVE_RWLOCK - Failed
-- Performing Test HAVE___DECLSPEC
-- Performing Test HAVE___DECLSPEC - Failed
-- Performing Test STL_NO_NAMESPACE
-- Performing Test STL_NO_NAMESPACE - Failed
-- Performing Test STL_STD_NAMESPACE
-- Performing Test STL_STD_NAMESPACE - Failed
-- Performing Test HAVE_USING_OPERATOR
-- Performing Test HAVE_USING_OPERATOR - Failed
-- Performing Test HAVE_NAMESPACES
-- Performing Test HAVE_NAMESPACES - Success
-- Performing Test HAVE_GCC_TLS
-- Performing Test HAVE_GCC_TLS - Success
-- Performing Test HAVE_MSVC_TLS
-- Performing Test HAVE_MSVC_TLS - Failed
-- Performing Test HAVE_CXX11_TLS
-- Performing Test HAVE_CXX11_TLS - Success
-- Performing Test HAVE_LOCALTIME_R
-- Performing Test HAVE_LOCALTIME_R - Success
-- Shared libraries disabled
-- Looking for ZLIB...
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'zzip-zlib-config'
-- No package 'zzip-zlib-config' found
-- Found ZLIB: optimized;/home/pl/anaconda3/envs/lookout/lib/libz.so;debug;/home/pl/anaconda3/envs/lookout/lib/libz.so
-- Checking for module 'minizip'
-- No package 'minizip' found
-- Found rt: /home/pl/anaconda3/envs/lookout/x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/librt.so
-- Enabled formats: AMF 3DS AC ASE ASSBIN ASSXML B3D BVH COLLADA DXF CSM HMP IRRMESH IRR LWO LWS MD2 MD3 MD5 MDC MDL NFF NDO OFF OBJ OGRE OPENGEX PLY MS3D COB BLEND IFC XGL FBX Q3D Q3BSP RAW SIB SMD STL TERRAGEN 3D X X3D GLTF 3MF MMD
-- Disabled formats:
-- Found Assimp: assimp
-- Could NOT find Vulkan (missing: VULKAN_LIBRARY VULKAN_INCLUDE_DIR)
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Using X11 for window creation
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
Could NOT find X11 (missing: X11_X11_INCLUDE_PATH X11_X11_LIB)
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.16/Modules/FindX11.cmake:366 (find_package_handle_standard_args)
deps/glfw/CMakeLists.txt:218 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/pl/habitat-sim/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeOutput.log".
See also "/home/pl/habitat-sim/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeError.log".
/home/pl/habitat-sim/setup.py:297: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
assert StrictVersion(
/home/pl/habitat-sim/setup.py:299: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
) >= StrictVersion("3.6"), "Must use python3.6 or newer"
/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
easy_install.initialize_options(self)
/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/pl/habitat-sim/setup.py", line 313, in <module>
setup(
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
self.install_for_development()
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/command/develop.py", line 111, in install_for_development
self.run_command('build_ext')
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/pl/habitat-sim/setup.py", line 166, in run
self.build_extension(ext)
File "/home/pl/habitat-sim/setup.py", line 213, in build_extension
subprocess.check_call(
File "/home/pl/anaconda3/envs/lookout/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-H/home/pl/habitat-sim/src', '-Bbuild/temp.linux-x86_64-cpython-38', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/pl/habitat-sim/habitat_sim/_ext', '-DPYTHON_EXECUTABLE=/home/pl/anaconda3/envs/lookout/bin/python', '-DCMAKE_EXPORT_COMPILE_COMMANDS=OFF', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-GNinja', '-DBUILD_GUI_VIEWERS=ON', '-DBUILD_TESTS=OFF']' returned non-zero exit status 1.
----------------------------------------
ERROR: Command errored out with exit status 1: /home/pl/anaconda3/envs/lookout/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/pl/habitat-sim/setup.py'"'"'; file='"'"'/home/pl/habitat-sim/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
"
Hi,
I'm trying to train your model on SceneNet-RGBD Dataset. The resolution of images in SceneNet is 320x240.
But the PtsManipulator
class in models/projection/z_buffer_manipulator.py
takes only width as argument and assumes height is same as width. How can I modify this class to allow for rectangular images?
Hello. Thanks for the helpful repository. It is really insightful. I have just started to learn about Pytorch3D. I was a bit confused about the process of generating point cloud from the input image and generating an image from the point cloud again. I will be grateful if you kindly help me understand the codebase exactly where and how the point cloud is generated, re-rendered into an image and backpropagated.
Dear authors,
When I tried to infer the model locally on RealEstate10k with the provided pretrained model, the accuracy difference was very large from the ones mentioned in the paper. As mentioned in the Readme, I understand the reason of this issue. But as I am working on a variant of Synsin and a smaller subset of RealEstate10k, I needed to report the result obtained using Pytorch3D distribution. Can you kindly suggest me how I should proceed?
I ran the inference on Windows OS on GTX 1070 GPU with batch size 1. My python version is 3.8 and pytorch version is 1.5.1. I saved the extracted images as 256*256 in the disk.
Hi,
I'm trying to train the SynSin model on SceneNet database. But I'm not able to train the model. I would really appreciate it, if you can give me some tips.
--use_gt_depth
flag.D_Real
and D_Fake
have similar values in each batch (around 0.1 to 0.3). So, discriminator isn't training as well.I don't know what else to try. Can you kindly help me out here?
Does the Interactive Demo
extract the frames?
[this](The frames need to be extracted based on the text files; we extract them to: ${REAL_ESTATE_10K}/frames.) says...
"The frames need to be extracted based on the text files; we extract them to: ${REAL_ESTATE_10K}/frames."
How do "we" do that?
Hey.
Thanks for the great project and code.
I am running the code by myself and I just wonder does the "lr" in the train.sh actually control anything?
In the model/base_model.py, the learning rates for G and D are specified by lr_g and lr_d (line 21),
and it seems that lr_g and lr_d are default values that are different from the reported number in the paper.
I just wonder is my understanding correct?
i want use the code of appearance flow prediction;the result of my training is so bad,i want ask that the training setting ,the parameter is look like?
this is my training setting,for the gpu setting my batchsize is only 8:
python train.py --batch-size 8 --folder 'temp' --num_workers 2
--resume --dataset 'kitti' --use_inv_z --use_inverse_depth --accumulation 'alphacomposite'
--model_type 'viewappearance' --refine_model_type 'resnet_256W8UpDown64'
--norm_G 'sync:spectral_batch' --gpu_ids 0,1 --render_ids 0
--suffix '' --normalize_image --lr 0.000025 --log-dir="/home/zhuge/try/sin1/store_old_model/%s"
my traing result:
ssim : 0.2528 | psnr : 2.3678 | Perceptual : 0.8591 | Total Loss : 8.8668 | L1 : 0.2760 |
so what kind of setting should i set?
Hey -- thanks for releasing the code, it's been very useful.
I've been reading through to try to make I understand everything, and there's one part that has me a bit confused:
In the top-level training script, train.py
, an instance of the main training model ZbufferModelPts
is instantiated, which in turn instantiates all the sub-models, which include the depth estimator, the feature encoder, and the feature decoder.
In particular, the feature decoder is instantiated as:
synsin/models/z_buffermodel.py
Line 35 in 82ff948
which, if we've chosen to use the ResNet blocks (as is default in the parameters shown in train.sh
), will return:
synsin/models/networks/utilities.py
Line 31 in 82ff948
Shouldn't the channels_in
here be 64, since it's taking as input the re-projected features? Or am I missing something?
Hi olivia:
your work inspired me a lot! Thank you!
Here I just wonder --norm_G 'sync:spectral_batch'
in the train.sh
, does it use for single GPU multiple Nvidia graphic cards? Which means if I want to test it on only one card, It's ok for me to change this to --norm_G 'spectral_batch'
?
Thank you!
the 256x256 output looks blurry, wondering in the training/inference pipeline, does it support 512x512 or even higher resolution.
Hi,
I'm trying to run baseline comparison with SynSin on RealEstate10K. Since SynSin is not scale-invariant, I computed a scale factor using a sparse point cloud (from point triangulation using the camera poses) and the depth image produced by the model, and scale both of the source pose and the target pose accordingly as follows:
ref_cams = batch["cameras"][0]
dst_cams = batch["cameras"][1]
ref_pose = K_offset_inv @ ref_cams["P"][:, :-1].cuda()
dst_pose = K_offset_inv @ dst_cams["P"][:, :-1].cuda()
ref_pose[:, 0:3, 3] /= scale_factor.view(-1, 1)
dst_pose[:, 0:3, 3] /= scale_factor.view(-1, 1)
After doing this I ran forward again, but this led to much worse results than without the scale factor. What's more, scaling the ref pose only seems to not make any difference in the results, which looks strange. Is this the right way to do scale-invariant evaluation? If not, what is the correct way of doing this?
Thanks.
I have downloaded all images following realestate.txt except these files
00ae21ab50209282.txt 10eebcbb9021f437.txt 24ac24abf3057732.txt 2cf32e2408107ea7.txt 3916390b35258215.txt 44b2ab5292c06a7e.txt
0a5e107e1961d01d.txt 23341c3a0b420e54.txt 2bd43375196ce1a7.txt 3172ad0d099430da.txt 416f82fdbad68e21.txt 477a0a9f77c00480.txt
I cannot get the videos of these files, because they all need 5uP-le4wTaM, but I have no permission of it. Can you provide it?
My email is [email protected]
Hello,
I am just wondering why you only use 500 steps for each epoch during training ? Is there any particular reason why you do this ?
Why dont you just let the model iterate through the whole dataset like this :
for iteration in range(1, len(data_loader):
Line 91 in 82ff948
Hi,
I have been trying to replicate your result on the MP3D dataset and I receive this error :
/home/phong/anaconda3/envs/pytorch/bin/python /home/phong/data/Work/Paper3/Code/synsin/evaluation/eval.py --result_folder ../eval_mp3d/ --old_model ../modelcheckpoints/mp3d/synsin.pth --batch_size 8 --num_workers 0 --images_before_reset 200 --dataset mp3d
STARTING MAIN METHOD...
Model is: ../modelcheckpoints/mp3d/synsin.pth
Loading dataset mp3d ...
Loading model %s ...
RESNET encoder
RESNET decoder
['1.0_l1', '10.0_content']
<zip object at 0x7f353981a488>
Loaded models...
Restarting image_generator.... with seed 0 in train mode? True
gpu_id 0
data/scene_episodes/mp3d_test
One ep per scene
2020-04-06 10:08:19,606 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0406 10:08:19.607789 19727 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/oLBMNvg9in8/oLBMNvg9in8.navmesh
I0406 10:08:19.608074 19727 Simulator.cpp:64] Loaded.
I0406 10:08:19.608234 19727 SceneGraph.h:92] Created DrawableGroup:
Renderer: GeForce RTX 2080 Ti/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 435.21
Using optional features:
GL_ARB_ES2_compatibility
GL_ARB_direct_state_access
GL_ARB_get_texture_sub_image
GL_ARB_invalidate_subdata
GL_ARB_multi_bind
GL_ARB_robustness
GL_ARB_separate_shader_objects
GL_ARB_texture_filter_anisotropic
GL_ARB_texture_storage
GL_ARB_texture_storage_multisample
GL_ARB_vertex_array_object
GL_KHR_debug
Using driver workarounds:
no-layout-qualifiers-on-old-glsl
nv-zero-context-profile-mask
nv-implementation-color-read-format-dsa-broken
nv-cubemap-inconsistent-compressed-image-size
nv-cubemap-broken-full-compressed-image-query
nv-compressed-block-size-in-bits
I0406 10:08:19.680424 19727 ResourceManager.cpp:1119] Importing Basis files as BC7
I0406 10:08:23.214200 19727 Simulator.cpp:130] Loading house from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/oLBMNvg9in8/oLBMNvg9in8.house
I0406 10:08:23.214274 19727 Simulator.cpp:136] Loading semantic mesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/oLBMNvg9in8/oLBMNvg9in8_semantic.ply
I0406 10:08:23.214280 19727 SceneGraph.h:92] Created DrawableGroup:
I0406 10:08:31.861382 19727 Simulator.cpp:149] Loaded.
I0406 10:08:31.953205 19727 simulator.py:143] Loaded navmesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/oLBMNvg9in8/oLBMNvg9in8.navmesh
2020-04-06 10:08:31,955 Initializing task Nav-v0
I0406 10:08:32.014276 19727 Simulator.cpp:36] Deconstructing Simulator
I0406 10:08:32.014317 19727 SemanticScene.h:40] Deconstructing SemanticScene
I0406 10:08:32.017783 19727 SceneManager.h:24] Deconstructing SceneManager
I0406 10:08:32.017791 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:08:32.017853 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:08:32.018307 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:08:32.018316 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:08:32.018569 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:08:32.018579 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:08:32.027341 19727 Renderer.cpp:33] Deconstructing Renderer
I0406 10:08:32.027349 19727 WindowlessContext.h:16] Deconstructing WindowlessContext
I0406 10:08:32.027352 19727 WindowlessContext.cpp:245] Deconstructing GL context
I0406 10:08:32.047761 19727 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/X7HyMhZNoso/X7HyMhZNoso.navmesh
I0406 10:08:32.048007 19727 Simulator.cpp:64] Loaded.
I0406 10:08:32.048141 19727 SceneGraph.h:92] Created DrawableGroup:
Renderer: GeForce RTX 2080 Ti/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 435.21
Using optional features:
GL_ARB_ES2_compatibility
GL_ARB_direct_state_access
GL_ARB_get_texture_sub_image
GL_ARB_invalidate_subdata
GL_ARB_multi_bind
GL_ARB_robustness
GL_ARB_separate_shader_objects
GL_ARB_texture_filter_anisotropic
GL_ARB_texture_storage
GL_ARB_texture_storage_multisample
GL_ARB_vertex_array_object
GL_KHR_debug
Using driver workarounds:
no-layout-qualifiers-on-old-glsl
nv-zero-context-profile-mask
nv-implementation-color-read-format-dsa-broken
nv-cubemap-inconsistent-compressed-image-size
nv-cubemap-broken-full-compressed-image-query
nv-compressed-block-size-in-bits
I0406 10:08:32.123594 19727 ResourceManager.cpp:1119] Importing Basis files as BC7
I0406 10:08:34.366274 19727 Simulator.cpp:130] Loading house from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/X7HyMhZNoso/X7HyMhZNoso.house
I0406 10:08:34.366349 19727 Simulator.cpp:136] Loading semantic mesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/X7HyMhZNoso/X7HyMhZNoso_semantic.ply
I0406 10:08:34.366355 19727 SceneGraph.h:92] Created DrawableGroup:
I0406 10:08:40.764156 19727 Simulator.cpp:149] Loaded.
I0406 10:08:40.813067 19727 simulator.py:143] Loaded navmesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/X7HyMhZNoso/X7HyMhZNoso.navmesh
EPISODES A 10
I0406 10:08:40.946849 19727 Simulator.cpp:36] Deconstructing Simulator
I0406 10:08:40.946892 19727 SemanticScene.h:40] Deconstructing SemanticScene
I0406 10:08:40.948513 19727 SceneManager.h:24] Deconstructing SceneManager
I0406 10:08:40.948518 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:08:40.948561 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:08:40.949321 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:08:40.949329 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:08:40.949751 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:08:40.949761 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:08:40.955967 19727 Renderer.cpp:33] Deconstructing Renderer
I0406 10:08:40.955977 19727 WindowlessContext.h:16] Deconstructing WindowlessContext
I0406 10:08:40.955981 19727 WindowlessContext.cpp:245] Deconstructing GL context
I0406 10:08:40.982967 19727 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/Z6MFQCViBuw/Z6MFQCViBuw.navmesh
I0406 10:08:40.983465 19727 Simulator.cpp:64] Loaded.
I0406 10:08:40.983618 19727 SceneGraph.h:92] Created DrawableGroup:
I0406 10:08:41.095933 19727 ResourceManager.cpp:1119] Importing Basis files as BC7
Renderer: GeForce RTX 2080 Ti/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 435.21
Using optional features:
GL_ARB_ES2_compatibility
GL_ARB_direct_state_access
GL_ARB_get_texture_sub_image
GL_ARB_invalidate_subdata
GL_ARB_multi_bind
GL_ARB_robustness
GL_ARB_separate_shader_objects
GL_ARB_texture_filter_anisotropic
GL_ARB_texture_storage
GL_ARB_texture_storage_multisample
GL_ARB_vertex_array_object
GL_KHR_debug
Using driver workarounds:
no-layout-qualifiers-on-old-glsl
nv-zero-context-profile-mask
nv-implementation-color-read-format-dsa-broken
nv-cubemap-inconsistent-compressed-image-size
nv-cubemap-broken-full-compressed-image-query
nv-compressed-block-size-in-bits
I0406 10:08:47.203900 19727 Simulator.cpp:130] Loading house from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/Z6MFQCViBuw/Z6MFQCViBuw.house
I0406 10:08:47.204015 19727 Simulator.cpp:136] Loading semantic mesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/Z6MFQCViBuw/Z6MFQCViBuw_semantic.ply
I0406 10:08:47.204020 19727 SceneGraph.h:92] Created DrawableGroup:
I0406 10:08:51.339296 19727 Simulator.cpp:149] Loaded.
I0406 10:08:51.390200 19727 simulator.py:143] Loaded navmesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/Z6MFQCViBuw/Z6MFQCViBuw.navmesh
EPISODES B 10
I0406 10:08:51.597631 19727 Simulator.cpp:36] Deconstructing Simulator
I0406 10:08:51.597692 19727 SemanticScene.h:40] Deconstructing SemanticScene
I0406 10:08:51.599572 19727 SceneManager.h:24] Deconstructing SceneManager
I0406 10:08:51.599578 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:08:51.599647 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:08:51.600244 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:08:51.600253 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:08:51.600458 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:08:51.600469 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:08:51.608188 19727 Renderer.cpp:33] Deconstructing Renderer
I0406 10:08:51.608198 19727 WindowlessContext.h:16] Deconstructing WindowlessContext
I0406 10:08:51.608201 19727 WindowlessContext.cpp:245] Deconstructing GL context
I0406 10:08:51.625839 19727 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/QUCTc6BB5sX/QUCTc6BB5sX.navmesh
I0406 10:08:51.626281 19727 Simulator.cpp:64] Loaded.
I0406 10:08:51.626430 19727 SceneGraph.h:92] Created DrawableGroup:
Renderer: GeForce RTX 2080 Ti/PCIe/SSE2 by NVIDIA Corporation
OpenGL version: 4.6.0 NVIDIA 435.21
Using optional features:
GL_ARB_ES2_compatibility
GL_ARB_direct_state_access
GL_ARB_get_texture_sub_image
GL_ARB_invalidate_subdata
GL_ARB_multi_bind
GL_ARB_robustness
GL_ARB_separate_shader_objects
GL_ARB_texture_filter_anisotropic
GL_ARB_texture_storage
GL_ARB_texture_storage_multisample
GL_ARB_vertex_array_object
GL_KHR_debug
Using driver workarounds:
no-layout-qualifiers-on-old-glsl
nv-zero-context-profile-mask
nv-implementation-color-read-format-dsa-broken
nv-cubemap-inconsistent-compressed-image-size
nv-cubemap-broken-full-compressed-image-query
nv-compressed-block-size-in-bits
I0406 10:08:51.675937 19727 ResourceManager.cpp:1119] Importing Basis files as BC7
I0406 10:08:56.669844 19727 Simulator.cpp:130] Loading house from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/QUCTc6BB5sX/QUCTc6BB5sX.house
I0406 10:08:56.669992 19727 Simulator.cpp:136] Loading semantic mesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/QUCTc6BB5sX/QUCTc6BB5sX_semantic.ply
I0406 10:08:56.669998 19727 SceneGraph.h:92] Created DrawableGroup:
I0406 10:09:06.969951 19727 Simulator.cpp:149] Loaded.
I0406 10:09:07.064879 19727 simulator.py:143] Loaded navmesh /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat/mp3d/QUCTc6BB5sX/QUCTc6BB5sX.navmesh
RESETTING 10 to 10
0%| | 0/3600 [00:00<?, ?it/s]/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py:2506: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
Warning : not backpropagating through the projection -- is this what you want??
0%| | 0/3600 [00:02<?, ?it/s]
Traceback (most recent call last):
File "/home/phong/data/Work/Paper3/Code/synsin/evaluation/eval.py", line 75, in check_initial_batch
np.loadtxt('/home/phong/data/Work/Paper3/Code/synsin/data/files/eval_cached_cameras_mp3d.txt'))
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
verbose=verbose, header=header, equal_nan=equal_nan)
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 96 / 128 (75%)
Max absolute difference: 38.9685936
Max relative difference: 89.41226322
x: array([-7.876723e-03, -5.965701e-02, -9.981878e-01, -1.659706e+01,
-3.896221e-02, 9.974791e-01, -5.930720e-02, 1.113027e+00,
9.992096e-01, 3.842445e-02, -1.018124e-02, -2.654723e+00,...
y: array([ 6.228155e-01, 1.615186e-02, 7.822020e-01, 1.141124e+01,
6.692571e-02, 9.950223e-01, -7.383493e-02, -2.435736e+00,
-7.795010e-01, 9.833497e-02, 6.186343e-01, 8.109766e+00,...
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/phong/data/Work/Paper3/Code/synsin/evaluation/eval.py", line 203, in <module>
check_initial_batch(batch, test_ops.dataset)
File "/home/phong/data/Work/Paper3/Code/synsin/evaluation/eval.py", line 78, in check_initial_batch
\n\nYour results will NOT be comparable with results in the paper or online.")
Exception:
There is an error with your setup or options.
Your results will NOT be comparable with results in the paper or online.
I0406 10:09:10.500044 19727 Simulator.cpp:36] Deconstructing Simulator
I0406 10:09:10.500248 19727 SemanticScene.h:40] Deconstructing SemanticScene
I0406 10:09:10.506703 19727 SceneManager.h:24] Deconstructing SceneManager
I0406 10:09:10.506712 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:09:10.506816 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:09:10.508659 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:09:10.508671 19727 RenderTarget.h:51] Deconstructing RenderTarget
I0406 10:09:10.509035 19727 Sensor.h:80] Deconstructing Sensor
I0406 10:09:10.509048 19727 SceneGraph.h:25] Deconstructing SceneGraph
I0406 10:09:10.523064 19727 Renderer.cpp:33] Deconstructing Renderer
I0406 10:09:10.523079 19727 WindowlessContext.h:16] Deconstructing WindowlessContext
I0406 10:09:10.523083 19727 WindowlessContext.cpp:245] Deconstructing GL context
Process finished with exit code 1
So i guess the problem is in this line
Line 74 in 82ff948
The only change is that I have to edit the opts.config since you hardcoded it
Line 119 in 82ff948
opts.config = '/home/phong/data/Work/Paper3/Libraries/habitat-api/configs/tasks/pointnav_rgbd.yaml'
When I tried to comment out the checking function
Line 202 in 82ff948
hi,Thanks for you research.I have one question that I hope you can answer.
When I extract frames as your format,The content of image's txt file is different.
Can you explain it? I appreciate your help!
for example
you provide(https://raw.githubusercontent.com/facebookresearch/synsin/master/data/files/realestate.txt) :
000c3ab189999a83 49716333 49716333 123.477562 219.515671 128.0 128.0 0.994582 0.007716 -0.103669 0.066855 -0.007079 0.999954 0.006514 0.050014 0.103715 -0.005744 0.994591 -2.286677 0.994582 0.007716 -0.103669 0.066855 -0.007079 0.999954 0.006514 0.050014 0.103715 -0.005744 0.994591 -2.286677
the raw data(RealEstate10K/test/000c3ab189999a83.txt):
49716333 0.482334223 0.857483078 0.500000000 0.500000000 0.000000000 0.000000000 0.994581878 0.007715873 -0.103669293 0.066854894 -0.007078611 0.999953747 0.006513574 0.050014287 0.103714757 -0.005744448 0.994590521 -2.286676987
the link does not work
Probably not an issue, it's more a clarification: can you point me to the neural point cloud renderer code in this repository, as described in the paper? The proposed solutions to naive rendering are very interesting, I'd like to explore them further in some other tasks.
Can someone tell me what inference time is in this method?can this method used in real time system? thanks
Hello,
This is not an issue about the code of SynSin but is a technical question related to RealEstate10K dataset. So I have been trying to train SynSin with this dataset and I have manage to train a small model with about 100 videos. The performance is far worse than the provided checkpoint but I would like to train with the whole dataset so I would like to ask how large is this RealEstate10K dataset.
I had a limit of 1TB storage in my own cloud server so I don't know whether if I can ever train SynSin on the whole dataset or not ^^
conda create –-name synsin_env –-file requirements.txt
gives the error: CondaValueError: The target prefix is the base prefix. Aborting.
I tried to install the required libs using virtual environment, but import quaternion
fails and pytorch3d also report an error.
Hello, in the z_buffer_layers.py class, there is an optional argument, learn_feature. I was unsure about the functionality of this argument as it is commented out in the forward function. Can you kindly help me to understand this?
Hi,
Thanks for sharing this! I was running the Simple Demo.ipynb and got this error
It seems that pytorch3d v0.2.0 removed compositing.CompositeParams. Also the fourth param of compositing function is not used anywhere in compositing.py.
def alpha_composite(pointsidx, alphas, pt_clds, blend_params=None)
The blend_params is not used.
Does this change matter? Then we need to change models/layers/z_buffer_layers.py according right? I tried to comment out every "params" and it worked. Is it the right solution?
Hi! Great work! I would like to know how you generate dataset_one_ep_per_scene.json.gz
?(https://github.com/facebookresearch/synsin/blob/main/data/scene_episodes/mp3d_train/dataset_one_ep_per_scene.json.gz)
Can you share your generation code? Since Habitat-Matterport3D(HM3D) is available, I would like to imitate your generation steps on HM3D. But I have no idea how to deal with it.
I am currently setting the Replica dataset as instructed here, but I was unable to find the pointnav dataset for replica. Habitat-api seems to offer PointNav datasets for Gibson and Matterport3D, but not Replica. I wasn't sure if I missed the link somewhere, or if I should be generating the dataset independently. Thank you for your help!
I downloaded the models and those load, but I get
OSError: /checkpoint/ow045820/data/realestate10K/RealEstate10K//frames/train/video_loc.txt not found.
When I try to run the cell 'Load the dataset' in the 'Interactive Demo' notebook.
Hi,could you explain why the relative pose is calculated as RA.T@RB and RA.T.dot(TB-TA)/50 when processing the kitti dataset?
Looking forward to your reply.
Thank you!
Hello. Thanks for great work.
I have some questions about the project.
In realestate10k.py, you calculate the K matrix with the intrinsics of the camera but then you use the self.K which is an eye matrix defined in init to return in loader. In other datasets while intrinsics are same for all the dataset you define the K in the init and use it in getitem which is clearly correct for me. But in realestate why you don't use the correct K matrix.
`
origK = np.array(
[
[intrinsics[0], 0, intrinsics[2]],
[0, intrinsics[1], intrinsics[3]],
[0, 0, 1],
],
dtype=np.float32,
)
K = np.matmul(self.offset, origK)
origP = extrinsics.reshape(3, 4)
P = np.matmul(K, origP) # Merge these together to match habitat
P = np.vstack((P, np.zeros((1, 4), dtype=np.float32))).astype(
np.float32
)
P[3, 3] = 1
Pinv = np.linalg.inv(P)
cameras += [
{
"P": P,
"Pinv": Pinv,
"OrigP": origP,
"K": self.K,
"Kinv": self.invK,
}
]
return {"images": rgbs, "cameras": cameras}`
And one another issue, in kitti.py there is a matrix multiplication which i don't understand. Multiplying offset with K will make the [0,0] and [1,1] cells' values 2*(718.9/256) and will make [0,2],[1,2] cells, 0. And -1 does not multiply with anything just zero. Which point do I miss? Can you clarify?
` K = np.array(
[718.9 / 256., 0., 128 / 256., 0,
0., 718.9 / 256., 128 / 256., 0,
0., 0., 1., 0.,
0., 0., 0., 1.]).reshape((4, 4)).astype(np.float32)
offset = np.array(
[[2, 0, -1, 0], [0, -2, 1, 0], [0, 0, 1, 0], [0, 0, 0, 1]], # Flip ys to match habitat
dtype=np.float32,
) # Make z negative to match habitat (which assumes a negative z)
K = np.matmul(offset, K)
`
Hi, there seems to be a bug in the code. In the function PtsManipulator, there are lines:
projected_coors = self.xyzs * pts3D
projected_coors[:, -1, :] = 1
and
sampler = torch.cat((xy_proj[:, 0:2, :] / -zs, xy_proj[:, 2:3, :]), 1)
sampler[mask.repeat(1, 3, 1)] = -10
However, by doing this, you are treating depth as the last element of homogenous coordinate, [x*d1, y*d1, d1] is equal to [x*d2, y*d2, d2].
As a result, this function do not care what is the depth, it basically treats every pixel as they are in the same plane.
Even putting random noise (as long as positive) as depth map, the rendered RGBs will be the same as the ones using GT depth maps. The rendered RGBs only depend on T, R, and the input RGBs.
Can you check and fix it? Thank you.
It seems like the evaluation of SynSin on RealEstate10K is performed on 256x256 images. However, I couldn't find some of the data preprocessing hyperparameters in the REALESTATE.md
for results reproduction, such as (i) what resolution of the raw video did the authors download and work with, (ii) how was the 256x256 patch extracted from the video frames (e.g., center crop? or center crop after resizing?), and (iii) how did the author choose the video frames to evaluate within each video clip (each video clip has a huge set of labeled frames in RealEstate10K)?
Could the authors make a clarification on these hyperparameters?
Sincerely thanks!
Hi, I have downloaded the Mp3D dataset and I am trying to run train.sh file using a single GPU with small batch size =2 for debugging.
This is my train.sh:
python train.py --batch-size 2 --folder 'mp3d' --num_workers 0 \
--resume --accumulation 'alphacomposite' \
--model_type 'zbuffer_pts' --refine_model_type 'resnet_256W8UpDown64' \
--norm_G 'sync:spectral_batch' --gpu_ids 0 --render_ids 1 \
--suffix '' --normalize_image --lr 0.0001
I also have changed Mp3D directory inside the options.py and some hard-coded str in the train_options.py. However , I am experiencing some broken pip error related to glX context does not support multiple GPUs.
phong@phong-Server:~/data/Work/Paper3/Code/synsin$ ./train.sh
Timestamp 2020-03-01
./checkpoint/phong/logging/viewsynthesis3d/mp3d//2020-03-01/mp3d/models/lr0.00010_bs2_modelzbuffer_pts_splxyblending/noise_bnsync:spectral_batch_refresnet_256W8UpDown64_dunet_camxysFalse|False/_init_databoth_seed0/_multiFalse_losses1.0|l110.0|content_izFalse_alphaFalse__vol_ganpix2pixHD/
Model ./checkpoint/phong/logging/viewsynthesis3d/mp3d//2020-03-01/mp3d/models/lr0.00010_bs2_modelzbuffer_pts_splxyblending/noise_bnsync:spectral_batch_refresnet_256W8UpDown64_dunet_camxysFalse|False/_init_databoth_seed0/_multiFalse_losses1.0|l110.0|content_izFalse_alphaFalse__vol_ganpix2pixHD//model_epoch.pth
Loading dataset mp3d ...
Loading model %s ...
RESNET encoder
RESNET decoder
['1.0_l1', '10.0_content']
<zip object at 0x7fea940d0bc8>
./checkpoint/phong/logging/viewsynthesis3d/mp3d//%s//2020-03-01/mp3d/runs/lr0.00010_bs2_modelzbuffer_pts_splxyblending/noise_bnsync:spectral_batch_refresnet_256W8UpDown64_dunet_camxysFalse|False/_init_databoth_seed0/_multiFalse_losses1.0|l110.0|content_izFalse_alphaFalse__vol_ganpix2pixHD/
[0]
Starting run...
WARNING: Model path does not exist??
./checkpoint/phong/logging/viewsynthesis3d/mp3d//2020-03-01/mp3d/models/lr0.00010_bs2_modelzbuffer_pts_splxyblending/noise_bnsync:spectral_batch_refresnet_256W8UpDown64_dunet_camxysFalse|False/_init_databoth_seed0/_multiFalse_losses1.0|l110.0|content_izFalse_alphaFalse__vol_ganpix2pixHD//model_epoch.pth
Loading train dataset ....
Loaded train dataset ...
Starting epoch 0
At train
Restarting image_generator.... with seed 0 in train mode? True
gpu_id 1
data/scene_episodes/mp3d_train
One ep per scene
61
2020-03-27 10:15:28,995 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0327 10:15:28.999747 3051 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat//mp3d/gZ6f7yhEvPG/gZ6f7yhEvPG.navmesh
2020-03-27 10:15:29,003 initializing sim Sim-v0
2020-03-27 10:15:29,011 initializing sim Sim-v0
2020-03-27 10:15:29,011 initializing sim Sim-v0
I0327 10:15:29.017060 3051 Simulator.cpp:64] Loaded.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0327 10:15:29.017076 3049 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat//mp3d/JeFG25nYj2p/JeFG25nYj2p.navmesh
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0327 10:15:29.017110 3048 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat//mp3d/1pXnuDYAj8r/1pXnuDYAj8r.navmesh
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0327 10:15:29.017143 3050 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat//mp3d/S9hNv5qa7GM/S9hNv5qa7GM.navmesh
2020-03-27 10:15:29,025 initializing sim Sim-v0
I0327 10:15:29.029383 3049 Simulator.cpp:64] Loaded.
I0327 10:15:29.043566 3051 SceneGraph.h:92] Created DrawableGroup:
F0327 10:15:29.043596 3051 WindowlessContext.cpp:232] Check failed: device == 0 (1 vs. 0) glX context does not support multiple GPUs. Please compile with BUILD_GUI_VIEWERS=0 for multi-gpu support via EGL
I0327 10:15:29.043599 3049 SceneGraph.h:92] Created DrawableGroup:
F0327 10:15:29.043618 3049 WindowlessContext.cpp:232] Check failed: device == 0 (1 vs. 0) glX context does not support multiple GPUs. Please compile with BUILD_GUI_VIEWERS=0 for multi-gpu support via EGL
*** Check failure stack trace: ***
*** Check failure stack trace: ***
I0327 10:15:29.043759 3048 Simulator.cpp:64] Loaded.
I0327 10:15:29.043802 3050 Simulator.cpp:64] Loaded.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0327 10:15:29.043810 3052 Simulator.cpp:62] Loading navmesh from /media/phong/Data2TB/dataset/Matterport/Habitat/v1/tasks/mp3d_habitat//mp3d/qoiz87JEwZ2/qoiz87JEwZ2.navmesh
I0327 10:15:29.053377 3048 SceneGraph.h:92] Created DrawableGroup:
F0327 10:15:29.053406 3048 WindowlessContext.cpp:232] Check failed: device == 0 (1 vs. 0) glX context does not support multiple GPUs. Please compile with BUILD_GUI_VIEWERS=0 for multi-gpu support via EGL
*** Check failure stack trace: ***
I0327 10:15:29.053519 3050 SceneGraph.h:92] Created DrawableGroup:
F0327 10:15:29.053547 3050 WindowlessContext.cpp:232] Check failed: device == 0 (1 vs. 0) glX context does not support multiple GPUs. Please compile with BUILD_GUI_VIEWERS=0 for multi-gpu support via EGL
I0327 10:15:29.053588 3052 Simulator.cpp:64] Loaded.
*** Check failure stack trace: ***
I0327 10:15:29.054833 3052 SceneGraph.h:92] Created DrawableGroup:
F0327 10:15:29.054863 3052 WindowlessContext.cpp:232] Check failed: device == 0 (1 vs. 0) glX context does not support multiple GPUs. Please compile with BUILD_GUI_VIEWERS=0 for multi-gpu support via EGL
*** Check failure stack trace: ***
Traceback (most recent call last):
File "train.py", line 370, in <module>
run(model, Dataset, log_path, plotter, CHECKPOINT_tempfile)
File "train.py", line 265, in run
epoch, train_data_loader, model, log_path, plotter, opts
File "train.py", line 93, in train
iter_data_loader, isval=False, num_steps=opts.num_accumulations
File "/home/phong/data/Work/Paper3/Code/synsin/models/base_model.py", line 108, in __call__
t_losses, output_images = self.model(next(dataloader))
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/phong/data/Work/Paper3/Code/synsin/data/habitat_data.py", line 119, in __getitem__
self.__restart__()
File "/home/phong/data/Work/Paper3/Code/synsin/data/habitat_data.py", line 42, in __restart__
seed=self.worker_id + self.seed,
File "/home/phong/data/Work/Paper3/Code/synsin/data/create_rgb_dataset.py", line 189, in __init__
multiprocessing_start_method="forkserver",
File "/home/phong/data/Work/Paper3/Libraries/habitat-api/habitat/core/vector_env.py", line 135, in __init__
read_fn() for read_fn in self._connection_read_fns
File "/home/phong/data/Work/Paper3/Libraries/habitat-api/habitat/core/vector_env.py", line 135, in <listcomp>
read_fn() for read_fn in self._connection_read_fns
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Exception ignored in: <bound method VectorEnv.__del__ of <habitat.core.vector_env.VectorEnv object at 0x7fea87773ef0>>
Traceback (most recent call last):
File "/home/phong/data/Work/Paper3/Libraries/habitat-api/habitat/core/vector_env.py", line 468, in __del__
self.close()
File "/home/phong/data/Work/Paper3/Libraries/habitat-api/habitat/core/vector_env.py", line 350, in close
write_fn((CLOSE_COMMAND, None))
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/home/phong/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
I guess this error comes from the habitat-sim and it is not coming from this repo synsin.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.