3D reconstruction with neural networks using Tensorflow. See link for Video (https://www.youtube.com/watch?v=iI6ZMST8Ri0)

Shell 5.18% Python 94.82%

tensorflow 3d-reconstruction deep-learning ml neural-network python computer-vision convolutional-neural-networks recurrent-neural-networks voxel

3d-reconstruction-with-neural-networks's Introduction

3D reconstruction with Neural Networks

Note: This repo is no longer under development.

Introduction

This is work that I did as part of my Senior thesis at Princeton University. It is an implementation in Tensorflow of the network described by Choy et al in 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. The project is a neural network capable of performing 3D reconstruction using a variable number of images.

Demonstration

See the video below to see the network in action over a period of 40 epochs. The more red a voxel is the more certain the network is of its prediction at that position.

Getting Started

The network was trained using an AWS EC2 p2.xlarge instance.

Prerequisite

The projects make use of the several python packages. It is possible install these packages using pip. Use the following command to install the list packages in the requirements.txt.

pip3 install -r requirements.txt

It is possible to install tensorflow using pip as shown above but if you are having issues installing tensorflow on your specific platform follow the instructions here.

Usage

You can use preexisting shell scripts to make some of the tasks easier. For example

Setup

To start training the network, one must first setup the network using a shell script. This creates folders for the data, models and a JSON file to store the parameters of the network being trained.

sh scripts/setup_dir.sh

After setting up the dirs, we use a shell script to download the renders and low dimensional voxel models used by choy et al to train the their network. We then preprocesses the dataset by serializing the data to numpy npy files for easy loading and manipulation.

sh scripts/preprocess_dataset.sh

Training

After preparing the dataset for training, you can start training the network with the following shell script

sh scripts/train.sh

3d-reconstruction-with-neural-networks's People

Contributors

Stargazers

Watchers

3d-reconstruction-with-neural-networks's Issues

How is the performance?

Hi,

I found this while I was looking for a tensorflow implementation of the 3DR2N2. This implementation looks very cool, and code is easy to follow. I just wonder how is the performance compared to the release from the 3DR2N2 author.

Also, would you like to share several pre-trained snapshots? That would be also greatly useful. Thanks!

Hello, I am also interested in it.

Hi, sorry to bother you, I am also interested in using deep learning to do 3D-reconstruction works, recently I am reading the code wrote by Christophe, but I am not familiar with theano, then I saw you guys rebuilt it with tensorflow which I am more familiar with, so have you finished it yet? May you have a good day!

ImportError: No module named lib

There is some problem for the instruction.
The helper script "setup_dir.sh" doesn't work.
When running it accordingly, it returns
File "<string>", line 4, in <module> ImportError: No module named lib

Dataset Size in Params.JSON for Training

I try to reproduce the result from Michael's paper.

On his paper page 27, it looks like the training conducted with these parameter;

Batch Size: 16
Epoch: 40
(Max) Iteration: 30,000

With this parameter, the dataset size should be around 12,000.

Anybody or @micmelesse train the RNN with these parameters?

Error in Training Setting, Cannot infer num from shape (?, ?, 12192, 127)

I try to train the network by bash scripts/train.sh command, but I have an error during tensor initialization.

Error Message

Already up-to-date.
Using TensorFlow backend.
{'INITIALIZER': 'XAVIER', 'TIME_STEP_COUNT': 'RANDOM', 'SHUFFLE_IMAGE_SEQUENCE': 5, 'ENCODER_MODE': 'RESIDUAL'}
encoder
Traceback (most recent call last):
  File "run.py", line 45, in <module>
    net = network.Network()
  File "/3D-reconstruction-with-Neural-Networks/lib/network.py", line 68, in __init__
    en = encoder.Residual_Encoder(X_preprocessed)
  File "/3D-reconstruction-with-Neural-Networks/lib/encoder.py", line 217, in __init__
    sequence, 3, feature_map_count[0], K_1=7, K_2=3, K_3=0, initializer=init)
  File "/3D-reconstruction-with-Neural-Networks/lib/encoder.py", line 135, in block_residual_encoder
    out_featuremap_count, K=K_1, D=D, initializer=init)
  File "/3D-reconstruction-with-Neural-Networks/lib/encoder.py", line 46, in conv_sequence
    tf.unstack(feature_map_1, axis=1), axis=2)
  File "/anaconda2/envs/tdreconnn/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1091, in unstack
    raise ValueError("Cannot infer num from shape %s" % value_shape)
ValueError: Cannot infer num from shape (?, ?, 12192, 127)

For the information, this is my params.json file:

{
    "DIRS": {        
        "values":"/3D-reconstruction-with-Neural-Networks/process_folder",
        "DATA_PREPROCESSED": "/3D-reconstruction-with-Neural-Networks/process_folder/process",
        "MODELS_LOCAL": "/3D-reconstruction-with-Neural-Networks/process_folder/m_l",
        "MODELS_REMOTE": "/3D-reconstruction-with-Neural-Networks/process_folder/m_r"},
    "TRAIN": {
        "INITIALIZER": "XAVIER",
        "TIME_STEP_COUNT": "RANDOM",
        "SHUFFLE_IMAGE_SEQUENCE": 5,
        "ENCODER_MODE": "RESIDUAL"
    },
    "VIS": {
        "IMAGE_COUNT": 3,
        "KERNELS": "20",
        "FEATURE_MAPS": "20"
    }
    
}

how to test

hello！ Sorry to bother you. I see test.sh in scripts but there is not tester.py. Can you provide tester.py if possible? Thanks a lot

Error in Training Setting, Cannot infer num from shape (?, ?, 12192, 127)

What is way to initialize the tensor in params.json file?

I try to train the network by bash scripts/train.sh command, but I have an error during tensor initialization.

Error Message

Already up-to-date.
Using TensorFlow backend.
{'INITIALIZER': 'XAVIER', 'TIME_STEP_COUNT': 'RANDOM', 'SHUFFLE_IMAGE_SEQUENCE': 5, 'ENCODER_MODE': 'RESIDUAL'}
encoder
Traceback (most recent call last):
  File "run.py", line 45, in <module>
    net = network.Network()
  File "/3D-reconstruction-with-Neural-Networks/lib/network.py", line 68, in __init__
    en = encoder.Residual_Encoder(X_preprocessed)
  File "/3D-reconstruction-with-Neural-Networks/lib/encoder.py", line 217, in __init__
    sequence, 3, feature_map_count[0], K_1=7, K_2=3, K_3=0, initializer=init)
  File "/3D-reconstruction-with-Neural-Networks/lib/encoder.py", line 135, in block_residual_encoder
    out_featuremap_count, K=K_1, D=D, initializer=init)
  File "/3D-reconstruction-with-Neural-Networks/lib/encoder.py", line 46, in conv_sequence
    tf.unstack(feature_map_1, axis=1), axis=2)
  File "/anaconda2/envs/tdreconnn/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1091, in unstack
    raise ValueError("Cannot infer num from shape %s" % value_shape)
ValueError: Cannot infer num from shape (?, ?, 12192, 127)

For the information, this is my params.json file:

{
    "DIRS": {        
        "values":"/3D-reconstruction-with-Neural-Networks/process_folder",
        "DATA_PREPROCESSED": "/3D-reconstruction-with-Neural-Networks/process_folder/process",
        "MODELS_LOCAL": "/3D-reconstruction-with-Neural-Networks/process_folder/m_l",
        "MODELS_REMOTE": "/3D-reconstruction-with-Neural-Networks/process_folder/m_r"},
    "TRAIN": {
        "INITIALIZER": "XAVIER",
        "TIME_STEP_COUNT": "RANDOM",
        "SHUFFLE_IMAGE_SEQUENCE": 5,
        "ENCODER_MODE": "RESIDUAL"
    },
    "VIS": {
        "IMAGE_COUNT": 3,
        "KERNELS": "20",
        "FEATURE_MAPS": "20"
    }
    
}

Provide more documentation

Hi,

we are interested in building something based on your project in out EmTech lab,
can you provide in your README.md a section on how to call your projects API to convert an image to a voxel-set?

Currently we are using the original 3D-R2N2 project and it doesn't provide good results so we're interested in your project.

Thanks!
Sam

micmelesse / 3d-reconstruction-with-neural-networks Goto Github PK