How to create black-white mask, any repo or tutorial available

Regarding data generation, what are the points to pay attention to in the preparation of training data, or can you recommend some data sets for learning, thank you very much

Hello shunsuke, I'm here again :)

I've tried to run the code with multiple GPUs but it did not seem to work.

At first, I rewrote and ran codes in def init_net (net_util.py) , but just one gpu worked except very few increasing memory allocation in other GPUs. How do I use this code with multi GPU?

Second, I added the torch.nn.Dataparallel with 'netG' in line 50(train_shape.py) but some error related to 'mismatching gpu between model and input' had been occured. Have you met kind of this error when you tried to use multiple GPUs?

Thank you in advance, I'll wait your response!

Hi,

Thanks for the training code. May I know if we could train PIFu with BUFF data as this data is noisy and is not watertight?

Thanks,

Hi,

Is it possible to share chamfer distance code for BUFF?

Thanks,

You described the ground truth surface to be a "0.5 level set of a continuous 3D occupancy field". To my understanding, this sounds like the middle slice of the model you got from RenderPeople. If this is true, you would only have the ground truths for PIFu outputs with the same input z-depth. I seek your clarification on this matter.

I would like to additionally enquire if you have the code used to obtain the 3D occupancy field from the raw model files you got from RenderPeople.

I look forward to your response and learning from it. Thank you.

Hi, I was wondering how can I add hdri background to the rendered images, as was described in the paper?

Hey,
While generating the data using render_data.py with OpenGL 3.1.5 (as recommended), the following error was encountered:

Linker failure:
b"error: vertex shader lacks ``main'\n"`
3
Traceback (most recent call last):
File "/home/iiit/anaconda3/envs/ENV_PIFU_TRAIN/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/iiit/anaconda3/envs/ENV_PIFU_TRAIN/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/iiit/PIFu_Training/PIFu-master/apps/render_data.py", line 290, in
rndr = PRTRender(width=args.size, height=args.size, ms_rate=args.ms_rate, egl=args.egl)
File "/home/iiit/PIFu_Training/PIFu-master/lib/renderer/gl/prt_render.py", line 10, in init
CamRender.init(self, width, height, name, program_files=program_files, color_size=8, ms_rate=ms_rate, egl=egl)
File "/home/iiit/PIFu_Training/PIFu-master/lib/renderer/gl/cam_render.py", line 8, in init
Render.init(self, width, height, name, program_files, color_size, ms_rate=ms_rate, egl=egl)
File "/home/iiit/PIFu_Training/PIFu-master/lib/renderer/gl/render.py", line 42, in init
self.model_mat_unif = glGetUniformLocation(self.program, 'ModelMat')
File "/home/iiit/anaconda3/envs/ENV_PIFU_TRAIN/lib/python3.7/site-packages/OpenGL/latebind.py", line 63, in call
return self.wrapperFunction( self.baseFunction, *args, **named )
File "/home/iiit/anaconda3/envs/ENV_PIFU_TRAIN/lib/python3.7/site-packages/OpenGL/GL/VERSION/GL_2_0.py", line 430, in glGetUniformLocation
return baseOperation( program, name )
File "/home/iiit/anaconda3/envs/ENV_PIFU_TRAIN/lib/python3.7/site-packages/OpenGL/platform/baseplatform.py", line 415, in call
return self( *args, **named )
File "/home/iiit/anaconda3/envs/ENV_PIFU_TRAIN/lib/python3.7/site-packages/OpenGL/error.py", line 234, in glCheckError
baseOperation = baseOperation,
OpenGL.error.GLError: GLError(
err = 1282,
description = b'invalid operation',
baseOperation = glGetUniformLocation,
cArguments = (3, b'ModelMat\x00'),
result = -1
)

I've tried various versions of OpenGL but the issue is still remaining? Can you look into this.

Hello shunsuke, I have 3 questions:

1. After running the code, a colored image like the input is saved. Is this a render of the reconstructed 3D shape or is it just a copy of the input image?
2. The reconstructed objects I get are without material. Is there an option to save the color?
3. I cant seem to get the multiview part of the code to work. I set --num_views to the number of pictures I have but it gives errors with tensor sizes.

Great work and thank you in advance!

Thank you very much for your work. Now I can convert the mesh in the shapenet data to a watertight mesh through ManifoldPlus, but I don’t know how to set up the data set structure for training. If you can give some advice, I would appreciate it.

Notice that there are two pieces of code in PIFu/lib/renderer/gl/render.py

get_color(self, color_id=0): ...
get_z_value(self): ...

When I try to recover the point cloud from the depth value, the result shows that there is an error in the depth direction. If I translate the point cloud, it can match the original mesh.

I don't know if there's something wrong with my understanding of the camera model that makes the code incorrect, I hope to get your help, thanks!

out_all_z = rndr.get_z_value()

v = []
for ii in range(512):
    for jj in range(512):
        if out_all_z[ii, jj] != 1.0:
            X = (jj - 256) * cam.ortho_ratio
            Y = (256 - ii) * cam.ortho_ratio
            Z = (1 - out_all_z[ii, jj] * 2) * (cam.far - cam.near) / 2
            P = np.array([X, Y, Z]) / y_scale + vmed
            v.append(P)
v = np.array(v)

Dear Shunsukesaito,

Nice work. Thanks for releasing the code.
May I know how to compute chamfer loss on the output meshes w.r.t ground truth meshes? As both the meshes will be in different coordinate systems.

Thanks,

Hi,

I have my models scaled to unit box in meshlab. I have trained this data with sigma=5 in which i observed no change in the error. I have trained the same data with sigma 0.1 where I observed error decreasing.
However, marching cubes error still persists even after 15th epoch. May I know if any changes to code should be incorporated from changing meshes to unit box?

Thanks,

Hello shunsuke, I trained a netG using my own data, but the reconstructed meshes are not as good as the result of your released model. I'm not sure whether it is the data problem or did I miss something. Can you show some details like how is the MSE error going during your training process? Thank you so much!

Hello, I have questions about the way of evaluation with multi-views inputs.

After reading your paper, I couldn't understand the procedure of evaluation with multi-view inputs.
For training phase with multi-views input, we know corresponding pixel (xi, noted in your paper) between images from different view point for a single point X on the 3d model. However, for test phase, we don't know any 3d information or the relation between images. So, how do I pick the set of xi for a single voxel in 3d occupancy field in test phase?

I'm sorry to bother you if the answer could have been obtained from your paper.

Hi. I am having some trouble with my understanding. I hope you can enlighten and will truly appreciate it! You use spatial sampling with the ground truth mesh. Does this not mean you have a ground truth 3d occupancy field that is incomplete? For corresponding pixels without a groundtruth inside/outside prediction, will their feature vectors and z-values still be fed through PIFu?

I noticed in an earlier paragraph that you mentioned the use of bilinear sampling to obtain the feature vectors. How is the purpose of this sampling different from spatial sampling used with the ground truth meshes?

I also checked out the script you attached in the previous issue.
surface_points, _ = trimesh.sample.sample_surface(mesh, 4 * self.num_sample_inout)
sample_points = surface_points + np.random.normal(scale=self.opt.sigma, size=surface_points.shape)
Is this above spatial sampling? I do not see a direct connection to the spatial sampling described in the paper so I just want to confirm.

Thank you for you patience. I have picked this paper to try and learn as much as possible. Hopefully I'm not annoying you. I look forward to your reply.

@shunsukesaito Hi, thanks for the wonderful work. Can you tell me a way that the obj file which I generated using your code, I can convert it to a SMPL model. I can create an SMPL model with the arbitary parameters but I cannot texture it like PIFu . So any way in which I can use PIFu to generate a textured SMPL model

Dear Sir;
For single view Pifu training, I almost got the data format:
[image_tensor -> our images( 4-dim )
sample_tensor -> for mesh points,
calib_tensor ,
labels (1 for inside points zero for the outsides)]

But I didn't find any solution for multi-view format. Should I create an image_tensor parameter for each view? #31 here, u mentioned about transform matrixes. When we creating data with "apps.render_data" we just rotate the object, right?
Should I find the rotation matrices between the images and give them to the function?
Can you give more information about data format when you are available?
Sincerely

Hi,

Thank you for releasing the code, very nice.
Do you have an example how I can process multiple images and create one model?

Hi,

I'm trying to run the demo script. The mesh generation (obj file) works perfectly. How can I generate the texture or am I missing something?

The script also generates a png file, but that seems to be a screenshot?

Thank you

dear sir, how can i extended the codebase to multi-view setup.

• modify the data loader

• check SurfaceClassifier arguments

What else should I do?

Hello shunsuke, I tried the apps.render_data to process the 'rp_dennis_posed_004' sample, while the rendered images look like this:

I have no idea about this problem, can you give me some advice? Thank you!

Dear Sir, I don't know why but here the getitem function is called twice. (batch_size = 1 number_views = 1)

for epoch in range(start_epoch, opt.num_epoch): #number of epoch is 100
     for train_idx, train_data in enumerate(train_data_loader): 
         iter_start_time = time.time()
         break
     break

ex:
epoch=0;
train_idx=0
the first iteration calls the getitem
i have two index here: they are 352 and 97.
but the index returned : 352.
Is there any problem here or am I missing someting?

For non-pythoners who wants to try it out, how to install this stuff? Tried to pip install and got 'json not found', conda create... and got no resolved packages (json and others).
(Does it work on windows or only linux?)

It seems we can change the size of rendering image of apps.render_data by using -s augment like this:
python -m apps.render_data -s 1024 -i ./rp_dennis_posed_004_OBJ/ -o ./data_1024px/

And we can also change the size of loading image of apps.train_shape like this (I changed the batch size because of my small gpu memory):
python -m apps.train_shape --dataroot ./data_1024px/ --loadSize 1024 --random_flip --random_scale --random_trans --batch_size 1

I think this just changes the input size and won't affect the output size, but the output looks a bit bigger and head and foot are out of range. The output of 1st epoch looks like this

Do we have to change more options to use 1024px input? or is this a problem of network design (HourGlass, MLP etc.) ?

Thanks for your awesome work!

Here in the index() function definition (https://github.com/shunsukesaito/PIFu/blob/master/lib/geometry.py#L8), it is written that uv parameter is in range [0,1]. However, in pytorch documentation (https://pytorch.org/docs/stable/nn.functional.html#grid-sample) it is noted as [-1,1].

Is this a bug or typo in writing of the function definition?

very interesting idea in 3d reconstruction from single image.I want to apply it in my work, but I find the generated volume mesh doesn't smooth . Besides, I test the case you provide in code and that result is also coarse but not fine, especially face. Do you have any idea to improve the mesh quality? Thank very much.

If change the 'num_views' set in lib/options.py to a value bigger than 1, which stands for multi-view, the network code will have a bug, specificly in the line 114(maybe a little different because I add some comments in my file before) of lib/model/HGPIFuNet.py

pred = in_img[:, None].float() * self.surface_classifier(point_local_feat)

here the size of in_img[:, None].float() is [B*num_views, 1, N] and the size of self.surface_classifier(point_local_feat) is [B, 1, N]. So if num_views=1 as default there won't be this bug, but if set to more than 1, the bug of tensor dimension can't align will occur.

In advance, thank you for your amazing works and I'm very enjoying it!

Here's problem: when I ran the 'prt_util.py' file, I found there seemed no progress in the progress bar in console and only my CPU was used 100%. (I set up all the libraries you noted.) Also, you wrote "Warning: the following code becomes extremely slow without pyembree.", however, I couldn't find any usage of pyembree library in the entire project.

I don't know what's the next step. If you have a any idea about my problem, give me a comment!

Hi Shunsukesaito, amazing work! I want to get the depth map from the predict mesh (e.g. result_ryota.obj reconstructed from the sample imgs in the repo), but have trouble in reprojecting the mesh to (512*512) image plane. I use the Camera in lib.renderer.camera for projection but get a all-black image. Can you or anyone give me some tips on how to correctly set the camera and reproject the reconstructed mesh to depth map? Thanks a lot!

Hi
I am running the data generation code (render_data.py) on a slurm-managed cluster. I get the following error:

File "/.../PIFu/lib/renderer/gl/glcontext.py", line 110, in create_opengl_context
    egl.eglInitialize(egl_display, pointer(major), pointer(minor))
  File "/.../lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 415, in __call__
    return self( *args, **named )
  File "/.../lib/python3.8/site-packages/OpenGL/error.py", line 230, in glCheckError
    raise self._errorClass(
OpenGL.raw.EGL._errors.EGLError: EGLError(
        err = EGL_BAD_ACCESS,
        baseOperation = eglInitialize,
        cArguments = (
                <OpenGL._opaque.EGLDisplay_pointer object at 0x14b10fbd64c0>,
                <OpenGL.arrays.arraydatatype.LP_c_int object at 0x14b10cdf6f40>,
                <OpenGL.arrays.arraydatatype.LP_c_int object at 0x14b10cc44140>,
        ),
        result = 0
)

The code runs sometimes but mostly errors as above. Is there any known reason for this? Digging a bit into PyOpenGL, I found that it tries to access /dev/dri/renderD*. Is there a way to specify which egl_display to use? Could it be caused due to the GPUs being shared across jobs?

Other info

PyOpenGL ver - 3.1.5
Ubuntu 16.04.3
NVIDIA GTX 1080 Ti
NVIDIA Driver Version 440.95.01
CUDA Version: 10.2

[UPDATE]

The following code seems to resolve the issue:

from OpenGL import error
from OpenGL.EGL.EXT.device_base import *
from OpenGL.raw.EGL.EXT.platform_device import EGL_PLATFORM_DEVICE_EXT

def create_initialized_headless_egl_display():
  """Creates an initialized EGL display directly on a device."""
  for device in egl_get_devices():
    display = egl.eglGetPlatformDisplayEXT(EGL_PLATFORM_DEVICE_EXT, device, None)

    if display != egl.EGL_NO_DISPLAY and egl.eglGetError() == egl.EGL_SUCCESS:
      # `eglInitialize` may or may not raise an exception on failure depending
      # on how PyOpenGL is configured. We therefore catch a `GLError` and also
      # manually check the output of `eglGetError()` here.
      try:
        initialized = egl.eglInitialize(display, None, None)
      except error.GLError:
        pass
      else:
        if initialized == egl.EGL_TRUE and egl.eglGetError() == egl.EGL_SUCCESS:
          return display
  return egl.EGL_NO_DISPLAY

egl_display = create_initialized_headless_egl_display()
if egl_display == egl.EGL_NO_DISPLAY:
  raise ImportError('Cannot initialize a headless EGL display.')

Taken from here which was referenced in this issue

Please let me know if this can work as a permanent fix.

Do i need to change select_sampling_method for multi-view training?
thanks in advance.

Hi,
Loved the paper. Thanks for releasing the code.
I'm unable to do multi-view reconstruction using the pretrained models available. Can you help with the same?

Sir, I've created ./bounce/bounce0.txt and ./bounce/face.npy under {path_to_rp_dennis_posed_004_OBJ}.
Now, I try to train, but get RunTimeError:

(tokurEnv) hamit@hamit-MS-7B49:~/Softwares/environments/PIFu$ python -m apps.train_shape --dataroot /home/hamit/Softwares/environments/PIFu/tempImages --random_flip --random_scale --random_trans
/home/hamit/Softwares/environments/PIFu/lib/data/TrainDataset.py:102: UserWarning: loadtxt: Empty input file: "/home/hamit/Softwares/environments/PIFu/tempImages/val.txt"
  var_subjects = np.loadtxt(os.path.join(self.root, 'val.txt'), dtype=str)
train data size:  180
test data size:  360
initialize network with normal
Using Network:  hgpifu
Traceback (most recent call last):
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 12928) is killed by signal: Killed. 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 183, in <module>
    train(opt)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 90, in train
    for train_idx, train_data in enumerate(train_data_loader):
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 12928) exited unexpectedly

PyTorch version : 1.4.0

Dear Sir,
If possible can you check my results.
There is problem with my face 😄. Do i miss something?

and my inputs:

Hello! @shunsukesaito awesome work.

My question is the following: I have several images of the same person from different angles (the images were created at the same time), how can I interpolate one common object of better quality from these images? So that this object takes into account all projections. Is it possible to implement this feature using this solution?

I would be grateful if you can help me with this.
Thanks!

Using your source code on geometry we generated, we got this error during the training "Error cannot marching cubes". This error appears when the iteration is finished, it restarts the iteration again. Is this expected or something is wrong with our settings?

Thanks again for sharing your source.

Hi Shunsuke , Thanks for your great work!

I run some demo and found there will be some problems when reconstructing only upper body .It seems that it will judge the half body as the whole body.I'm wondering if there is some way to let PIFu fit the half body?

Thanks in advance!

Thanks for this impressive work! In the training process, there are two stages (shape module and color module). Do they refer to 'PIFu' and 'Texture Inference' mentioned in you paper respectively?

The data preprocessing part cannot be realized

Dear Sir;
in single view PIFu if batch size is 1 the following are the tensor dimensions:
calib_tensor: 1, 4, 4
image_tensor: 1, 3, 512, 512
label_tensor: 1, 1, 5000
sample_tensor : 1, 3, 5000

And we send feature tensor the Surface Classifier.
its size 1, 257, 5000 and SC returns y (1,1,5000)

but if we set number_views to 2
calib_tensor: 2, 4, 4
image_tensor: 2, 3, 512, 512
label_tensor: 1, 1, 5000
sample_tensor : 2, 3, 5000

i guess everything is ok so far but new feature tensor : 2,257,5000 and SC returns 1,1,5000 again. Is this our problem in the Surface Classifier right?

Seems like there is a slight restriction on input image size. When i input an image with size not equal to 512 x 512 (could be smaller, could be larger), i got the
the size of tensor a (blablabla) must match the size of tensor b (blablabla) at non singleton dimension blablabla

So is there a restriction for input shape?

With reference to Appendix I Experimental Setup, I would like to ask/clarify the following:

1. You mentioned the used of "second-order spherical harmonics" from HDRI Haven. I would like to ask how you were able to get them from the .hdr file format provided on HDRI Haven. And if you have provided the code to do so.

2. Based on my brief understanding of computer graphics, you have used PRT as well as the 163 "second-order spherical harmonics of indoor scene from HDRI Haven" to essentially light ("simulate light transport effects") the models from RenderPeople. Is this accurate?

3. If my understanding in 2 is accurate, what is the distribution of the seperate lighting conditions used for the 159120 training images. For example, did you use the "163 second-order spherical harmonics of indoor scene from HDRI Haven" for 163 models respectively and then PRT for the other 279 models? Or did you use use a mix of lighting conditions for a single model at different angles.

Thank you for reading. I look forward to your reply and learning from it.

Sorry to bother you, I want to know how many high-resolution photogrammetry scans from RenderPeople and BUFF needed to train this novel model. I try to train this model with only one photogrammetry scan from RenderPeople which is free to download and get a dissatisfied result.

Hi, I'm Hyunwoo.
I am very interested in your research.
Using the pretrained model you provided, I pulled the results using my own input image.
But I found one problem here.
I often found the problem of not properly distinguishing the front and back of a person.
Can you tell me if there is an option in the code to fix that element?

Hi, it would be great if you could document how to use this method in the multi-view setting and also provide the model needed. :)

Thank you, amazing work.

Hi Sir,
When I run the following script, should I get any output?
python -m apps.prt_util -i {path_to_rp_dennis_posed_004_OBJ}
My first time I got "Killed"
My second time I didn't get any error but progress bar remained zero.

gpu: gtx1070

Thanks in advance.