Question about the use of time_id about hypernerf HOT 6 OPEN

Honestly appreciate your project! Qusetions about the use_time mode of eval.py. When I changed the value of 'warp_embed_key' to 'time', I got this error It seems 'get_time_id' returns a tuple and a tuple cannot be divided by an int value. So how should I do to generate the dataset for using time_id. BTW, how can I get time_based images with fixed camera params ? Thanks a lot!

After studying HyperNeRF code, I realized how to set the camera position and the time ID of each frames by myself when rendering videos using trained model. Hope my experience may be helpful.

To set the camera position and viewing orientaion, just look the path './camera-paths/orbit-mild/' in your dataset. Those json files inside describe the position and the orientation of the test camera.

When it come to the time ID of each frames, it actually took me a lot of time to figure out. Under the default setting of the demo notebook HyperNeRF_Render_Video.ipynb, you can only render a video of a fixed time and with a rotating camera. The camera motion trail can be changed by rewriting the json file mentioned above, but how to change the time ID of each frames?

Finally, I found this line of code in HyperNeRF_Render_Video.ipynb.

results = []
for i in range(len(test_cameras)):
  print(f'Rendering frame {i+1}/{len(test_cameras)}')
  camera = test_cameras[i]
  batch = datasets.camera_to_rays(camera)
  batch['metadata'] = {
      'appearance': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
      'warp': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
  }

Here, when traversing all json files of camera infomation, batch['metadata'] means the time ID of the frame which is going to be rendered. In my experiment, it seemed that batch['metadata']['warp'] determines the time ID rather than batch['metadata']['appearance']. So, I simply added a line of code, like this,

results = []
for i in range(len(test_cameras)):
  print(f'Rendering frame {i+1}/{len(test_cameras)}')
  camera = test_cameras[i]
  batch = datasets.camera_to_rays(camera)
  batch['metadata'] = {
      'appearance': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
      'warp': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
  batch['metadata']['warp'] += i
  }

Then I successfully rendered a video with the change of time ID.
Note that the elements of batch['metadata']['warp'] should be int instead of float. You will meet an error otherwise.
My experiment config: ./configs/test_local.gin

from hypernerf.

@JunfengZou-123 I got the same error. did you solve this?

from hypernerf.

Honestly appreciate your project! Qusetions about the use_time mode of eval.py. When I changed the value of 'warp_embed_key' to 'time', I got this error It seems 'get_time_id' returns a tuple and a tuple cannot be divided by an int value. So how should I do to generate the dataset for using time_id. BTW, how can I get time_based images with fixed camera params ? Thanks a lot!

After studying HyperNeRF code, I realized how to set the camera position and the time ID of each frames by myself when rendering videos using trained model. Hope my experience may be helpful.

To set the camera position and viewing orientaion, just look the path './camera-paths/orbit-mild/' in your dataset. Those json files inside describe the position and the orientation of the test camera.

When it come to the time ID of each frames, it actually took me a lot of time to figure out. Under the default setting of the demo notebook HyperNeRF_Render_Video.ipynb, you can only render a video of a fixed time and with a rotating camera. The camera motion trail can be changed by rewriting the json file mentioned above, but how to change the time ID of each frames?

Finally, I found this line of code in HyperNeRF_Render_Video.ipynb.

results = []
for i in range(len(test_cameras)):
  print(f'Rendering frame {i+1}/{len(test_cameras)}')
  camera = test_cameras[i]
  batch = datasets.camera_to_rays(camera)
  batch['metadata'] = {
      'appearance': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
      'warp': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
  }

Here, when traversing all json files of camera infomation, batch['metadata'] means the time ID of the frame which is going to be rendered. In my experiment, it seemed that batch['metadata']['warp'] determines the time ID rather than batch['metadata']['appearance']. So, I simply added a line of code, like this,

results = []
for i in range(len(test_cameras)):
  print(f'Rendering frame {i+1}/{len(test_cameras)}')
  camera = test_cameras[i]
  batch = datasets.camera_to_rays(camera)
  batch['metadata'] = {
      'appearance': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
      'warp': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
  batch['metadata']['warp'] += i
  }

Then I successfully rendered a video with the change of time ID. Note that the elements of batch['metadata']['warp'] should be int instead of float. You will meet an error otherwise. My experiment config: ./configs/test_local.gin

同学你好，你的方法确实有了效果，但是我还是不能rerender我的原始video，视频只有前面部分动，而且是很快速的动，后面又处于静止了。
关于如何复现我的video，你这里有什么见解吗，感谢你的帮助，谢谢！

from hypernerf.

Honestly appreciate your project! Qusetions about the use_time mode of eval.py. When I changed the value of 'warp_embed_key' to 'time', I got this error It seems 'get_time_id' returns a tuple and a tuple cannot be divided by an int value. So how should I do to generate the dataset for using time_id. BTW, how can I get time_based images with fixed camera params ? Thanks a lot!

After studying HyperNeRF code, I realized how to set the camera position and the time ID of each frames by myself when rendering videos using trained model. Hope my experience may be helpful.
To set the camera position and viewing orientaion, just look the path './camera-paths/orbit-mild/' in your dataset. Those json files inside describe the position and the orientation of the test camera.
When it come to the time ID of each frames, it actually took me a lot of time to figure out. Under the default setting of the demo notebook HyperNeRF_Render_Video.ipynb, you can only render a video of a fixed time and with a rotating camera. The camera motion trail can be changed by rewriting the json file mentioned above, but how to change the time ID of each frames?
Finally, I found this line of code in HyperNeRF_Render_Video.ipynb.

results = []
for i in range(len(test_cameras)):
  print(f'Rendering frame {i+1}/{len(test_cameras)}')
  camera = test_cameras[i]
  batch = datasets.camera_to_rays(camera)
  batch['metadata'] = {
      'appearance': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
      'warp': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
  }

Here, when traversing all json files of camera infomation, batch['metadata'] means the time ID of the frame which is going to be rendered. In my experiment, it seemed that batch['metadata']['warp'] determines the time ID rather than batch['metadata']['appearance']. So, I simply added a line of code, like this,

results = []
for i in range(len(test_cameras)):
  print(f'Rendering frame {i+1}/{len(test_cameras)}')
  camera = test_cameras[i]
  batch = datasets.camera_to_rays(camera)
  batch['metadata'] = {
      'appearance': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
      'warp': jnp.zeros_like(batch['origins'][..., 0, jnp.newaxis], jnp.uint32),
  batch['metadata']['warp'] += i
  }

Then I successfully rendered a video with the change of time ID. Note that the elements of batch['metadata']['warp'] should be int instead of float. You will meet an error otherwise. My experiment config: ./configs/test_local.gin

同学你好，你的方法确实有了效果，但是我还是不能rerender我的原始video，视频只有前面部分动，而且是很快速的动，后面又处于静止了。 关于如何复现我的video，你这里有什么见解吗，感谢你的帮助，谢谢！

这是因为你的HyperNeRF的time_id只有某一些值是valid，比如其取值只能为0, 1, 2, 3, ..., 29。如果能取值的点很少，那么相邻两个time_id的时间跨度就会比较大，那么你渲染的视频就会很快速地动；如果你给定的time_id超过了其范围，那么就会渲染出黑色图案。如果你想知道time_id的取值范围到底有哪些，请在debug时在./hypernerf/modules.py中的以下代码中我标注的地方打断点：

@gin.configurable(denylist=['name'])
class GLOEmbed(nn.Module):
  """A GLO encoder module, which is just a thin wrapper around nn.Embed.

  Attributes:
    num_embeddings: The number of embeddings.
    features: The dimensions of each embedding.
    embedding_init: The initializer to use for each.
  """

  num_embeddings: int = gin.REQUIRED
  num_dims: int = gin.REQUIRED
  embedding_init: types.Activation = nn.initializers.uniform(scale=0.05)

  def setup(self):
    self.embed = nn.Embed(
        num_embeddings=self.num_embeddings,  # 这里num_embeddings代表可用的time_id个数
        features=self.num_dims,
        embedding_init=self.embedding_init)

  def __call__(self, inputs: jnp.ndarray) -> jnp.ndarray:
    """Method to get embeddings for specified indices.

    Args:
      inputs: The indices to fetch embeddings for.

    Returns:
      The embeddings corresponding to the indices provided.
    """
    if inputs.shape[-1] == 1:
      inputs = jnp.squeeze(inputs, axis=-1)

    return self.embed(inputs)  # 渲染视频时，启用Pycharm的debug模式，在这里打断点

注意，上面的inputs代表了你的time_id信息，而num_embeddings代表了可用的time_id个数，必须确保inputs<num_embeddings，才能渲染出非全黑的图片。
我拍了很多种video来训练HyperNeRF，但似乎每次的模型的num_embeddings都不一样，最多有八十多，最少只有十几。num_embeddings太少，严重影响我渲染的视频的流畅性。我目前还没有搞清楚在训练前如何自定义num_embeddings的值，如果你找到了修改的办法，也请麻烦告知我~

from hypernerf.

响我渲染的视频的流畅性。我目前还没有搞清楚在训练前如何自定义num_embeddings的值，如果你找到了修改的办法，也请麻烦告知我~
同学你好，感谢你的帮助！我按照你说的断点调试了，确实我的num_embeddings只有70多，但我本身有280帧，然后通过研究代码，最后看起来work的方法是在interp.py中line 92

 # if interval < 2 or interval % 2 != 0: 
 # raise ValueError('interval must be a positive even number.')  

首先把这两行注释掉，然后我用的config是hypernerf_interp_ds_2d.gin，在这里面把
InterpDataSource.interval = 1 #原本是4
这样我大概rerender出了我的原视频，我是原视频大概有298帧，但是在数据预处理的时候只有281个id，按照这样的处理方式重新训练网络，最后重新渲染出了281帧的原始视频。
谢谢你的帮助，我也在研究这个代码，有问题继续交流！

from hypernerf.

响我渲染的视频的流畅性。我目前还没有搞清楚在训练前如何自定义num_embeddings的值，如果你找到了修改的办法，也请麻烦告知我~
同学你好，感谢你的帮助！我按照你说的断点调试了，确实我的num_embeddings只有70多，但我本身有280帧，然后通过研究代码，最后看起来work的方法是在interp.py中line 92

 # if interval < 2 or interval % 2 != 0: 
 # raise ValueError('interval must be a positive even number.')  

首先把这两行注释掉，然后我用的config是hypernerf_interp_ds_2d.gin，在这里面把 InterpDataSource.interval = 1 #原本是4 这样我大概rerender出了我的原视频，我是原视频大概有298帧，但是在数据预处理的时候只有281个id，按照这样的处理方式重新训练网络，最后重新渲染出了281帧的原始视频。 谢谢你的帮助，我也在研究这个代码，有问题继续交流！

感谢你的分享~

from hypernerf.