Comments (6)
bruinxiong
commented on April 30, 2024
I follow this tensorflow/models#9706 to fix upper issue.
But I get another error:
(base) linxiong:nerfactor$ bash trainvali_run.sh
2021-08-16 16:55:04.319120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-08-16 16:55:04.389781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-16 16:55:04.390613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-16 16:55:04.391180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-16 16:55:04.391415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-08-16 16:55:04.392896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-08-16 16:55:04.394275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-08-16 16:55:04.394544: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-08-16 16:55:04.396130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-08-16 16:55:04.396979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-08-16 16:55:04.400638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-08-16 16:55:04.404467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2
2021-08-16 16:55:04.404805: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-08-16 16:55:04.410786: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3499910000 Hz
2021-08-16 16:55:04.411172: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbb2c000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-08-16 16:55:04.411197: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-08-16 16:55:04.640402: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562d39dcfa40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-08-16 16:55:04.640433: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-16 16:55:04.640441: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-16 16:55:04.640448: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2): GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-16 16:55:04.644029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-16 16:55:04.644549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-16 16:55:04.645066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-16 16:55:04.645111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-08-16 16:55:04.645124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-08-16 16:55:04.645142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-08-16 16:55:04.645155: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-08-16 16:55:04.645167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-08-16 16:55:04.645179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-08-16 16:55:04.645193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-08-16 16:55:04.648050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2
2021-08-16 16:55:04.648087: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-08-16 16:55:04.649975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-16 16:55:04.649994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 1 2
2021-08-16 16:55:04.650003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N Y Y
2021-08-16 16:55:04.650011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 1: Y N Y
2021-08-16 16:55:04.650019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 2: Y Y N
2021-08-16 16:55:04.654752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10372 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)
2021-08-16 16:55:04.655613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10093 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
2021-08-16 16:55:04.656427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10372 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:b3:00.0, compute capability: 6.1)
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0816 16:55:04.659288 140447264786240 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
[util/io] Output directory already exisits:
/home/linxiong/nerfactor/output/train/hotdog_nerf/lr5e-4
[util/io] Overwrite is off, so doing nothing
[trainvali] For results, see:
/home/linxiong/nerfactor/output/train/hotdog_nerf/lr5e-4
[datasets/nerf] Number of 'train' views: 100
Traceback (most recent call last):
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4675, in parallel_map_dataset
_result = pywrap_tfe.TFE_Py_FastPathExecute(
tensorflow.python.eager.core._FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linxiong/nerfactor/nerfactor/trainvali.py", line 350, in
app.run(main)
File "/home/linxiong/.local/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/linxiong/.local/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/home/linxiong/nerfactor/nerfactor/trainvali.py", line 93, in main
datapipe_train = dataset_train.build_pipeline(no_batch=no_batch)
File "../nerfactor/datasets/base.py", line 116, in build_pipeline
dataset = dataset.map(
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1623, in map
return ParallelMapDataset(
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4043, in init
variant_tensor = gen_dataset_ops.parallel_map_dataset(
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4684, in parallel_map_dataset
return parallel_map_dataset_eager_fallback(
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4761, in parallel_map_dataset_eager_fallback
_attr_Targuments, other_arguments = _execute.convert_to_mixed_eager_tensors(other_arguments, ctx)
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 283, in convert_to_mixed_eager_tensors
types = [t._datatype_enum() for t in v] # pylint: disable=protected-access
File "/home/linxiong/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 283, in
types = [t._datatype_enum() for t in v] # pylint: disable=protected-access
AttributeError: 'Tensor' object has no attribute '_datatype_enum'
from nerfactor.
xiumingzhang
commented on April 30, 2024
https://stackoverflow.com/questions/58479556/notimplementederror-cannot-convert-a-symbolic-tensor-2nd-target0-to-a-numpy seems to suggest this is a NumPy version mismatch issue.
Have you tried setting up a fresh Conda environment using the environment.yml of this repo?
from nerfactor.
bruinxiong
commented on April 30, 2024
https://stackoverflow.com/questions/58479556/notimplementederror-cannot-convert-a-symbolic-tensor-2nd-target0-to-a-numpy seems to suggest this is a NumPy version mismatch issue.
Have you tried setting up a fresh Conda environment using the
environment.ymlof this repo?
@xiumingzhang
Thank you for your instant reply. Yes, I try create new conda enviroment using the enviroment.yml. But I directly get the above error. The exactly location of code is in def _process_example_postcache of nerf.py file, also related with tf.stack op in def _sample_rays function.
from nerfactor.
bruinxiong
commented on April 30, 2024
@xiumingzhang After, I create new conda environment using the environment.yml. I get the same error as above.
(/home/linxiong/.conda/env/nerfactor) linxiong:nerfactor$ bash trainvali_run.sh
2021-08-18 09:34:57.981556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-08-18 09:34:58.054786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.055606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.056281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.056475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-08-18 09:34:58.057963: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-08-18 09:34:58.059375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-08-18 09:34:58.059635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-08-18 09:34:58.061163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-08-18 09:34:58.062120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-08-18 09:34:58.065684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-08-18 09:34:58.070558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2
2021-08-18 09:34:58.070872: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-08-18 09:34:58.076698: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3499910000 Hz
2021-08-18 09:34:58.077118: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe524000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-08-18 09:34:58.077148: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-08-18 09:34:58.284690: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5638da4f7360 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-08-18 09:34:58.284729: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-18 09:34:58.284738: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-18 09:34:58.284745: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2): GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-18 09:34:58.290014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.290549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.291044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.291097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-08-18 09:34:58.291113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-08-18 09:34:58.291126: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-08-18 09:34:58.291139: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-08-18 09:34:58.291152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-08-18 09:34:58.291170: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-08-18 09:34:58.291187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-08-18 09:34:58.293966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2
2021-08-18 09:34:58.294005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-08-18 09:34:58.295807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-18 09:34:58.295825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 1 2
2021-08-18 09:34:58.295834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N Y Y
2021-08-18 09:34:58.295843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 1: Y N Y
2021-08-18 09:34:58.295854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 2: Y Y N
2021-08-18 09:34:58.297826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10372 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)
2021-08-18 09:34:58.298675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10104 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
2021-08-18 09:34:58.299561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10372 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:b3:00.0, compute capability: 6.1)
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0818 09:34:58.302564 140628559923008 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
[trainvali] For results, see:
/home/linxiong/nerfactor/output/train/hotdog_nerf/lr5e-4
[datasets/nerf] Number of 'train' views: 100
Traceback (most recent call last):
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4680, in parallel_map_dataset
sloppy, "preserve_cardinality", preserve_cardinality)
tensorflow.python.eager.core._FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linxiong/nerfactor/nerfactor/trainvali.py", line 350, in
app.run(main)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/home/linxiong/nerfactor/nerfactor/trainvali.py", line 93, in main
datapipe_train = dataset_train.build_pipeline(no_batch=no_batch)
File "../nerfactor/datasets/base.py", line 119, in build_pipeline
num_parallel_calls=self.n_map_parallel_calls)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1628, in map
preserve_cardinality=True)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4050, in init
**self._flat_structure)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4688, in parallel_map_dataset
preserve_cardinality=preserve_cardinality, name=name, ctx=_ctx)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 4761, in parallel_map_dataset_eager_fallback
_attr_Targuments, other_arguments = _execute.convert_to_mixed_eager_tensors(other_arguments, ctx)
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 283, in convert_to_mixed_eager_tensors
types = [t._datatype_enum() for t in v] # pylint: disable=protected-access
File "/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 283, in
types = [t._datatype_enum() for t in v] # pylint: disable=protected-access
AttributeError: 'Tensor' object has no attribute '_datatype_enum'
from nerfactor.
xiumingzhang
commented on April 30, 2024
Thanks for the detailed tracestack. Looks like a TF "eager vs. graph mode"
issue. Our code was developed using the eager TF 2.2. Could you please make
sure that the TF that got used is indeed 2.2, by `print(tf.__version__)`?
Sometimes what you thought got used may not be what is actually used.
Thanks.
…On Wed, Aug 18, 2021 at 5:37 AM Xiong Lin ***@***.***> wrote:
@xiumingzhang <https://github.com/xiumingzhang> After, I create new conda
environment using the environment.yml. I get the same error as above.
(/home/linxiong/.conda/env/nerfactor) linxiong:nerfactor$ bash
trainvali_run.sh
2021-08-18 09:34:57.981556: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcuda.so.1
2021-08-18 09:34:58.054786: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with
properties:
pciBusID: 0000:17:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB
deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.055606: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with
properties:
pciBusID: 0000:65:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB
deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.056281: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with
properties:
pciBusID: 0000:b3:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB
deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.056475: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcudart.so.10.1
2021-08-18 09:34:58.057963: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcublas.so.10
2021-08-18 09:34:58.059375: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcufft.so.10
2021-08-18 09:34:58.059635: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcurand.so.10
2021-08-18 09:34:58.061163: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcusolver.so.10
2021-08-18 09:34:58.062120: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcusparse.so.10
2021-08-18 09:34:58.065684: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcudnn.so.7
2021-08-18 09:34:58.070558: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu
devices: 0, 1, 2
2021-08-18 09:34:58.070872: I
tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports
instructions that this TensorFlow binary was not compiled to use: AVX2
AVX512F FMA
2021-08-18 09:34:58.076698: I
tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency:
3499910000 Hz
2021-08-18 09:34:58.077118: I
tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe524000b20
initialized for platform Host (this does not guarantee that XLA will be
used). Devices:
2021-08-18 09:34:58.077148: I
tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0):
Host, Default Version
2021-08-18 09:34:58.284690: I
tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5638da4f7360
initialized for platform CUDA (this does not guarantee that XLA will be
used). Devices:
2021-08-18 09:34:58.284729: I
tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0):
GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-18 09:34:58.284738: I
tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1):
GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-18 09:34:58.284745: I
tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2):
GeForce GTX 1080 Ti, Compute Capability 6.1
2021-08-18 09:34:58.290014: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with
properties:
pciBusID: 0000:17:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB
deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.290549: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with
properties:
pciBusID: 0000:65:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB
deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.291044: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with
properties:
pciBusID: 0000:b3:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6325GHz coreCount: 28 deviceMemorySize: 10.92GiB
deviceMemoryBandwidth: 451.17GiB/s
2021-08-18 09:34:58.291097: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcudart.so.10.1
2021-08-18 09:34:58.291113: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcublas.so.10
2021-08-18 09:34:58.291126: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcufft.so.10
2021-08-18 09:34:58.291139: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcurand.so.10
2021-08-18 09:34:58.291152: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcusolver.so.10
2021-08-18 09:34:58.291170: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcusparse.so.10
2021-08-18 09:34:58.291187: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcudnn.so.7
2021-08-18 09:34:58.293966: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu
devices: 0, 1, 2
2021-08-18 09:34:58.294005: I
tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcudart.so.10.1
2021-08-18 09:34:58.295807: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect
StreamExecutor with strength 1 edge matrix:
2021-08-18 09:34:58.295825: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 1 2
2021-08-18 09:34:58.295834: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N Y Y
2021-08-18 09:34:58.295843: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 1: Y N Y
2021-08-18 09:34:58.295854: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 2: Y Y N
2021-08-18 09:34:58.297826: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow
device (/job:localhost/replica:0/task:0/device:GPU:0 with 10372 MB memory)
-> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id:
0000:17:00.0, compute capability: 6.1)
2021-08-18 09:34:58.298675: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow
device (/job:localhost/replica:0/task:0/device:GPU:1 with 10104 MB memory)
-> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id:
0000:65:00.0, compute capability: 6.1)
2021-08-18 09:34:58.299561: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow
device (/job:localhost/replica:0/task:0/device:GPU:2 with 10372 MB memory)
-> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id:
0000:b3:00.0, compute capability: 6.1)
INFO:tensorflow:Using MirroredStrategy with devices
('/job:localhost/replica:0/task:0/device:GPU:0',
'/job:localhost/replica:0/task:0/device:GPU:1',
'/job:localhost/replica:0/task:0/device:GPU:2')
I0818 09:34:58.302564 140628559923008 mirrored_strategy.py:500] Using
MirroredStrategy with devices
('/job:localhost/replica:0/task:0/device:GPU:0',
'/job:localhost/replica:0/task:0/device:GPU:1',
'/job:localhost/replica:0/task:0/device:GPU:2')
[trainvali] For results, see:
/home/linxiong/nerfactor/output/train/hotdog_nerf/lr5e-4
[datasets/nerf] Number of 'train' views: 100
Traceback (most recent call last):
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py",
line 4680, in parallel_map_dataset
sloppy, "preserve_cardinality", preserve_cardinality)
tensorflow.python.eager.core._FallbackException: This function does not
handle the case of the path where all inputs are not already EagerTensors.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/linxiong/nerfactor/nerfactor/trainvali.py", line 350, in
app.run(main)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/absl/app.py",
line 312, in run
_run_main(main, args)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/absl/app.py",
line 258, in _run_main
sys.exit(main(argv))
File "/home/linxiong/nerfactor/nerfactor/trainvali.py", line 93, in main
datapipe_train = dataset_train.build_pipeline(no_batch=no_batch)
File "../nerfactor/datasets/base.py", line 119, in build_pipeline
num_parallel_calls=self.n_map_parallel_calls)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py",
line 1628, in map
preserve_cardinality=True)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py",
line 4050, in *init*
**self._flat_structure)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py",
line 4688, in parallel_map_dataset
preserve_cardinality=preserve_cardinality, name=name, ctx=_ctx)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py",
line 4761, in parallel_map_dataset_eager_fallback
_attr_Targuments, other_arguments =
_execute.convert_to_mixed_eager_tensors(other_arguments, ctx)
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/eager/execute.py",
line 283, in convert_to_mixed_eager_tensors
types = [t._datatype_enum() for t in v] # pylint: disable=protected-access
File
"/home/linxiong/.conda/env/nerfactor/lib/python3.6/site-packages/tensorflow/python/eager/execute.py",
line 283, in
types = [t._datatype_enum() for t in v] # pylint: disable=protected-access
AttributeError: 'Tensor' object has no attribute '_datatype_enum'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACXICPQK64ROWYVQSDHRHL3T5N5N7ANCNFSM5CHMZGWA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
from nerfactor.
xiumingzhang
commented on April 30, 2024
Closing due to no response. Please reopen this if you still have problems.
from nerfactor.
Related Issues (20)
- How to calculate geometry buffers from MVS geometry ? HOT 1
- Rendering scripts HOT 1
- questions about hdrs. HOT 1
- Question about incompatible shapes(0,3) and (100,3) at II. Joint Optimization in Training, Validation, and Testing HOT 19
- gradient error in Joint Optimization HOT 6
- brdf_scale HOT 5
- MLPs wrong skip connection
- Crash at shape pre-training HOT 7
- OOM at II. Joint Optimization in Training, Validation, and Testing HOT 2
- Wrong NeRF and surface
- How long will it take to run the third part in the ./nerfactor HOT 1
- Rendering results are all white after training the vanilla NeRF in step1 HOT 1
- Shape error at II. Joint Optimization HOT 1
- Can we extract mesh from the system by marching cubes? HOT 1
- About create my own dataset
- Relighting Results Background Color HOT 1
- When I train vanilla nerf, there are countless threads.
- It is slow to render my own synthetic data, can we use gpu to render? HOT 1
- Shape pre-trained stage error HOT 2
- How about the light direction and the vier direction?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nerfactor.