Git Product home page Git Product logo

Comments (2)

MY-CODE-1981 avatar MY-CODE-1981 commented on July 28, 2024

To be sure, I show the full text of the error.

(HOnnotate) initial@d3050eb4fa4f:~/workspace/HOnnotate/optimization$ python objectTrackingSingleFrame.py --seq 'test'
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/IPython/lib/pretty.py:91: DeprecationWarning: IPython.utils.signatures backport for Python 2 is deprecated in IPython 6, which only supports Python 3
from IPython.utils.signatures import signature
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:225: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.
Instructions for updating:
Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:

    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)
    ```
Additionally, to use tf.print in python 2.7, users must make sure to import
the following:

  `from __future__ import print_function`

/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/chumpy/ch.py:1203: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  want_out = 'out' in inspect.getargspec(func).args
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
  return matrix(data, dtype=dtype, copy=False)
/home/initial/workspace/HOnnotate/optimization/object/batch_object.py:65: RuntimeWarning: invalid value encountered in true_divide
  vn = mesh.vn / np.expand_dims(np.linalg.norm(mesh.vn, ord=2, axis=1), 1) # normalize to unit vec
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112: arg_min (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `argmin` instead
2024-01-14 10:50:00.623402: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2024-01-14 10:50:00.999041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:03:00.0
totalMemory: 15.90GiB freeMemory: 15.78GiB
2024-01-14 10:50:01.314176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725
pciBusID: 0000:02:00.0
totalMemory: 23.70GiB freeMemory: 22.76GiB
2024-01-14 10:50:01.314243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2024-01-14 10:50:28.016592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-01-14 10:50:28.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 
2024-01-14 10:50:28.016651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N N 
2024-01-14 10:50:28.016657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   N N 
2024-01-14 10:50:28.016832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15293 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1)
2024-01-14 10:50:28.033847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22064 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6)
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
maskPC for Image test/0/00002 is 0.132587
[Loading New frame ][0/00002]
2024-01-14 10:50:47.882958: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB.  Current allocation summary follows.
2024-01-14 10:50:47.883086: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 68, Chunks in use: 66. 17.0KiB allocated for chunks. 16.5KiB in use in bin. 788B client-requested in use in bin.
2024-01-14 10:50:47.883121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883151: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2024-01-14 10:50:47.883177: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883229: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 2, Chunks in use: 2. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin.
2024-01-14 10:50:47.883253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883281: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 2, Chunks in use: 2. 97.5KiB allocated for chunks. 97.5KiB in use in bin. 97.2KiB client-requested in use in bin.
2024-01-14 10:50:47.883309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 3, Chunks in use: 3. 301.8KiB allocated for chunks. 301.8KiB in use in bin. 301.4KiB client-requested in use in bin.
2024-01-14 10:50:47.883358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 0. 134.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 3, Chunks in use: 3. 1014.2KiB allocated for chunks. 1014.2KiB in use in bin. 900.0KiB client-requested in use in bin.
2024-01-14 10:50:47.883417: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 10, Chunks in use: 10. 8.79MiB allocated for chunks. 8.79MiB in use in bin. 8.79MiB client-requested in use in bin.
2024-01-14 10:50:47.883442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 4, Chunks in use: 3. 5.17MiB allocated for chunks. 4.00MiB in use in bin. 3.84MiB client-requested in use in bin.
2024-01-14 10:50:47.883468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 5.61MiB allocated for chunks. 5.61MiB in use in bin. 4.20MiB client-requested in use in bin.
2024-01-14 10:50:47.883495: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 5, Chunks in use: 3. 23.89MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin.
2024-01-14 10:50:47.883525: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 2, Chunks in use: 2. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 20.00MiB client-requested in use in bin.
2024-01-14 10:50:47.883550: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 1, Chunks in use: 0. 58.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883649: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883675: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 29.75GiB was 256.00MiB, Chunk State: 
2024-01-14 10:50:47.883700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7d0000000 of size 6291712
2024-01-14 10:50:47.883722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7d0600100 of size 60817152
2024-01-14 10:50:47.883743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e6000000 of size 12582912
2024-01-14 10:50:47.883763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7e6c00000 of size 4194560
2024-01-14 10:50:47.883784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7000100 of size 4194560
2024-01-14 10:50:47.883805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7400200 of size 12582400
2024-01-14 10:50:47.883825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea800000 of size 307200
2024-01-14 10:50:47.883845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b000 of size 256
2024-01-14 10:50:47.883865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b100 of size 4194560
2024-01-14 10:50:47.883885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b200 of size 256
2024-01-14 10:50:47.883905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b300 of size 256
2024-01-14 10:50:47.883924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b400 of size 256
2024-01-14 10:50:47.883944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b500 of size 256
2024-01-14 10:50:47.883963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b600 of size 256
2024-01-14 10:50:47.883982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b700 of size 256
2024-01-14 10:50:47.884001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b800 of size 256
2024-01-14 10:50:47.884021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b900 of size 256
2024-01-14 10:50:47.884040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4ba00 of size 921600
2024-01-14 10:50:47.884060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2ca00 of size 256
2024-01-14 10:50:47.884079: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cb00 of size 256
2024-01-14 10:50:47.884098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cc00 of size 921600
2024-01-14 10:50:47.884117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eae0dc00 of size 921600
2024-01-14 10:50:47.884136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeec00 of size 256
2024-01-14 10:50:47.884155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeed00 of size 256
2024-01-14 10:50:47.884174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeee00 of size 256
2024-01-14 10:50:47.884193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeef00 of size 256
2024-01-14 10:50:47.884213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef000 of size 256
2024-01-14 10:50:47.884233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eaeef100 of size 256
2024-01-14 10:50:47.884251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef200 of size 256
2024-01-14 10:50:47.884270: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef300 of size 256
2024-01-14 10:50:47.884289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef400 of size 256
2024-01-14 10:50:47.884308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef500 of size 256
2024-01-14 10:50:47.884327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef600 of size 256
2024-01-14 10:50:47.884346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef700 of size 256
2024-01-14 10:50:47.884365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef800 of size 256
2024-01-14 10:50:47.884384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef900 of size 256
2024-01-14 10:50:47.884403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefa00 of size 256
2024-01-14 10:50:47.884421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eaeefb00 of size 256
2024-01-14 10:50:47.884440: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefc00 of size 256
2024-01-14 10:50:47.884459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefd00 of size 921600
2024-01-14 10:50:47.884479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eafd0d00 of size 1228800
2024-01-14 10:50:47.884498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb0fcd00 of size 921600
2024-01-14 10:50:47.884517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eb1ddd00 of size 137984
2024-01-14 10:50:47.884536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb1ff800 of size 122112
2024-01-14 10:50:47.884556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eb21d500 of size 6171392
2024-01-14 10:50:47.884576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a00000 of size 9472
2024-01-14 10:50:47.884596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02500 of size 256
2024-01-14 10:50:47.884615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02600 of size 256
2024-01-14 10:50:47.884635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02700 of size 93440
2024-01-14 10:50:47.884654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19400 of size 256
2024-01-14 10:50:47.884674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19500 of size 49920
2024-01-14 10:50:47.884693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25800 of size 256
2024-01-14 10:50:47.884712: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25900 of size 256
2024-01-14 10:50:47.884731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25a00 of size 256
2024-01-14 10:50:47.884751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25b00 of size 1280
2024-01-14 10:50:47.884771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a26000 of size 9472
2024-01-14 10:50:47.884790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28500 of size 256
2024-01-14 10:50:47.884810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28600 of size 256
2024-01-14 10:50:47.884829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28700 of size 256
2024-01-14 10:50:47.884848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28800 of size 256
2024-01-14 10:50:47.884867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28900 of size 256
2024-01-14 10:50:47.884887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28a00 of size 256
2024-01-14 10:50:47.884906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28b00 of size 93440
2024-01-14 10:50:47.884925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f800 of size 256
2024-01-14 10:50:47.884944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f900 of size 49920
2024-01-14 10:50:47.884963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bc00 of size 256
2024-01-14 10:50:47.884983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bd00 of size 256
2024-01-14 10:50:47.885002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4be00 of size 256
2024-01-14 10:50:47.885021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bf00 of size 256
2024-01-14 10:50:47.885040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c000 of size 256
2024-01-14 10:50:47.885059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c100 of size 256
2024-01-14 10:50:47.885078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c200 of size 256
2024-01-14 10:50:47.885098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c300 of size 256
2024-01-14 10:50:47.885118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c400 of size 256
2024-01-14 10:50:47.885138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c500 of size 256
2024-01-14 10:50:47.885157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c600 of size 256
2024-01-14 10:50:47.885177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c700 of size 256
2024-01-14 10:50:47.885196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c800 of size 256
2024-01-14 10:50:47.885215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c900 of size 256
2024-01-14 10:50:47.885235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ca00 of size 256
2024-01-14 10:50:47.885253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cb00 of size 256
2024-01-14 10:50:47.885272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cc00 of size 256
2024-01-14 10:50:47.885292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cd00 of size 256
2024-01-14 10:50:47.885311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ce00 of size 256
2024-01-14 10:50:47.885330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cf00 of size 256
2024-01-14 10:50:47.885349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d000 of size 256
2024-01-14 10:50:47.885368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d100 of size 256
2024-01-14 10:50:47.885387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d200 of size 256
2024-01-14 10:50:47.885406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d300 of size 256
2024-01-14 10:50:47.885424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d400 of size 256
2024-01-14 10:50:47.885444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d500 of size 256
2024-01-14 10:50:47.885464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d600 of size 256
2024-01-14 10:50:47.885482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d700 of size 307200
2024-01-14 10:50:47.885502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a98700 of size 424192
2024-01-14 10:50:47.885523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2e00000 of size 2097152
2024-01-14 10:50:47.885543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3000000 of size 921600
2024-01-14 10:50:47.885563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f30e1000 of size 1260544
2024-01-14 10:50:47.885582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3214c00 of size 921600
2024-01-14 10:50:47.885602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f32f5c00 of size 1090560
2024-01-14 10:50:47.885622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3400000 of size 921600
2024-01-14 10:50:47.885640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f34e1000 of size 921600
2024-01-14 10:50:47.885659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f35c2000 of size 921600
2024-01-14 10:50:47.885679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f36a3000 of size 1843200
2024-01-14 10:50:47.885700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3865000 of size 3780608
2024-01-14 10:50:47.885720: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size: 
2024-01-14 10:50:47.885745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB
2024-01-14 10:50:47.885769: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2024-01-14 10:50:47.885791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9472 totalling 18.5KiB
2024-01-14 10:50:47.885813: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 49920 totalling 97.5KiB
2024-01-14 10:50:47.885836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 93440 totalling 182.5KiB
2024-01-14 10:50:47.885858: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 122112 totalling 119.2KiB
2024-01-14 10:50:47.885880: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 307200 totalling 600.0KiB
2024-01-14 10:50:47.885901: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 424192 totalling 414.2KiB
2024-01-14 10:50:47.885923: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 921600 totalling 8.79MiB
2024-01-14 10:50:47.885945: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1090560 totalling 1.04MiB
2024-01-14 10:50:47.885966: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1260544 totalling 1.20MiB
2024-01-14 10:50:47.885987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1843200 totalling 1.76MiB
2024-01-14 10:50:47.886008: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB
2024-01-14 10:50:47.886029: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3780608 totalling 3.61MiB
2024-01-14 10:50:47.886050: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4194560 totalling 8.00MiB
2024-01-14 10:50:47.886071: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6291712 totalling 6.00MiB
2024-01-14 10:50:47.886094: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582400 totalling 12.00MiB
2024-01-14 10:50:47.886116: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582912 totalling 12.00MiB
2024-01-14 10:50:47.886138: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 57.81MiB
2024-01-14 10:50:47.886165: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: 
Limit:                 16035977626
InUse:                    60618752
MaxInUse:                 64813312
NumAllocs:                     139
MaxAllocSize:             12582912

2024-01-14 10:50:47.886214: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *****_____________________________________________**********__***********xx*********____************
2024-01-14 10:50:47.901539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2024-01-14 10:50:48.043748: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e5fff700
2024-01-14 10:50:48.303976: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e5fff700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2)
2024-01-14 10:50:48.664465: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_egl.cpp:266] reinitialised framebuffer with size 320 x 480
2024-01-14 10:50:48.726655: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e57fe700
2024-01-14 10:50:48.749039: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e57fe700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2)
2024-01-14 10:50:48.751036: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_grad_egl.cpp:314] reinitialised framebuffer with size 320 x 480
Traceback (most recent call last):
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node map/while/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "objectTrackingSingleFrame.py", line 358, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 156, in objectTracker
    opti1.runOptimization(session, 1, {loadData:True})
  File "/home/initial/workspace/HOnnotate/optimization/ghope/utils.py", line 159, in timed
    result = method(*args, **kw)
  File "/home/initial/workspace/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization
    session.run(self.optOp, feed_dict=feedDict)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112)  = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Caused by op 'map/while/sub_1', defined at:
  File "objectTrackingSingleFrame.py", line 358, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 92, in objectTracker
    icpLoss = icp.getLoss(finalMesh.vUnClipped)
  File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 191, in getLoss
    loss = tf.map_fn(self.getLossPerFrame, (finalMesh, self.pcl2DAllViews, self.realObservs.seg, segColor), parallel_iterations=int(self.numViews), dtype=tf.float32)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 494, in map_fn
    maximum_iterations=n)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
    return_same_structure)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 483, in compute
    packed_fn_values = fn(packed_values)
  File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 112, in getLossPerFrame
    tf.arg_min(tf.reduce_sum(tf.abs(pclValid - verts), axis=2), dimension=1))  # N,
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub
    "Sub", x=x, y=y, name=name)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112)  = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

from honnotate.

Lxxiaodiao avatar Lxxiaodiao commented on July 28, 2024

To be sure, I show the full text of the error.

(HOnnotate) initial@d3050eb4fa4f:~/workspace/HOnnotate/optimization$ python objectTrackingSingleFrame.py --seq 'test' /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/IPython/lib/pretty.py:91: DeprecationWarning: IPython.utils.signatures backport for Python 2 is deprecated in IPython 6, which only supports Python 3 from IPython.utils.signatures import signature /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:225: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20. Instructions for updating: Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:

    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)
    ```
Additionally, to use tf.print in python 2.7, users must make sure to import
the following:

  `from __future__ import print_function`

/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/chumpy/ch.py:1203: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  want_out = 'out' in inspect.getargspec(func).args
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
  return matrix(data, dtype=dtype, copy=False)
/home/initial/workspace/HOnnotate/optimization/object/batch_object.py:65: RuntimeWarning: invalid value encountered in true_divide
  vn = mesh.vn / np.expand_dims(np.linalg.norm(mesh.vn, ord=2, axis=1), 1) # normalize to unit vec
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
  gaussKernel = np.expand_dims(gaussKernel, 3)
WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112: arg_min (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `argmin` instead
2024-01-14 10:50:00.623402: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2024-01-14 10:50:00.999041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:03:00.0
totalMemory: 15.90GiB freeMemory: 15.78GiB
2024-01-14 10:50:01.314176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725
pciBusID: 0000:02:00.0
totalMemory: 23.70GiB freeMemory: 22.76GiB
2024-01-14 10:50:01.314243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2024-01-14 10:50:28.016592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-01-14 10:50:28.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 
2024-01-14 10:50:28.016651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N N 
2024-01-14 10:50:28.016657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   N N 
2024-01-14 10:50:28.016832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15293 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1)
2024-01-14 10:50:28.033847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22064 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6)
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
maskPC for Image test/0/00002 is 0.132587
[Loading New frame ][0/00002]
2024-01-14 10:50:47.882958: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB.  Current allocation summary follows.
2024-01-14 10:50:47.883086: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 68, Chunks in use: 66. 17.0KiB allocated for chunks. 16.5KiB in use in bin. 788B client-requested in use in bin.
2024-01-14 10:50:47.883121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883151: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2024-01-14 10:50:47.883177: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883229: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 2, Chunks in use: 2. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin.
2024-01-14 10:50:47.883253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883281: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 2, Chunks in use: 2. 97.5KiB allocated for chunks. 97.5KiB in use in bin. 97.2KiB client-requested in use in bin.
2024-01-14 10:50:47.883309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 3, Chunks in use: 3. 301.8KiB allocated for chunks. 301.8KiB in use in bin. 301.4KiB client-requested in use in bin.
2024-01-14 10:50:47.883358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 0. 134.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 3, Chunks in use: 3. 1014.2KiB allocated for chunks. 1014.2KiB in use in bin. 900.0KiB client-requested in use in bin.
2024-01-14 10:50:47.883417: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 10, Chunks in use: 10. 8.79MiB allocated for chunks. 8.79MiB in use in bin. 8.79MiB client-requested in use in bin.
2024-01-14 10:50:47.883442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 4, Chunks in use: 3. 5.17MiB allocated for chunks. 4.00MiB in use in bin. 3.84MiB client-requested in use in bin.
2024-01-14 10:50:47.883468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 5.61MiB allocated for chunks. 5.61MiB in use in bin. 4.20MiB client-requested in use in bin.
2024-01-14 10:50:47.883495: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 5, Chunks in use: 3. 23.89MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin.
2024-01-14 10:50:47.883525: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 2, Chunks in use: 2. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 20.00MiB client-requested in use in bin.
2024-01-14 10:50:47.883550: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 1, Chunks in use: 0. 58.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883649: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883675: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 29.75GiB was 256.00MiB, Chunk State: 
2024-01-14 10:50:47.883700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7d0000000 of size 6291712
2024-01-14 10:50:47.883722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7d0600100 of size 60817152
2024-01-14 10:50:47.883743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e6000000 of size 12582912
2024-01-14 10:50:47.883763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7e6c00000 of size 4194560
2024-01-14 10:50:47.883784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7000100 of size 4194560
2024-01-14 10:50:47.883805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7400200 of size 12582400
2024-01-14 10:50:47.883825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea800000 of size 307200
2024-01-14 10:50:47.883845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b000 of size 256
2024-01-14 10:50:47.883865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b100 of size 4194560
2024-01-14 10:50:47.883885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b200 of size 256
2024-01-14 10:50:47.883905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b300 of size 256
2024-01-14 10:50:47.883924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b400 of size 256
2024-01-14 10:50:47.883944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b500 of size 256
2024-01-14 10:50:47.883963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b600 of size 256
2024-01-14 10:50:47.883982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b700 of size 256
2024-01-14 10:50:47.884001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b800 of size 256
2024-01-14 10:50:47.884021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b900 of size 256
2024-01-14 10:50:47.884040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4ba00 of size 921600
2024-01-14 10:50:47.884060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2ca00 of size 256
2024-01-14 10:50:47.884079: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cb00 of size 256
2024-01-14 10:50:47.884098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cc00 of size 921600
2024-01-14 10:50:47.884117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eae0dc00 of size 921600
2024-01-14 10:50:47.884136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeec00 of size 256
2024-01-14 10:50:47.884155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeed00 of size 256
2024-01-14 10:50:47.884174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeee00 of size 256
2024-01-14 10:50:47.884193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeef00 of size 256
2024-01-14 10:50:47.884213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef000 of size 256
2024-01-14 10:50:47.884233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eaeef100 of size 256
2024-01-14 10:50:47.884251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef200 of size 256
2024-01-14 10:50:47.884270: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef300 of size 256
2024-01-14 10:50:47.884289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef400 of size 256
2024-01-14 10:50:47.884308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef500 of size 256
2024-01-14 10:50:47.884327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef600 of size 256
2024-01-14 10:50:47.884346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef700 of size 256
2024-01-14 10:50:47.884365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef800 of size 256
2024-01-14 10:50:47.884384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef900 of size 256
2024-01-14 10:50:47.884403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefa00 of size 256
2024-01-14 10:50:47.884421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eaeefb00 of size 256
2024-01-14 10:50:47.884440: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefc00 of size 256
2024-01-14 10:50:47.884459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefd00 of size 921600
2024-01-14 10:50:47.884479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eafd0d00 of size 1228800
2024-01-14 10:50:47.884498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb0fcd00 of size 921600
2024-01-14 10:50:47.884517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eb1ddd00 of size 137984
2024-01-14 10:50:47.884536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb1ff800 of size 122112
2024-01-14 10:50:47.884556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7fc7eb21d500 of size 6171392
2024-01-14 10:50:47.884576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a00000 of size 9472
2024-01-14 10:50:47.884596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02500 of size 256
2024-01-14 10:50:47.884615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02600 of size 256
2024-01-14 10:50:47.884635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02700 of size 93440
2024-01-14 10:50:47.884654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19400 of size 256
2024-01-14 10:50:47.884674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19500 of size 49920
2024-01-14 10:50:47.884693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25800 of size 256
2024-01-14 10:50:47.884712: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25900 of size 256
2024-01-14 10:50:47.884731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25a00 of size 256
2024-01-14 10:50:47.884751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25b00 of size 1280
2024-01-14 10:50:47.884771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a26000 of size 9472
2024-01-14 10:50:47.884790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28500 of size 256
2024-01-14 10:50:47.884810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28600 of size 256
2024-01-14 10:50:47.884829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28700 of size 256
2024-01-14 10:50:47.884848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28800 of size 256
2024-01-14 10:50:47.884867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28900 of size 256
2024-01-14 10:50:47.884887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28a00 of size 256
2024-01-14 10:50:47.884906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28b00 of size 93440
2024-01-14 10:50:47.884925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f800 of size 256
2024-01-14 10:50:47.884944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f900 of size 49920
2024-01-14 10:50:47.884963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bc00 of size 256
2024-01-14 10:50:47.884983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bd00 of size 256
2024-01-14 10:50:47.885002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4be00 of size 256
2024-01-14 10:50:47.885021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bf00 of size 256
2024-01-14 10:50:47.885040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c000 of size 256
2024-01-14 10:50:47.885059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c100 of size 256
2024-01-14 10:50:47.885078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c200 of size 256
2024-01-14 10:50:47.885098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c300 of size 256
2024-01-14 10:50:47.885118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c400 of size 256
2024-01-14 10:50:47.885138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c500 of size 256
2024-01-14 10:50:47.885157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c600 of size 256
2024-01-14 10:50:47.885177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c700 of size 256
2024-01-14 10:50:47.885196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c800 of size 256
2024-01-14 10:50:47.885215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c900 of size 256
2024-01-14 10:50:47.885235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ca00 of size 256
2024-01-14 10:50:47.885253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cb00 of size 256
2024-01-14 10:50:47.885272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cc00 of size 256
2024-01-14 10:50:47.885292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cd00 of size 256
2024-01-14 10:50:47.885311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ce00 of size 256
2024-01-14 10:50:47.885330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cf00 of size 256
2024-01-14 10:50:47.885349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d000 of size 256
2024-01-14 10:50:47.885368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d100 of size 256
2024-01-14 10:50:47.885387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d200 of size 256
2024-01-14 10:50:47.885406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d300 of size 256
2024-01-14 10:50:47.885424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d400 of size 256
2024-01-14 10:50:47.885444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d500 of size 256
2024-01-14 10:50:47.885464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d600 of size 256
2024-01-14 10:50:47.885482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d700 of size 307200
2024-01-14 10:50:47.885502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a98700 of size 424192
2024-01-14 10:50:47.885523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2e00000 of size 2097152
2024-01-14 10:50:47.885543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3000000 of size 921600
2024-01-14 10:50:47.885563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f30e1000 of size 1260544
2024-01-14 10:50:47.885582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3214c00 of size 921600
2024-01-14 10:50:47.885602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f32f5c00 of size 1090560
2024-01-14 10:50:47.885622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3400000 of size 921600
2024-01-14 10:50:47.885640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f34e1000 of size 921600
2024-01-14 10:50:47.885659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f35c2000 of size 921600
2024-01-14 10:50:47.885679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f36a3000 of size 1843200
2024-01-14 10:50:47.885700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3865000 of size 3780608
2024-01-14 10:50:47.885720: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size: 
2024-01-14 10:50:47.885745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB
2024-01-14 10:50:47.885769: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2024-01-14 10:50:47.885791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9472 totalling 18.5KiB
2024-01-14 10:50:47.885813: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 49920 totalling 97.5KiB
2024-01-14 10:50:47.885836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 93440 totalling 182.5KiB
2024-01-14 10:50:47.885858: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 122112 totalling 119.2KiB
2024-01-14 10:50:47.885880: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 307200 totalling 600.0KiB
2024-01-14 10:50:47.885901: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 424192 totalling 414.2KiB
2024-01-14 10:50:47.885923: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 921600 totalling 8.79MiB
2024-01-14 10:50:47.885945: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1090560 totalling 1.04MiB
2024-01-14 10:50:47.885966: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1260544 totalling 1.20MiB
2024-01-14 10:50:47.885987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1843200 totalling 1.76MiB
2024-01-14 10:50:47.886008: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB
2024-01-14 10:50:47.886029: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3780608 totalling 3.61MiB
2024-01-14 10:50:47.886050: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4194560 totalling 8.00MiB
2024-01-14 10:50:47.886071: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6291712 totalling 6.00MiB
2024-01-14 10:50:47.886094: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582400 totalling 12.00MiB
2024-01-14 10:50:47.886116: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582912 totalling 12.00MiB
2024-01-14 10:50:47.886138: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 57.81MiB
2024-01-14 10:50:47.886165: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: 
Limit:                 16035977626
InUse:                    60618752
MaxInUse:                 64813312
NumAllocs:                     139
MaxAllocSize:             12582912

2024-01-14 10:50:47.886214: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *****_____________________________________________**********__***********xx*********____************
2024-01-14 10:50:47.901539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2024-01-14 10:50:48.043748: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e5fff700
2024-01-14 10:50:48.303976: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e5fff700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2)
2024-01-14 10:50:48.664465: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_egl.cpp:266] reinitialised framebuffer with size 320 x 480
2024-01-14 10:50:48.726655: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e57fe700
2024-01-14 10:50:48.749039: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e57fe700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2)
2024-01-14 10:50:48.751036: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_grad_egl.cpp:314] reinitialised framebuffer with size 320 x 480
Traceback (most recent call last):
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node map/while/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "objectTrackingSingleFrame.py", line 358, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 156, in objectTracker
    opti1.runOptimization(session, 1, {loadData:True})
  File "/home/initial/workspace/HOnnotate/optimization/ghope/utils.py", line 159, in timed
    result = method(*args, **kw)
  File "/home/initial/workspace/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization
    session.run(self.optOp, feed_dict=feedDict)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112)  = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Caused by op 'map/while/sub_1', defined at:
  File "objectTrackingSingleFrame.py", line 358, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 92, in objectTracker
    icpLoss = icp.getLoss(finalMesh.vUnClipped)
  File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 191, in getLoss
    loss = tf.map_fn(self.getLossPerFrame, (finalMesh, self.pcl2DAllViews, self.realObservs.seg, segColor), parallel_iterations=int(self.numViews), dtype=tf.float32)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 494, in map_fn
    maximum_iterations=n)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
    return_same_structure)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 483, in compute
    packed_fn_values = fn(packed_values)
  File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 112, in getLossPerFrame
    tf.arg_min(tf.reduce_sum(tf.abs(pclValid - verts), axis=2), dimension=1))  # N,
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub
    "Sub", x=x, y=y, name=name)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112)  = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Hello brother, I have the same problem as you. Have you solved it now?

from honnotate.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.