my result
2022-02-16 23:19:08.554267: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-02-16 23:19:08.997601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1251 MB memory: -> device: 0, name: GeForce MX450, pci bus id: 0000:01:00.0, compute capability: 7.5
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0216 23:19:09.045574 5088 mirrored_strategy.py:374] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 2000
I0216 23:19:09.045574 5088 config_util.py:552] Maybe overwriting train_steps: 2000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0216 23:19:09.045574 5088 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0216 23:19:09.061112 5088 deprecation.py:337] From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['Tensorflow\workspace\annotations\train.record']
I0216 23:19:09.076739 5088 dataset_builder.py:163] Reading unweighted datasets: ['Tensorflow\workspace\annotations\train.record']
INFO:tensorflow:Reading record datasets for input file: ['Tensorflow\workspace\annotations\train.record']
I0216 23:19:09.076739 5088 dataset_builder.py:80] Reading record datasets for input file: ['Tensorflow\workspace\annotations\train.record']
INFO:tensorflow:Number of filenames to read: 1
I0216 23:19:09.076739 5088 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0216 23:19:09.076739 5088 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)
instead. If sloppy execution is desired, use tf.data.Options.deterministic
.
W0216 23:19:09.076739 5088 deprecation.py:337] From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)
instead. If sloppy execution is desired, use tf.data.Options.deterministic
.
WARNING:tensorflow:From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map() W0216 23:19:09.092370 5088 deprecation.py:337] From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use
tf.data.Dataset.map()
WARNING:tensorflow:From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\dispatch.py:1082: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor
and use tf.sparse.to_dense
instead.
W0216 23:19:13.918504 5088 deprecation.py:337] From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\dispatch.py:1082: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor
and use tf.sparse.to_dense
instead.
WARNING:tensorflow:From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\dispatch.py:1082: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
seed2
arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0216 23:19:16.172085 5088 deprecation.py:337] From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\dispatch.py:1082: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
seed2
arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\dispatch.py:1082: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast
instead.
W0216 23:19:17.406628 5088 deprecation.py:337] From C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\dispatch.py:1082: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast
instead.
C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\keras\backend.py:450: UserWarning: tf.keras.backend.set_learning_phase
is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the training
argument of the __call__
method of your layer or model.
warnings.warn('tf.keras.backend.set_learning_phase
is deprecated and '
2022-02-16 23:19:32.862831: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8101
2022-02-16 23:19:34.137917: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.224891: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.397404: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.484402: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.587516: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.712935: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.733970: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.734360: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736050: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736125: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736274: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736298: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736359: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736375: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736442: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736458: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736515: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736526: W tensorflow/stream_executor/stream.cc:1260] attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736580: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736603: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736635: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736649: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736693: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2022-02-16 23:19:34.736709: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Attempting to perform BLAS operation using StreamExecutor without BLAS support
2022-02-16 23:19:34.736849: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.736881: W tensorflow/core/kernels/gpu_utils.cc:70] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'stream did not block host until done; was already in an error state'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2022-02-16 23:19:34.736946: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.737568: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.737629: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.738228: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.738286: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.739022: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.739079: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.739837: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.739895: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.750895: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.751451: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.761552: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.761640: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.762796: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.762893: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.764247: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.764356: I tensorflow/stream_executor/stream.cc:4442] [stream=00000211EEDF6A60,impl=00000211F66EE2C0] INTERNAL: stream did not block host until done; was already in an error state
2022-02-16 23:19:34.764458: E tensorflow/stream_executor/cuda/cuda_blas.cc:232] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
Traceback (most recent call last):
File "C:\Repos\tensorflow object detection\TFODCourse\Tensorflow\models\research\object_detection\model_main_tf2.py", line 115, in
tf.compat.v1.app.run()
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\platform\app.py", line 36, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\absl\app.py", line 312, in run
_run_main(main, args)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\absl\app.py", line 258, in _run_main
sys.exit(main(argv))
File "C:\Repos\tensorflow object detection\TFODCourse\Tensorflow\models\research\object_detection\model_main_tf2.py", line 106, in main
model_lib_v2.train_loop(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py", line 605, in train_loop
load_fine_tune_checkpoint(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py", line 400, in load_fine_tune_checkpoint
_ensure_model_is_built(model, input_dataset, unpad_groundtruth_tensors)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py", line 175, in _ensure_model_is_built
strategy.run(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1312, in run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 2888, in call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 676, in _call_for_each_replica
return mirrored_run.call_for_each_replica(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 82, in call_for_each_replica
return wrapped(args, kwargs)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\tensorflow\python\eager\execute.py", line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:
Detected at node 'Loss/MatMulGather/MatMul' defined at (most recent call last):
File "C:\Users\User\anaconda3\lib\threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "C:\Users\User\anaconda3\lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py", line 171, in _dummy_computation_fn
training_step=0)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\model_lib_v2.py", line 129, in _compute_losses_and_predictions_dicts
losses_dict = model.loss(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\meta_architectures\ssd_meta_arch.py", line 842, in loss
(batch_cls_targets, batch_cls_weights, batch_reg_targets,
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\meta_architectures\ssd_meta_arch.py", line 1066, in _assign_targets
if train_using_confidences:
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\meta_architectures\ssd_meta_arch.py", line 1083, in _assign_targets
groundtruth_weights_list)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\core\target_assigner.py", line 510, in batch_assign
for anchors, gt_boxes, gt_class_targets, gt_weights in zip(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\core\target_assigner.py", line 512, in batch_assign
(cls_targets, cls_weights,
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\core\target_assigner.py", line 202, in assign
reg_targets = self._create_regression_targets(anchors,
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\core\target_assigner.py", line 259, in _create_regression_targets
matched_gt_boxes = match.gather_based_on_match(
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\core\matcher.py", line 214, in gather_based_on_match
gathered_tensor = self._gather_op(input_tensor, gather_indices)
File "C:\Repos\tensorflow object detection\TFODCourse\tfod\lib\site-packages\object_detection-0.1-py3.9.egg\object_detection\utils\ops.py", line 1027, in matmul_gather_on_zeroth_axis
gathered_result_flattened = tf.matmul(indicator_matrix, params2d)
Node: 'Loss/MatMulGather/MatMul'
Attempting to perform BLAS operation using StreamExecutor without BLAS support
[[{{node Loss/MatMulGather/MatMul}}]] [Op:__inference__dummy_computation_fn_15081]