Comments (3)
I found this error I was changing my tensorflow version to 2.2.0 like your version.
So I finished this error. But I have a new error like this.
./checkpoints/202107060321_ssd512_coco
total: 0%
0/100 [00:48<?, ?it/s]
epoch 1/100
training: 0%
0/2736 [00:48<?, ?it/s]
ResourceExhaustedError Traceback (most recent call last)
in
60 for i in tqdm(range(gen_train.num_batches), 'training', leave=False):
61 x, y_true = next(iterator_train)
---> 62 metric_values = step(x, y_true, training=True)
63 metric_util.update(metric_values, training=True)
64
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\def_function.py in call(self, *args, **kwds)
578 xla_context.Exit()
579 else:
--> 580 result = self._call(*args, **kwds)
581
582 if tracing_count == self._get_tracing_count():
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
642 # Lifting succeeded, so variables are initialized and we can run the
643 # stateless function.
--> 644 return self._stateless_fn(*args, **kwds)
645 else:
646 canon_args, canon_kwds = \
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in call(self, *args, **kwargs)
2418 with self._lock:
2419 graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 2420 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
2421
2422 @Property
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in _filtered_call(self, args, kwargs)
1663 if isinstance(t, (ops.Tensor,
1664 resource_variable_ops.BaseResourceVariable))),
-> 1665 self.captured_inputs)
1666
1667 def _call_flat(self, args, captured_inputs, cancellation_manager=None):
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1744 # No tape is watching; skip to running the function.
1745 return self._build_call_outputs(self._inference_function.call(
-> 1746 ctx, args, cancellation_manager=cancellation_manager))
1747 forward_backward = self._select_forward_and_backward_functions(
1748 args,
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
596 inputs=args,
597 attrs=attrs,
--> 598 ctx=ctx)
599 else:
600 outputs = execute.execute_with_cancellation(
c:\users\엄희찬\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
58 ctx.ensure_initialized()
59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
62 if name is not None:
ResourceExhaustedError: OOM when allocating tensor with shape[24,64,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node model/conv1_1/Conv2D (defined at :44) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[Op:__inference_step_7509]
Errors may have originated from an input operation.
Input Source operations connected to node model/conv1_1/Conv2D:
x (defined at :62)
Function call stack:
step
CPU i5-6600 3.3GHz
my GPU is NVIDIA GeForce GTX 1060 3GB
RAM 8GB
Is it problem of batch size? Since my GPU memory too low to overwork?
Thanks...
from ssd_detectors.
The TensorFlow dataset
API has changed in some ways, but the data pipeline I wrote is far from optimal anyway and most likely contains memory leaks.... In general, the tf1 branch can be an option if you don't want to improve it yourself.
OOM means out of memory. You can try to reduce the batch size, but I'm afraid 3 GB is too little for training. I had 11 GB at the beginning and 32 GB later. In the tf1 branch there is also an Adam optimizer that performs gradient accumulation...
from ssd_detectors.
thanks~!!
from ssd_detectors.
Related Issues (20)
- TBPP model arbitrary input shape HOT 1
- encode/decode error for tbpp
- DSOD Low mAP HOT 1
- Target length zero error
- about environment set (tf version?) HOT 2
- SL_end2end_predict.ipynb: Model dimensions don't match that in weights file. HOT 4
- Request to add .pkl files to repo HOT 4
- DSODSL Output tensor format HOT 1
- CRNN output format HOT 1
- How to convert to tflite?
- SL_end2end_predict.ipynb fails on converting to .py with necessary modifications. HOT 9
- Light architectures for object detection HOT 5
- fit_generator for ssd training & Check loss during training HOT 1
- training with own dataset resize issue HOT 1
- While training, got <UnknownError: AttributeError: 'NoneType' object has no attribute 'shape'> HOT 1
- links to download models not working HOT 2
- How to modify the size of anchor? HOT 4
- Metrics issue ? HOT 4
- Problem during with Crowd Human dataset HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ssd_detectors.