RuntimeError: cuda runtime error (710) : device-side assert triggered at ...... about mlab HOT 2 CLOSED

--------------------------------------------
Traceback (most recent call last):
  File "train.py", line 32, in <module>
    train()
  File "/opt/gemfield/deepvac/deepvac/core/deepvac.py", line 671, in __call__
    self.process()
  File "/opt/gemfield/deepvac/deepvac/core/deepvac.py", line 664, in process
    self.train()
  File "/opt/gemfield/deepvac/deepvac/core/deepvac.py", line 617, in train
    self.doForward()
  File "train.py", line 22, in doForward
    output = self.config.net(self.config.sample, self.src_mask)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gemfield/hostpv/github/transformer/modules/model_transformer_encoder.py", line 45, in forward
    x = self.encoder(src)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2043, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self