hi, when I run the tdnn_lstm_ctc training of librispeech on Epoch 5, I got one error,please take a look,thanks
log of error:
2021-09-17 13:28:10,440 INFO [train.py:450] Epoch 5, batch 8620, batch avg loss 1.0633, total avg loss: 1.1221, batch size: 39
2021-09-17 13:28:22,302 INFO [train.py:450] Epoch 5, batch 8630, batch avg loss 1.1049, total avg loss: 1.1507, batch size: 41
2021-09-17 13:28:25,554 WARNING [cut.py:1694] To perform mix, energy must be non-zero and non-negative (got 0.0). MonoCut with id "845a0a69-f758-7b6a-90d8-ba99fa1795c4" will not be mixed in.
2021-09-17 13:28:36,682 INFO [train.py:450] Epoch 5, batch 8640, batch avg loss 1.1305, total avg loss: 1.1622, batch size: 40
2021-09-17 13:28:49,311 INFO [train.py:450] Epoch 5, batch 8650, batch avg loss 1.1228, total avg loss: 1.1774, batch size: 37
[F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void /workspace/k2/k2/csrc/intersect_dense.cu:863:lambda [](signed int)->void::operator()(signed int)->void block:[0,0,0], thread: [32,0,0] block:[0,0,0], thread: [33,0,0] block:[0,0,0], thread: [34,0,0] block:[0,0,0], thread: [35,0,0] block:[0,0,0], thread: [36,0,0] block:[0,0,0], thread: [37,0,0] block:[0,0,0], thread: [38,0,0] block:[0,0,0], thread: [39,0,0] block:[0,0,0], thread: [0,0,0] block:[0,0,0], thread: [1,0,0] block:[0,0,0], thread: [2,0,0] block:[0,0,0], thread: [3,0,0] block:[0,0,0], thread: [4,0,0] block:[0,0,0], thread: [5,0,0] block:[0,0,0], thread: [6,0,0] block:[0,0,0], thread: [7,0,0] block:[0,0,0], thread: [8,0,0] block:[0,0,0], thread: [9,0,0] block:[0,0,0], thread: [10,0,0] block:[0,0,0], thread: [11,0,0] block:[0,0,0], thread: [12,0,0] block:[0,0,0], thread: [13,0,0] block:[0,0,0], thread: [14,0,0] block:[0,0,0], thread: [15,0,0] block:[0,0,0], thread: [16,0,0] block:[0,0,0], thread: [17,0,0] block:[0,0,0], thread: [18,0,0] block:[0,0,0], thread: [19,0,0] block:[0,0,0], thread: [20,0,0] block:[0,0,0], thread: [21,0,0] block:[0,0,0], thread: [22,0,0] block:[0,0,0], thread: [23,0,0] block:[0,0,0], thread: [24,0,0] block:[0,0,0], thread: [25,0,0] block:[0,0,0], thread: [26,0,0] block:[0,0,0], thread: [27,0,0] block:[0,0,0], thread: [28,0,0] block:[0,0,0], thread: [29,0,0] block:[0,0,0], thread: [30,0,0] block:[0,0,0], thread: [31,0,0] Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_s/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [32,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [33,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [34,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [35,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [36,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [37,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [38,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [39,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [0,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [1,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [2,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [3,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [4,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [5,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [6,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [7,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [8,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [9,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [10,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [11,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [12,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [13,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [14,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [15,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [16,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [17,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [18,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [19,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [20,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [21,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [22,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [23,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [24,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [25,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [26,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [27,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [28,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [29,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [30,0,0] Assertion Some bad things happened
failed.
/workspace/k2/k2/csrc/intersect_dense.cu:863: lambda [](signed int)->void::operator()(signed int)->void: block: [0,0,0], thread: [31,0,0] Assertion Some bad things happened
failed.
tart || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0 nannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannan vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs vs nannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannan
[F] /workspace/k2/k2/csrc/array.h:341:T k2::Array1::operator const [with T = int; int32_t = int] Check failed: ret == cudaSuccess (710 vs. 0) Error: device-side assert triggered.
[ Stack-Trace: ]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2_log.so(k2::internal::GetStackTrace()+0x47) [0x7f8c456419f7]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::Array1::operator const+0xeb9) [0x7f8c4593c8e9]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::Renumbering::ComputeOld2New()+0x14e) [0x7f8c459377ee]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::Renumbering::ComputeNew2Old()+0x7f8) [0x7f8c45938f68]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::MultiGraphDenseIntersect::FormatOutput(k2::Array1, k2::Array1)+0x7ec) [0x7f8c45a9e47c]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/libk2context.so(k2::IntersectDense(k2::Raggedk2::Arc&, k2::DenseFsaVec&, k2::Array1 const*, float, k2::Raggedk2::Arc, k2::Array1, k2::Array1)+0x420) [0x7f8c45a8e900]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so(+0x65390) [0x7f8c4bad4390]
/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so(+0x1ee9e) [0x7f8c4ba8de9e]
python3(PyCFunction_Call+0x58) [0x55e5b8fb72d8]
python3(_PyObject_MakeTpCall+0x23c) [0x55e5b8fa6edc]
python3(_PyEval_EvalFrameDefault+0x11dd) [0x55e5b902f4ad]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(PyObject_CallObject+0x52) [0x55e5b9002982]
/opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so(THPFunction_apply(_object, _object*)+0x8fd) [0x7f8d427dc39d]
python3(PyCFunction_Call+0xe0) [0x55e5b8fb7360]
python3(_PyObject_MakeTpCall+0x23c) [0x55e5b8fa6edc]
python3(_PyEval_EvalFrameDefault+0x45a9) [0x55e5b9032879]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(+0x10425f) [0x55e5b8f6725f]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(+0x19aac9) [0x55e5b8ffdac9]
python3(PyObject_Call+0x414) [0x55e5b8fa7874]
python3(_PyEval_EvalFrameDefault+0x2088) [0x55e5b9030358]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyObject_Call_Prepend+0x181) [0x55e5b8ffe051]
python3(+0x19b3fa) [0x55e5b8ffe3fa]
python3(_PyObject_MakeTpCall+0x23c) [0x55e5b8fa6edc]
python3(_PyEval_EvalFrameDefault+0x475) [0x55e5b902e745]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(+0x103562) [0x55e5b8f66562]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(+0x103562) [0x55e5b8f66562]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(+0x103562) [0x55e5b8f66562]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(_PyFunction_Vectorcall+0x1e3) [0x55e5b8ffd593]
python3(+0x103562) [0x55e5b8f66562]
python3(_PyFunction_Vectorcall+0x10b) [0x55e5b8ffd4bb]
python3(+0x10425f) [0x55e5b8f6725f]
python3(_PyEval_EvalCodeWithName+0x300) [0x55e5b8ffc760]
python3(PyEval_EvalCode+0x23) [0x55e5b90914e3]
python3(+0x22e584) [0x55e5b9091584]
python3(+0x2547c4) [0x55e5b90b77c4]
python3(+0x115620) [0x55e5b8f78620]
Traceback (most recent call last):
File "./tdnn_lstm_ctc/train.py", line 616, in
main()
File "./tdnn_lstm_ctc/train.py", line 612, in main
run(rank=0, world_size=1, args=args)
File "./tdnn_lstm_ctc/train.py", line 575, in run
train_one_epoch(
File "./tdnn_lstm_ctc/train.py", line 424, in train_one_epoch
loss = compute_loss(
File "./tdnn_lstm_ctc/train.py", line 317, in compute_loss
loss = k2.ctc_loss(
File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/ctc_loss.py", line 136, in ctc_loss
return m(decoding_graph, dense_fsa_vec, target_lengths)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/ctc_loss.py", line 80, in forward
lattice = intersect_dense(decoding_graph, dense_fsa_vec,
File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/autograd.py", line 810, in intersect_dense
_IntersectDenseFunction.apply(a_fsas, b_fsas, out_fsa, output_beam,
File "/opt/conda/lib/python3.8/site-packages/k2-1.6.dev20210906+cuda11.1.torch1.8.0-py3.8-linux-x86_64.egg/k2/autograd.py", line 550, in forward
ragged_arc, arc_map_a, arc_map_b = _k2.intersect_dense(
RuntimeError: Some bad things happed.