It seems that overflow occurs when building HLG on large language model. The apra

Exception when building HLG with large language model about icefall HOT 11 OPEN

drawfish commented on June 20, 2024

Exception when building HLG with large language model

from icefall.

Comments (11)

danpovey commented on June 20, 2024

Debugging this will require getting it in gdb and looking at the symbols on the stack. We need to trace back where the negative number came from.
Would you mind doing that?
Compiling a debug version of the code might help, but I'm not sure whether it is necessary.

from icefall.

drawfish commented on June 20, 2024

Debugging this will require getting it in gdb and looking at the symbols on the stack. We need to trace back where the negative number came from. Would you mind doing that? Compiling a debug version of the code might help, but I'm not sure whether it is necessary.

OK, I will try.

from icefall.

drawfish commented on June 20, 2024

Debugging this will require getting it in gdb and looking at the symbols on the stack. We need to trace back where the negative number came from. Would you mind doing that? Compiling a debug version of the code might help, but I'm not sure whether it is necessary.

Hi Dan，below is the stack info, it seem that the negative part came from the function: ElementSize(), but I don't know how to get the return value of ElementSize() from core dump file.

[Current thread is 1 (Thread 0x7ff1a7b28700 (LWP 19704))]
(gdb) bt 10
#0  0x00007ff1a6a5e438 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ff1a6a6003a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ff0cc923499 in k2::Tensor::Tensor (this=0x7ffec394d540, type=k2::Dtype::kInt32Dtype, shape=..., region=..., byte_offset=-1163130320) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/tensor.cu:160
#3  0x00007ff0cc8c00f3 in k2::Array2<int>::Col (this=0x7ffec394d6d0, i=782959244) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/array.h:820
#4  0x00007ff0cc8963d6 in k2::IndexAxis0 (src=..., new2old=..., elem_indexes=0x7ffec394dc90) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/ragged_ops.cu:437
#5  0x00007ff0cc8970c5 in k2::Index (src=..., axis=0, indexes=..., elem_indexes=0x7ffec394dc90) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/ragged_ops.cu:565
#6  0x00007ff0cdd4262e in k2::Index<int> (src=..., axis=0, indexes=..., value_indexes_out=0x0) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/ragged_ops.h:1206
#7  0x00007ff0cdd982e2 in k2::Index<int> (src=..., indexes=..., remove_axis=false) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/ragged_ops_inl.h:453
#8  0x00007ff0cdd6985f in k2::RaggedAny::Index (this=0x55f3bffddff0, indexes=...) at /cfs/tmax_sata_01/sge/k2/k2/k2/python/csrc/torch/v2/ragged_any.cu:695
#9  0x00007ff0cdd4709f in pybind11::cpp_function::cpp_function<k2::RaggedAny, k2::RaggedAny, k2::RaggedAny&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, char const*>(k2::RaggedAny (k2::RaggedAny::*)(k2::RaggedAny&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, char const* const&)::{lambda(k2::RaggedAny*, k2::RaggedAny&)#1}::operator()(k2::RaggedAny*, k2::RaggedAny&) const () at /cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu2/_deps/pybind11-src/include/pybind11/pybind11.h:84
(More stack frames follow...)
(gdb) f 3
#3  0x00007ff0cc8c00f3 in k2::Array2<int>::Col (this=0x7ffec394d6d0, i=782959244) at /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/array.h:820
820         return Tensor(dtype_, shape, region_, byte_offset_ + (ElementSize() * i));
(gdb) l
815         NVTX_RANGE(K2_FUNC);
816         K2_CHECK_LT(static_cast<uint32_t>(i), static_cast<uint32_t>(dim1_));
817         std::vector<int32_t> dims = {dim0_};
818         std::vector<int32_t> strides = {elem_stride0_};
819         Shape shape(dims, strides);
820         return Tensor(dtype_, shape, region_, byte_offset_ + (ElementSize() * i));
821       }
822
823       // Note: const-ness is w.r.t. the metadata only.
824       T *Data() const {
(gdb) p byte_offset_
$1 = 0
(gdb) p i
$2 = 782959244

from icefall.

danpovey commented on June 20, 2024

ElementSize() should return 4, as a size_t. (However check that ElementSize() returns size_t in your code, your line numbers are different from mine). That should make a positive product as the 'int' should be promoted to size_t.
I can't find any plausible way that that number should become the byte_offset concerned, in the Tensor.
Are there any other variables, perhaps class members, that you can print out?

from icefall.

drawfish commented on June 20, 2024

ElementSize() should return 4, as a size_t. (However check that ElementSize() returns size_t in your code, your line numbers are different from mine). That should make a positive product as the 'int' should be promoted to size_t. I can't find any plausible way that that number should become the byte_offset concerned, in the Tensor. Are there any other variables, perhaps class members, that you can print out?

In my code, the return type of ElementSize() is int32_t, just the same as the code in k2 master branch here. Either int32_t or size_t is OK.
I think maybe we have found the cause of the problem.
As you said, ElementSize() return 4. Combined with the gdb debug info 'i=782959244, byte_offset_=0', then 'byte_offset_ + (ElementSize() * i) = 0+4*782959244 = 3131836976'.
But the type of the third parameter of the constructor of Tensor class is int32_t (see Tensor(Dtype type, const Shape &shape, RegionPtr region, int32_t byte_offset)) , which range from [-2147483648, 2147483647]. Casting 3131836976 into int32_t will be out of range.
Therefore, in order to solve this problem, do we need to change the parameter type to int64_t or size_t？
In addition, I'm not sure if there are similar problems elsewhere. More debugging may be required.

from icefall.

danpovey commented on June 20, 2024

Ah yes, you are right.
It think this issue has been fixed in most places already, but it seems to have somehow stayed in the codebase at this point.
Is there any chance you could make a pull request to fix this/

from icefall.

drawfish commented on June 20, 2024

OK，I will continue to generate the decode graph after modifying my local code to ensure that no other erros occur. If everything goes well, I will make a pull request.

from icefall.

drawfish commented on June 20, 2024

After fixing the code above and then rerun the code, another exception occurs:

[F] /cfs/tmax_sata_01/sge/k2/k2/k2/csrc/array_ops.cu:279:void k2::RowSplitsToRowIds(const k2::Array1<int>&, k2::Array1<int>*) Check failed: num_elems == row_splits[num_rows] (158681235 vs. 492176665)


[ Stack-Trace: ]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/libk2_log.so(k2::internal::GetStackTrace()+0x5b) [0x7ff08ae32a64]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/libk2context.so(k2::internal::Logger::~Logger()+0x44) [0x7ff08b2f8768]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/libk2context.so(k2::RowSplitsToRowIds(k2::Array1<int> const&, k2::Array1<int>*)+0x3ac) [0x7ff08b30783d]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/libk2context.so(+0x4305a2) [0x7ff08b4da5a2]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/libk2context.so(k2::Index(k2::RaggedShape&, int, k2::Array1<int> const&, k2::Array1<int>*)+0x1ce) [0x7ff08b4db081]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x21462e) [0x7ff08c98662e]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x26a2e2) [0x7ff08c9dc2e2]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x23b85f) [0x7ff08c9ad85f]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x21909f) [0x7ff08c98b09f]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x22936f) [0x7ff08c99b36f]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x2241b8) [0x7ff08c9961b8]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x21d5ca) [0x7ff08c98f5ca]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x21d6e7) [0x7ff08c98f6e7]
/cfs/tmax_sata_01/sge/k2/k2/build_debug_gpu/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x83b8a) [0x7ff08c7f5b8a]
python3(PyCFunction_Call+0x54) [0x5611d8b5bc64]
python3(_PyObject_MakeTpCall+0x31e) [0x5611d8b6594e]
python3(+0x1a7b8b) [0x5611d8bd6b8b]
python3(_PyEval_EvalFrameDefault+0x4dd3) [0x5611d8bf58d3]
python3(_PyEval_EvalCodeWithName+0x2c3) [0x5611d8bd4a33]
python3(_PyFunction_Vectorcall+0x378) [0x5611d8bd5e18]
python3(_PyEval_EvalFrameDefault+0x1824) [0x5611d8bf2324]
python3(_PyFunction_Vectorcall+0x1a6) [0x5611d8bd5c46]
python3(_PyEval_EvalFrameDefault+0x4dd3) [0x5611d8bf58d3]
python3(_PyEval_EvalCodeWithName+0x2c3) [0x5611d8bd4a33]
python3(_PyFunction_Vectorcall+0x378) [0x5611d8bd5e18]
python3(_PyEval_EvalFrameDefault+0x947) [0x5611d8bf1447]
python3(_PyFunction_Vectorcall+0x1a6) [0x5611d8bd5c46]
python3(_PyEval_EvalFrameDefault+0x947) [0x5611d8bf1447]
python3(_PyEval_EvalCodeWithName+0x2c3) [0x5611d8bd4a33]
python3(PyEval_EvalCodeEx+0x39) [0x5611d8bd5a99]
python3(PyEval_EvalCode+0x1b) [0x5611d8c7e1db]
python3(+0x24f273) [0x5611d8c7e273]
python3(+0x26cea3) [0x5611d8c9bea3]
python3(+0x272582) [0x5611d8ca1582]
python3(PyRun_SimpleFileExFlags+0x1b2) [0x5611d8ca1762]
python3(Py_RunMain+0x36d) [0x5611d8ca1cdd]
python3(Py_BytesMain+0x39) [0x5611d8ca1e99]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff165690840]
python3(+0x1dcd1d) [0x5611d8c0bd1d]

Traceback (most recent call last):
  File "./local/test_hlg.py", line 193, in <module>
    main()
  File "./local/test_hlg.py", line 181, in main
    HLG = compile_HLG(lang_dir, lm_dir, args.oov)
  File "./local/test_hlg.py", line 143, in compile_HLG
    LG = k2.remove_epsilon(LG)
  File "/cfs/tmax_sata_01/sge/k2/k2/k2/python/k2/fsa_algo.py", line 556, in remove_epsilon
    out_fsa = k2.utils.fsa_from_unary_function_ragged(fsa,
  File "/cfs/tmax_sata_01/sge/k2/k2/k2/python/k2/utils.py", line 540, in fsa_from_unary_function_ragged
    new_value = value.index(arc_map)
RuntimeError:
    Some bad things happened. Please read the above error messages and stack
    trace. If you are using Python, the following command may be helpful:

      gdb --args python /path/to/your/code.py

    (You can use `gdb` to debug the code. Please consider compiling
    a debug version of k2.).

    If you are unable to fix it, please open an issue at:

      https://github.com/k2-fsa/k2/issues/new

from icefall.

drawfish commented on June 20, 2024

Ah yes, you are right. It think this issue has been fixed in most places already, but it seems to have somehow stayed in the codebase at this point. Is there any chance you could make a pull request to fix this/
@danpovey
Hi, Dan. I think maybe the code in k2::IndexAxis0 needs to be totally modified. For example, I can be sure that 'Array2<int32_t> old_offsets, new_offsets' and 'Array1<int32_t> tot_sizes_out' also have int32_t out-of-range problems.
But I can't fully understand the code involved in this function. I need some help from your team.

from icefall.

danpovey commented on June 20, 2024

Hm. I'm sorry, that will be hard to fix. The only real solution would be to have int64_t versions of all the indexes, but we have chosen to keep all the indexes int32_t because the size of everything doubles if we make them all int64_t, and you can easily get OOM. Your graph may simply be too large. We support LM rescoring so you can always compile the HLG with a smaller LM and then rescore with a larger one. Can you make that PR though? That was a bug. In the longer term I want to work with decoding algorithms that are more memory-compact but can still use larger LMs, but that's in the future. Dan

…

On Mon, Nov 29, 2021 at 8:04 PM drawnfish ***@***.***> wrote: Ah yes, you are right. It think this issue has been fixed in most places already, but it seems to have somehow stayed in the codebase at this point. Is there any chance you could make a pull request to fix this/ @danpovey <https://github.com/danpovey> Hi, Dan. I think maybe the code in k2::IndexAxis0 <https://github.com/k2-fsa/k2/blob/8030001c9a002aa17e090a41de3f1146bdfe1e78/k2/csrc/ragged_ops.cu#L417> needs to be totally modified. For example, I can be sure that 'Array2<int32_t> old_offsets, new_offsets' <https://github.com/k2-fsa/k2/blob/8030001c9a002aa17e090a41de3f1146bdfe1e78/k2/csrc/ragged_ops.cu#L431> and 'Array1<int32_t> tot_sizes_out' <https://github.com/k2-fsa/k2/blob/8030001c9a002aa17e090a41de3f1146bdfe1e78/k2/csrc/ragged_ops.cu#L437> also have int32_t out-of-range problems. But I can't fully understand the code involved in this function. I need some help from your team. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#132 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLOYB5BFLSDKHCLKPNZ3UONT3VANCNFSM5I2CMNAQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

from icefall.

drawfish commented on June 20, 2024

Yh, at this stage, rescoring is the only solution.
I have made a PR to k2 repo.

from icefall.

Exception when building HLG with large language model about icefall HOT 11 OPEN

Comments (11)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent