Comments (12)
Oh, please change
G["dummy"] = 1
G.dummy = 1
. Fixed in #134
from icefall.
Thanks!
from icefall.
For some reason I got Segmentation fault (core dumped)
that happens on next line
torch.save(G.as_dict(), params.lm_dir / "G_4_gram.pt")
from icefall.
For some reason I got Segmentation fault (core dumped) that happens on next line
Are there additional error logs?
from icefall.
No just simply
(base) audiodan@audiodan-MS-7A94:~/asr/icefall/egs/librispeech/ASR$ ./conformer_ctc/decode.py --exp-dir conformer_ctc/exp_500_att0.8 --lang-dir data/lang_bpe_500 --max-duration 30 --concatenate-cuts 0 --bucketing-sampler true --num-paths 1000 --epoch 5 --avg 1 --method attention-decoder --nbest-scale 0.5
2021-11-26 22:23:36,049 INFO [decode.py:549] Decoding started
2021-11-26 22:23:36,049 INFO [decode.py:550] {'subsampling_factor': 4, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'feature_dim': 80, 'nhead': 8, 'attention_dim': 512, 'num_decoder_layers': 6, 'search_beam': 20, 'output_beam': 8, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'env_info': {'k2-version': '1.10', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': 'fd5565d32ffa8274ff9700453b1e543f34343ed1', 'k2-git-date': 'Wed Nov 10 08:31:12 2021', 'lhotse-version': '0.12.0.dev+git.d5e7815.dirty', 'torch-cuda-available': True, 'torch-cuda-version': '11.2', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': 'b945223-dirty', 'icefall-git-date': 'Thu Nov 11 03:26:02 2021', 'icefall-path': '/home/audiodan/asr/icefall', 'k2-path': '/home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/home/audiodan/asr/lhotse/lhotse/init.py'}, 'epoch': 5, 'avg': 1, 'method': 'attention-decoder', 'num_paths': 1000, 'nbest_scale': 0.5, 'export': False, 'exp_dir': PosixPath('conformer_ctc/exp_500_att0.8'), 'lang_dir': PosixPath('data/lang_bpe_500'), 'lm_dir': PosixPath('data/lm'), 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 30, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 10}
2021-11-26 22:23:36,241 INFO [lexicon.py:176] Loading pre-compiled data/lang_bpe_500/Linv.pt
2021-11-26 22:23:36,273 INFO [decode.py:560] device: cpu
2021-11-26 22:23:37,856 INFO [decode.py:597] Loading G_4_gram.fst.txt
2021-11-26 22:23:37,856 WARNING [decode.py:598] It may take 8 minutes.
Segmentation fault (core dumped)
I've checked It's successfully executed G.dummy = 1
and it can cast G to a dictionary with G.as_dict()
the problem happens with torch.save()
for sure but I didn't investigate with gdb
. I'm curious if it's reproducible or not.
from icefall.
I just tested the fix and it works. Not sure why it does not work in your case.
from icefall.
I don't understand why torch.save(G.as_dict(), params.lm_dir / "G_4_gram.pt")
fails.
from icefall.
It should be possible to run it in gdb and get a stack trace from the segfault.
from icefall.
Thanks guys, I got this error in gdb
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:384
384 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
BTW I'm decoding with CPU.
from icefall.
After starting gdb
,
(gdb) catch throw
(gdb) run
When segfault occurs, I think you are able to do
(gdb) bt
to show the backtrace.
from icefall.
(gdb) bt
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:384
#1 0x00007fff712fb356 in k2::Array1<int> k2::Cat<int>(std::shared_ptr<k2::Context>, int, k2::Array1<int> const**) ()
from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/libk2context.so
#2 0x00007fff712e3b0c in k2::FsaVecToTensor(k2::Ragged<k2::Arc> const&) ()
from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/libk2context.so
#3 0x00007fff726e725b in ?? ()
from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so
#4 0x00007fff726cc7f7 in ?? ()
from /home/audiodan/anaconda3/lib/python3.8/site-packages/k2-1.10.dev20211112+cuda11.2.torch1.10.0a0-py3.8-linux-x86_64.egg/_k2.cpython-38-x86_64-linux-gnu.so
#5 0x00005555556a8348 in cfunction_call_varargs (kwargs=<optimized out>, args=<optimized out>, func=0x7fff7337a720)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:743
#6 PyCFunction_Call (func=0x7fff7337a720, args=<optimized out>, kwargs=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:773
#7 0x0000555555697dbc in _PyObject_MakeTpCall (callable=0x7fff7337a720, args=<optimized out>, nargs=<optimized out>, keywords=0x0)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:159
#8 0x0000555555723666 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fff6fc89fd8, callable=0x7fff7337a720)
at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:125
#9 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5cf0)
at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4963
#10 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3469
#11 0x00005555556ee36b in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:284
#12 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x5555592410b0, func=0x7fff70cdb940)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:411
#13 _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=0x5555592410b0, callable=0x7fff70cdb940)
at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:127
#14 method_vectorcall (method=<optimized out>, args=0x5555592410b8, nargsf=<optimized out>, kwnames=<optimized out>)
at /tmp/build/80754af9/python_1618343417471/work/Objects/classobject.c:60
#15 0x0000555555657a61 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x5555592410b8, callable=0x7fff6fd86380)
at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:127
#16 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5cf0)
at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4963
#17 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3469
#18 0x00005555556edfcb in function_code_fastcall (globals=<optimized out>, nargs=0, args=<optimized out>, co=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--<RET>
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:284
#19 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7ffff6e0f058, nargsf=<optimized out>, kwnames=<optimized out>)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:411
#20 0x0000555555698754 in PyVectorcall_Call (kwargs=0x7fff73388d80, tuple=0x7ffff6e0f040, callable=0x7fff6fc72c10)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:200
#21 PyObject_Call (callable=0x7fff6fc72c10, args=0x7ffff6e0f040, kwargs=0x7fff73388d80) at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:228
#22 0x00005555557210b6 in do_call_core (kwdict=0x7fff73388d80, callargs=0x7ffff6e0f040, func=0x7fff6fc72c10, tstate=<optimized out>)
at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:5010
#23 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3559
#24 0x00005555556ed821 in PyEval_EvalFrameEx (throwflag=0, f=0x7fff7020b800) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:741
#25 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>,
kwnames=<optimized out>, kwargs=0x55555594b790, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7fff73422b40, name=0x7ffff6d8c0f0,
qualname=0x7ffff6d8c0f0) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4298
#26 0x00005555556ee0a3 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x55555594b790, nargsf=<optimized out>, kwnames=<optimized out>)
at /tmp/build/80754af9/python_1618343417471/work/Objects/call.c:436
#27 0x00005555556575db in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55555594b790, callable=0x7fff6fc72ca0)
at /tmp/build/80754af9/python_1618343417471/work/Include/cpython/abstract.h:127
#28 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5cf0)
at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4963
#29 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:3500
#30 0x00005555556ed270 in PyEval_EvalFrameEx (throwflag=0, f=0x55555594b620) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:741
#31 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0,
kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4298
#32 0x0000555555782543 in PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:4327
#33 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/ceval.c:718
#34 0x00005555557825e4 in run_eval_code_obj (co=0x7ffff6c6d500, globals=0x7ffff6dbf1c0, locals=0x7ffff6dbf1c0)
at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:1165
#35 0x00005555557a8854 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff6dbf1c0, locals=0x7ffff6dbf1c0, flags=<optimized out>,
arena=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:1187
#36 0x0000555555669390 in pyrun_file (fp=0x5555558f4130, filename=0x7ffff6d84210, start=<optimized out>, globals=0x7ffff6dbf1c0, locals=0x7ffff6dbf1c0, closeit=1,
flags=0x7fffffffdad8) at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:1084
#37 0x000055555566c0d2 in pyrun_simple_file (flags=0x7fffffffdad8, closeit=1, filename=0x7ffff6d84210, fp=0x5555558f4130)
--Type <RET> for more, q to quit, c to continue without paging--<RET>
at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:439
#38 PyRun_SimpleFileExFlags (fp=0x5555558f4130, filename=<optimized out>, closeit=1, flags=0x7fffffffdad8)
at /tmp/build/80754af9/python_1618343417471/work/Python/pythonrun.c:472
#39 0x000055555566cbf0 in pymain_run_file (cf=0x7fffffffdad8, config=0x5555558f4ec0) at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:391
#40 pymain_run_python (exitcode=0x7fffffffdad0) at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:616
#41 Py_RunMain () at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:695
#42 0x00005555557aba09 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /tmp/build/80754af9/python_1618343417471/work/Modules/main.c:1141
#43 0x00007ffff7dbe0b3 in __libc_start_main (main=0x55555566d460 <main>, argc=22, argv=0x7fffffffdcd8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffdcc8) at ../csu/libc-start.c:308
#44 0x000055555573afe5 in _start () at ../sysdeps/x86_64/elf/start.S:103
from icefall.
from icefall.
Related Issues (20)
- CTC/AED PROBLEMS IN EXPORTING JIT MODULE HOT 4
- Error happens with egs/librispeech/ASR/prepare_mmi.sh HOT 3
- Use CutSet.mux to effect? HOT 10
- Help with training/finetuning a zipformer based model HOT 6
- Different Training Loss with Single Node (8 GPUs) vs. Two Nodes (4 GPUs Each)
- Data cleaning HOT 3
- ONNX decode error HOT 2
- OTC with conformer librispeech/WASR isn't converage.
- ONNX bug HOT 9
- Questions about modifying prepare.sh for training ASR model on custom data HOT 2
- How to use my own dataset based on another dataset HOT 3
- kaldifeat installation error HOT 2
- Why unique lexicon is needed in Chinese ASR, but not in English ASR?
- Error during training OTC conformer_ctc2 HOT 1
- What is difference between zipformer and zipformer_ctc models? HOT 1
- ONNX and Torch models HOT 1
- how to decrease the right chunk size when using zipformer model?
- How to load the base model in the fine-tuning task of KWS HOT 3
- Getting segmentation fault HOT 9
- Training break down. details is showed below HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from icefall.