Comments (3)
csukuangfj
commented on August 21, 2024
Seems that the G is too large and there is an overflow in k2.
from icefall.
danpovey
commented on August 21, 2024
Hm. I'd like to see some debug information about what's in shape that is causing this; I suspect error may have been earlier, and may have been fixable.
We should actually support loading Kenlm LM's onto GPU and using them via a deterministic-FST interaface. But that will take some time.
from icefall.
EmreOzkose
commented on August 21, 2024
I tried to reproduce this case for getting shapes, but it takes too long (2-3 hours) and the script is just killled:
2021-08-24 09:29:22 (prepare.sh:44:main) pre_file_dir: /path/to/icefall/egs/sestek/ASR/pre_files
2021-08-24 09:29:22 (prepare.sh:119:main) Stage 6: Compile HLG
2021-08-24 09:29:23,508 INFO [compile_hlg.py:128] Processing data/lang_phone
2021-08-24 09:29:24,062 INFO [lexicon.py:96] Loading pre-compiled data/lang_phone/Linv.pt
2021-08-24 09:29:24,289 INFO [compile_hlg.py:48] Building ctc_topo. max_token_id: 52
2021-08-24 09:29:24,489 INFO [compile_hlg.py:53] Loading pre-compiled G_3_gram
2021-08-24 09:29:30,649 INFO [compile_hlg.py:68] Intersecting L and G
2021-08-24 09:42:25,292 INFO [compile_hlg.py:70] LG shape: (301909252, None)
2021-08-24 09:42:25,292 INFO [compile_hlg.py:72] Connecting LG
(301909252, None)
2021-08-24 09:42:25,292 INFO [compile_hlg.py:75] LG shape after k2.connect: (301909252, None)
(301909252, None)
2021-08-24 09:42:25,292 INFO [compile_hlg.py:77] <class 'torch.Tensor'>
2021-08-24 09:42:25,292 INFO [compile_hlg.py:78] Determinizing LG
2021-08-24 09:58:24,804 INFO [compile_hlg.py:81] <class '_k2.RaggedInt'>
(214860596, None)
2021-08-24 09:58:24,804 INFO [compile_hlg.py:83] Connecting LG after k2.determinize
2021-08-24 09:58:24,804 INFO [compile_hlg.py:86] Removing disambiguation symbols on LG
(214860596, None)
(214860596, None)
(214860596, None)
./prepare.sh: line 126: 23181 Killed ./local/compile_hlg.py --lang-dir data/lang_phone
2021-08-24 10:04:32,846 INFO [compile_hlg.py:128] Processing data/lang_bpe_5000
2021-08-24 10:04:33,354 INFO [lexicon.py:96] Loading pre-compiled data/lang_bpe_5000/Linv.pt
2021-08-24 10:04:33,526 INFO [compile_hlg.py:48] Building ctc_topo. max_token_id: 4999
2021-08-24 10:04:34,405 INFO [compile_hlg.py:53] Loading pre-compiled G_3_gram
2021-08-24 10:04:41,941 INFO [compile_hlg.py:68] Intersecting L and G
2021-08-24 10:07:20,234 INFO [compile_hlg.py:70] LG shape: (80262949, None)
2021-08-24 10:07:20,234 INFO [compile_hlg.py:72] Connecting LG
(80262949, None)
2021-08-24 10:07:20,234 INFO [compile_hlg.py:75] LG shape after k2.connect: (80262949, None)
(80262949, None)
2021-08-24 10:07:20,234 INFO [compile_hlg.py:77] <class 'torch.Tensor'>
2021-08-24 10:07:20,234 INFO [compile_hlg.py:78] Determinizing LG
./prepare.sh: line 122: 27422 Killed ./local/compile_hlg.py --lang-dir $lang_dir
'lang_bpe/lang_bpe_5000' -> 'lang_bpe_5000'
I tried to count via irstlm instead of kenlm. When I pruned kenlm model, it was reduced to 2.1 gb from 5 gb for 3-gram. However I pruned irstlm model (with threshold 3e-7) and now sizes are 68M and 85M for 3-gram and 4-gram respectively.
In the end, preparing G is done successfully.
from icefall.
Related Issues (20)
- How to run a streaming zipformer transducer model with my own dataset? HOT 1
- get data manifest for SSL recipe
- Training speed is not improved by using a better GPU HOT 14
- librispeech SSL finetune.py throwing error
- Training with disfluencies in speech
- librispeech hubert pretrain.py throwing error : UnboundLocalError: local variable 'sub_batch_idx' referenced before assignment
- error in librispeech SSL pretrain.py HOT 1
- How did you prepare the manifest dir for pretrain and in which format? HOT 1
- early context injection HOT 3
- LLM based speech recognition HOT 1
- Recommended recipe for noisy 5K hours noisy training data HOT 1
- How to fine-tune KWS without downloading wenetspeedch HOT 1
- Libriheavy train_bert_encoder.py incompatible with Lhotse 1.27.0 HOT 3
- Early Stopping of Token Generation in Streaming Model Training HOT 16
- How to specify zipformer model training with different specifications HOT 3
- Zipformer-s model training problem , thanks!
- gigaspeech数据集下载成功，但解压缩失败
- Troubles with streaming decode
- Use multi_zh-hans finetune whisper get erro
- [Transducer Loss] Why not normalize transducer loss
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from icefall.