Comments (6)
greedyuser
commented on June 2, 2024
Chris,
The full spec is:
weights:
initial: "/path/to/load/first/weights"
last: "/path/to/save/when/exiting"
checkpoint:
path: "/path/to/save/checkpoints"
minutes: "{{ checkpoint_time }}"
validation: "{{ batch size }}"
Are you using the default speech.yml in your tests?
from kur.
chrisspen
commented on June 2, 2024
Yes. I see no weights or checkpoint sections in your speech.yml file. Where are those defined?
from kur.
greedyuser
commented on June 2, 2024
When you look in the 'train' section you see a 'weights' section that imports in the weights from the 'settings' section:
train:
data:
# A "speech_recognition" data supplier will create these data sources:
# utterance, utterance_length, transcript, transcript_length, duration
- speech_recognition:
<<: *data
url: "https://kur.deepgram.com/data/lsdc-train.tar.gz"
checksum: >-
fc414bccf4de3964f895eaa9d0e245ea28810a94be3079b55505cf0eb1644f94
weights: *weights
If you replace that with what I have above it should save checkpoints and a 'last' set of weights when the model stops. Paths can be relative so naming things like 'last.model.kur' and 'checkpoint.model.kur' allows you to just create a new directory, copy in the yml files and execute if you want to start a new model.
You also need to make any similar changes to the 'validate' section.
from kur.
chrisspen
commented on June 2, 2024
If I make those changes, then kur doesn't run at all and fails with the exception:
Traceback (most recent call last):
File "~/.env/bin/kur", line 8, in <module>
sys.exit(main())
File "~/.env/lib/python3.7/site-packages/kur/__main__.py", line 492, in main
sys.exit(args.func(args) or 0)
File "~/.env/lib/python3.7/site-packages/kur/__main__.py", line 62, in train
spec = parse_kurfile(args.kurfile, args.engine)
File "~/.env/lib/python3.7/site-packages/kur/__main__.py", line 48, in parse_kurfile
spec.parse()
File "~/.env/lib/python3.7/site-packages/kur/kurfile.py", line 129, in parse
self.engine, builtin['train'], stack, include_key=True)
File "~/.env/lib/python3.7/site-packages/kur/kurfile.py", line 960, in _parse_section
evaluated = engine.evaluate(self.data[key], recursive=True)
File "~/.env/lib/python3.7/site-packages/kur/engine/engine.py", line 228, in evaluate
for k, v in expression.items()}
File "~/.env/lib/python3.7/site-packages/kur/engine/engine.py", line 228, in <dictcomp>
for k, v in expression.items()}
File "~/.env/lib/python3.7/site-packages/kur/engine/engine.py", line 228, in evaluate
for k, v in expression.items()}
File "~/.env/lib/python3.7/site-packages/kur/engine/engine.py", line 228, in <dictcomp>
for k, v in expression.items()}
File "~/.env/lib/python3.7/site-packages/kur/engine/engine.py", line 208, in evaluate
new_expression = self._evaluate(expression)
File "~/.env/lib/python3.7/site-packages/kur/engine/jinja_engine.py", line 189, in _evaluate
result = self.env.from_string(expression).render(**self._scope)
File "~/.env/lib/python3.7/site-packages/jinja2/environment.py", line 880, in from_string
return cls.from_code(self, self.compile(source), globals, None)
File "~/.env/lib/python3.7/site-packages/jinja2/environment.py", line 591, in compile
self.handle_exception(exc_info, source_hint=source_hint)
File "~/.env/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "~/.env/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<unknown>", line 1, in template
File "~/.env/lib/python3.7/site-packages/jinja2/environment.py", line 497, in _parse
return Parser(self, source, name, encode_filename(filename)).parse()
File "~/.env/lib/python3.7/site-packages/jinja2/parser.py", line 901, in parse
result = nodes.Template(self.subparse(), lineno=1)
File "~/.env/lib/python3.7/site-packages/jinja2/parser.py", line 876, in subparse
self.stream.expect('variable_end')
File "~/.env/lib/python3.7/site-packages/jinja2/lexer.py", line 384, in expect
self.name, self.filename)
jinja2.exceptions.TemplateSyntaxError: expected token 'end of print statement', got 'size'
Why would the default speech.yml need modifications just to save the trained network? Shouldn't those settings be the default?
from kur.
chrisspen
commented on June 2, 2024
I suspect instead of:
validation: "{{ batch size }}"
you meant to type:
validation: "{{ batch_size }}"
That lets it run for a short while, but then it still errors with:
Traceback (most recent call last):
File "~/.env/bin/kur", line 8, in <module>
sys.exit(main())
File "~/.env/lib/python3.7/site-packages/kur/__main__.py", line 492, in main
sys.exit(args.func(args) or 0)
File "~/.env/lib/python3.7/site-packages/kur/__main__.py", line 64, in train
func(step=args.step)
File "~/.env/lib/python3.7/site-packages/kur/kurfile.py", line 434, in func
return trainer.train(**defaults)
File "~/.env/lib/python3.7/site-packages/kur/model/executor.py", line 295, in train
**kwargs
File "~/.env/lib/python3.7/site-packages/kur/model/executor.py", line 564, in wrapped_train
checkpoint[k]))
ValueError: Expected "minutes" key in "checkpoint" to be an integer. Received:
Are you sure minutes: "{{ checkpoint_time }}" is the correct syntax and variable name?
from kur.
greedyuser
commented on June 2, 2024
Sorry about the typo there! I should have told you to define batch_size and checkpoint_time in the settings section. They can be literals, but for clarity I would recommend defining them in settings:
batch_size: 16
checkpoint_time: 30
Of course, select whatever batch size and checkpoint you need here.
from kur.
Related Issues (20)
- Weights are not getting saved. HOT 2
- Validation loss diverging in speech example HOT 1
- add support for mxnet
- Truth Data generation Error HOT 5
- How to get text output from evaluate.pkl file ? HOT 1
- Running STT on specific file?
- AttributeError: module 'keras.backend' has no attribute 'set_image_dim_ordering' HOT 1
- out of memory error
- How do I transcribe the wav file?
- Validation loss become higher after 20 hours training time HOT 3
- Weights for speech recognition are not restored when again starting the training as loss value climbs back to 1st epoch value i.e 316 instead of starting from reduced loss HOT 4
- Training for speech recognition using KUR on a distributed system. Is it possible? HOT 1
- CUDA_ERROR_NO_DEVICE HOT 1
- Import Voxforge to Kur HOT 1
- speech.yml prediction is empty HOT 2
- failed to find any available GPUs HOT 1
- The speed of training gradually decreases when using gpu HOT 4
- Training a dataset with non-latin characters HOT 1
- Error when training on custom data HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kur.