Comments (11)
@xuy2 This code is merged into master now.
from finetune.
lt means the latter -- randomly choosing 512 contiguous tokens from an article. A random slice of text.
from finetune.
Thanks for the ticket @elyase!
I have support for language model pretraining up in a branch now: https://github.com/IndicoDataSolutions/finetune/tree/madison/lm-pretraining.
Wasn't too hard to add, we actually already had the interface support for this but unintentionally stopped supporting it during a past refactor.
Still need to put together some documentation, but it works as you've requested so for now you can work directly off of that branch if you'd like.
from finetune.
Thought about this a bit more, the PR we have in will not allow you to train on different languages, but will give you some benefit in the scenario where you have a lot of unlabeled data for a specific English domain and a limited amount of a labeled training data. Just wanted to clarify what half of your issue we would be able to resolve.
from finetune.
@madisonmay , thanks a lot, great work. Is this line:
Line 67 in da624ae
the only missing piece in order to train on lets say German or is there something else missing?
from finetune.
@elyase there are a few pieces that would need to be modified in order to support a new language.
One of those changes would be swapping out the english tokenizer for a German tokenizer. You can find a German tokenizer as part of spacy so that portion should be relatively straightforward.
Secondly, the byte-pair encoder (that's used to decide how to split up words into subword pieces to have a useful fallback for out of vocabulary words) was "fit" on English text. This means that the current pretrained model's vocabulary primarily contains English text and uses word pieces that are based on English frequencies.
Finally, there would be some minimal changes to make in order to ensure that you're starting from randomly initialized weights rather than the weights learned by the English model.
There might be some other required changes that I'm overlooking but that's what comes to mind right now.
Know that training a language model from scratch on a new language will be a pretty big computational investment -- think along the lines of 4-8 GPUs + a week of training time.
from finetune.
Hi, I cannot open the branch for pretraining language model anymore. Can you kindly tell me how can I find it?
from finetune.
Closing this issue as finetuning the language model only is now fully supported on the master branch as of #58. Thanks again for the feature request / bug report @xuy2! Feel free to open another issue if there's something else we can help out with.
from finetune.
@madisonmay @benleetownsend I have a question for the pre-trained model. The paper said it uses "randomly sampled, contiguous sequences of 512 tokens" to pre-train. Does it mean doing padding for a sentence to 512 tokens, or randomly choosing contiguous 512 tokens from an article?
from finetune.
@xuy2 that's actually a valid point -- I'm not sure if the model ever received inputs of < 512 tokens at train time.
from finetune.
@madisonmay Thank you! In the latter method, it seems that we can only train the model by the batch instead of the epoch. How can we evaluate the model performance using test data? If using the random slice, it's hard to guarantee that the whole test set is evaluated.
from finetune.
Related Issues (20)
- Add chunk long sequences support to featurize_sequence HOT 1
- Make tqdm behavior more consistent between train / predict
- ModuleNotFoundError: No module named 'tensorflow.contrib' HOT 1
- Make try/except in visible devices search less broad and print traceback to enable debugging deployed finetune
- Soft targets for sequence labeling models (with use_crf=False) HOT 2
- Add chunk long sequences support to featurize
- Add "download=False" argument to optionally prevent automatic download on instantiation
- Check md5sums of hashes base model file hashes
- McDonalds dataset link is broken HOT 1
- Document tensorflow 2.0 requirement and removal of 1.x support
- How to save checkpoint when finetune?
- Pin Spacy Version to < 3.0 HOT 1
- [Need help] How can I load a created base model HOT 4
- LayoutLM documentation
- Add absl-py to requirements.txt
- Question: Does the Bert base model support multiple languages?
- Use on Colab Fails HOT 2
- Finetune a model Q/A Response
- MultiLabelClassifier predictions bug HOT 2
- Unavailable pre-trained model configs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from finetune.