Comments (7)
Hello,
I don't have specific experience in spell checkers, but if you can model it as a classic sequence to sequence model, then yes you can use OpenNMT-tf as is. Otherwise, the code should be friendly enough to customize it.
from opennmt-tf.
Hello, you may want to check this as an entry point: http://nlp.seas.harvard.edu/papers/aesw2016.pdf
from opennmt-tf.
Hello,
I understand that this is an old topic now, but I just wanted to add a slight note on this.
With the BERT paper, and with this Seq2Seq library, it is possible to create a Spell Checker using openNMT by the introduction of "masking". In BERT they mask some words, and they replace some other words with random words. Using the same approach, we can replace some characters with other character on purpose, to teach the model to fix them. Using 12 enc-dec layers as CNN did the trick for me.
from opennmt-tf.
This is the BERT paper I'm referring to.
Basically the concept that I derived from the paper (it is not exactly how it's done in the paper, but it is what I did) is to purposefully replace some characters with wrong ones. For example:
I love to play with my cat
would turn into characters
I l o v e t o p l a y w i t h m y c a t
Then I would create 5 train cases from this sentence:
I l o v r t o p l e y w i h h m y c a t
I l u v e t u p l a a w e t h m y c a t
... etc
The target for these 5 sentences would be
I love to play with my cat
Which is the same sentence we began with. This way the model will learn how to deduce sentences on its own, and at the same time it'll learn to fix some characters.
For me the example was character to words, but you can word to word at a character level. Example:
I love to play with my cat
becomes
I <sep> l o v e <sep> t o <sep> p l a y <sep> w i t h <sep> m y <sep> c a t <sep> <eos>
this way you teach the model to only fix the word with itself, instead of trying to fix the sentence. It will depend on your case. Let me know if you need more clarification.
from opennmt-tf.
Thanks @guillaumekln and @jsenellart for your replies and advice. I appreciate it.
from opennmt-tf.
Thanks @ridhwan-saal for your update. Do you have code to share? This sounds really interesting. Thanks again for your useful update. Maybe, you should write a Medium article about it. It would be great if you can share the paper you are referring to.
from opennmt-tf.
Thanks a million @ridhwan-saal. I really appreciate taking the time to give such a great and detailed explanation.
from opennmt-tf.
Related Issues (20)
- xla_ops failed when use multi gpu
- word-level knowledge distillation
- Input size mismatch
- tensorflow lite model example HOT 8
- OpenNMT-tf onnx export ?
- Wrong prediction length from exported OpenNMT-tf models with shared embeddings HOT 1
- Error when inferencing with gpt. HOT 3
- An issue with SequenceRecordInputter ? HOT 1
- cpp for tensorflow serving HOT 2
- SequenceClassifier doesn't seem to learn leaked target HOT 2
- What material did you use to train your pretrained model? HOT 1
- Fine-tune GPT2 model from Transformers HOT 4
- Modify loss function or mask attention in this code HOT 1
- Parallel text inputter fails with compressed text files HOT 1
- An input error occurred while expanding the dataset HOT 1
- ModuleNotFoundError: No module named 'lxml.etree' HOT 1
- Training fails with --mixed_precision and guided alignments
- "Horovod has been shut down" error when training is finished due to early stopping HOT 2
- Make timeout value configurable while searching for an optimal batch size
- Quickstart example runs for a long time HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opennmt-tf.