Q&A like SQuAD about nlp.js HOT 4 CLOSED

Well, SQuAD dataset contains more than 100k pairs of question and answer. The target of NLP.js is to more modest datasets. I didn't do the test, but for training the logistic regression classifier with 100K intents and the amount of features (different words) of SQuAD, is for sure too much for training with CPU, and only the matrix of features and labels can be too much.

Perhaps with tensorflow, NLTK and with GPUs can be a better approach. Who knows, perhaps in the future we will go to the path of tensorflow.js and NLTK, but right now is not in the roadmap.

from nlp.js.

I tried ~100K Q&A train with AWS 32GB Memory 8 vCPU 75 GB SSD Space, but didn't create model.nlp file and didn't show any error, no error, no model.nlp :/ I don't know problem. When ı training dataset memory is ~16GB level.

from nlp.js.

We have an example supporting 10.000 intents, extracted from SQuAD v2.
Open Question with BERT is added, but with it needs a python app to work.

Right now I'm working on doing OpenQuestion fully javascript. The first problem is that tokenizers only work in node 11 or 13 and is a bridge to rust, I solved that by building BERT word piece tokenizer by myself. Now I'm working on the tenworflow open question model and runtime

from nlp.js.

Closing as integration with BERT API is provided

from nlp.js.