himkt / awesome-bert-japanese Goto Github PK
View Code? Open in Web Editor NEW📝 A list of pre-trained BERT models for Japanese with word/subword tokenization + vocabulary construction algorithm information
📝 A list of pre-trained BERT models for Japanese with word/subword tokenization + vocabulary construction algorithm information
Hello,
Thank you for collecting links to the bert based models for Japanese
Just wanted to ask if you know any models or investigations regarding raw text (after automatic speech recognition the text is not splitted at all, just characters one by one) segmentation? Something simple like splitting text on sentences or more complicated like adding punctuation to the text. For example, nvidia provides models for punctuation based on bert and distilbert: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/punctuation_and_capitalization.html
That would be great if there is something for raw text split for Japanese language
Although not sure if we would include BART.
https://tech.stockmark.co.jp/blog/bart-japanese-base-news/
東北大学 (a)の「サブワード分割のための語彙構築アルゴリズム」はSentencepieceだと思います。
以下のscriptで、Sentencepieceでまずvocabを学習してから、BERTのvocab.txtのフォーマットになるように変換しています。
https://github.com/cl-tohoku/bert-japanese/blob/master/build_vocab.py
東北大学 (b)の「単語 -> サブワード」は文字単位なので Character とかの方がいいのではないでしょうか。
(「サブワード分割のための語彙構築アルゴリズム」のところ、正確にはSentencepieceの --model_type=char
オプションで学習していますが、実質文字単位なので Character でいいと思います。)
NICT (a)の「単語 -> サブワード」はWordPieceであっていると思います。
NICT (b)が「BPEなし」モデルだと思いますが、「BPE」が人によって何を指しているのかがまちまちというのもあるのですが、ここでは「BPEなし」は「サブワードに分割せずに形態素単位」という意味なので、「単語 -> サブワード」「サブワード分割のための語彙構築アルゴリズム」ともに「--」が正しいと思います。
https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp
(Thank you @icoxfog417 for pointing out!)
Although not decided whether to include T5.
https://qiita.com/sonoisa/items/a9af64ff641f0bbfed44
Although not decided whether to include T5.
https://huggingface.co/megagonlabs/t5-base-japanese-web-8k
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.