Bert 做 文本生成 的一些實驗
How bert perform on text generation ?
Try it in different fine tuning ways
- Generate char one by one
- Generate result one time
- Generate from LSTM
https://colab.research.google.com/drive/19wgXJPdb_282M0_puMgQ8qid0jvmJghG
[CLS] I go to school by bus [SEP]
我搭公車上學
Bert generate once
['[CLS]', 'i', 'go', 'to', 'school', 'by', 'bus', '[SEP]', '[MASK]']
[-1, -1, -1, -1, -1, -1, -1, -1, 2769]
Bert generate one by one
tensor([[ 101, 151, 8373, 8228, 9467, 8120, 10411, 102, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103, 103, 103,
103, 103, 103, 103, 103, 103, 103, 103]],
device='cuda:0') tensor([[ -1, -1, -1, -1, -1, -1, -1, -1, 2769, 3022, 1062, 6722,
677, 2119, 102, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1]], device='cuda:0')
Bert generate lstm
Same as Bert generate once
['[CLS]', 'i', 'go', 'to', 'school', 'by', 'bus', '[SEP]', '[MASK]']
[-1, -1, -1, -1, -1, -1, -1, -1, 2769]
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent