wind91725 / gpt2-ml-finetune- Goto Github PK

想问下loss下降到多少是差不多收敛了，参考下你的训练结果

gpt2-ml-Finetune/finetune_model/model.ckpt-220000 is not in all_model_checkpoint_paths. Manually adding it.
不在检查路径中。。

小白一枚，finetune的时候，自己的数据集需要多大会比较好？1M，10M还是说需要更大的？

提示prepare_data.py也执行完成了，但是在data文件夹中只有22.json文件，没有train.tfrecord文件，并且没有错误提示。
CALLING CLOSE
DONE UPLOADING

执行生成tfrecord训练数据这一步的时候
python prepare_data.py -input_fn /data/home/share1/gpt2-ml-Finetune/data
出错了
Traceback (most recent call last):
File "./prepare_data.py", line 17, in
from tokenization import tokenization
ModuleNotFoundError: No module named 'tokenization'
请问如何解决？谢谢

文件格式是什么？需要分词吗

i found my text to this script output no text.
then I debug and find pre_data text2 = re.sub(pattern1,'',text) .the text is not None but text2 is None

really thanks to helping me

请问输入文件这个目录在哪里呢？/data/home/share1/gpt2-ml-Finetune/data-mayun_xiugai

tensorflow2.x一直报错，因为 'contrib'在2.x中已经删除，降级成1.x（1.14、1.15）能运行，
开始训练后会出现一堆warring：

【Start trainning.............................................
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0616 06:45:01.207822 140262356580224 deprecation.py:506] From /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0616 06:45:01.208348 140262356580224 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /content/drive/MyDrive/gpt2-ml/train/dataloader.py:63: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic.
W0616 06:45:01.223439 140262356580224 deprecation.py:323] From /content/drive/MyDrive/gpt2-ml/train/dataloader.py:63: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic.
WARNING:tensorflow:From /content/drive/MyDrive/gpt2-ml/train/dataloader.py:81: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
】

开始循环训练之后会出现致命错误：

【ERROR:tensorflow:Error recorded from training_loop: module 'tensorflow._api.v1.compat.v1' has no attribute 'contrib'
E0616 06:45:48.644567 140262356580224 error_handling.py:75] Error recorded from training_loop: module 'tensorflow._api.v1.compat.v1' has no attribute 'contrib'
INFO:tensorflow:training_loop marked as finished】

Google了一圈没有找到解决办法，我猜最大的问题出现在 'tensorflow._api.v1.compat.v1' has no attribute 'contrib'上，估计修改API就好？但是找不到这段代码在哪里。
小白一个，还请大佬指点迷津。

定位到model.ckpt-100000，一直说找不到匹配的文件，您有出现这种情况吗？

觉得奇怪的地方， train和 inference的时候使用的vocabulary 不是同一个文件的？

为什么我finetune之后的模型生成的文本更加乱了， 使用原版的模型倒是没有问题

AttributeError: module 'tensorflow.compat.v1' has no attribute 'contrib'试过tf1.15.2和1.13.1第三步都报这个。求教

我尝试了下使用32G-v100训练, 剩余显存可能没达到32G,可能是28G左右, 卡在这一句2020-07-11 15:00:55.541191: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally就没有输出了, 这是因为显卡剩余显存不够吗？还是说需要更好的显卡？

您好，这个程序把数据变成什么形式的了？

![image](https://user-images.githubusercontent.com/6139188/93421785-e49dcc00-f8e4-11ea-9d64-2471fa1e9d6a.png
如题

您好，这个参数是干嘛的

我看wind91725老师写的是：python train/train_wc.py --input_file=/data/home/share1/gpt2-ml-Finetune/data/train.tfrecord --output_dir=/data/home/share1/gpt2-ml-Finetune/finetune_model --init_checkpoint=/data/home/share1/gpt2-ml/models/mega/ceshi/model.ckpt-100000

实际的--init_checkpoint参数，是不是应该改为gpt2-ml的现有模型目录：models/mega/model.ckpt-220000，感谢！