atselousov / transformer_chatbot Goto Github PK

Hi,
I wanted to test transformer_chatbot and apply my task.
I followed the README.md(install nvidia-docker, prepare_environment.sh) and tried to run environment/docker_{}.sh.
And I got messages such that,

jglee@jglee-System-Product-Name:~/Research/transformer_chatbot$ sh environment/docker_eval_fi.sh
[sudo] password for jglee:
/opt/conda/lib/python3.6/site-packages/nltk/twitter/init.py:20: UserWarning: The twython library has not been installed. Some functionality from the twitter package will not be available.
warnings.warn("The twython library has not been installed. "
[ optional arguments: ]
[ aggregate_micro: False ]
[ display_examples: False ]
[ log_every_n_secs: 2 ]
[ metrics: f1 ]
[ num_examples: -1 ]
[ Main ParlAI Arguments: ]
[ batchsize: 20 ]
[ datapath: /workspace/ParlAI/data ]
[ datatype: valid ]
[ download_path: /workspace/ParlAI/downloads ]
[ hide_labels: False ]
[ image_mode: raw ]
[ init_opt: None ]
[ multitask_weights: [1] ]
[ numthreads: 1 ]
[ show_advanced_args: False ]
[ task: convai2:self:no_cands ]
[ ParlAI Model Arguments: ]
[ dict_class: None ]
[ init_model: None ]
[ model: agent:TransformerAgent ]
[ model_file: None ]
[ PytorchData Arguments: ]
[ batch_length_range: 5 ]
[ batch_sort_cache_type: pop ]
[ batch_sort_field: text ]
[ numworkers: 4 ]
[ pytorch_context_length: -1 ]
[ pytorch_datapath: None ]
[ pytorch_include_labels: True ]
[ pytorch_preprocess: False ]
[ pytorch_teacher_batch_sort: False ]
[ pytorch_teacher_dataset: None ]
[ pytorch_teacher_task: None ]
[ shuffle: False ]
[ Tensorboard Arguments: ]
[ tensorboard_comment: ]
[ tensorboard_log: False ]
[ tensorboard_metrics: None ]
[ tensorboard_tag: None ]
[ ParlAI Image Preprocessing Arguments: ]
[ image_cropsize: 224 ]
[ image_size: 256 ]
[ Agent parameters: ]
[ add_questions: 0 ]
[ annealing: 0 ]
[ annealing_topk: None ]
[ beam_size: 3 ]
[ check_grammar: False ]
[ clean_emoji: False ]
[ correct_generative: False ]
[ detokenize: False ]
[ diversity_coef: 0 ]
[ diversity_groups: 1 ]
[ emoji_prob: 0 ]
[ gpu: -1 ]
[ length_penalty: 0.6 ]
[ max_seq_len: 256 ]
[ ngram_size: 3 ]
[ no_cuda: False ]
[ rank_candidates: False ]
[ replace_ngram: False ]
[ replace_repeat: False ]
[ sample: False ]
[ split_into_sentences: False ]
[ wild_mode: False ]
HEAD http://localhost:9200/ [status:N/A request:0.000s]
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/conda/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/conda/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5c329a78d0>: Failed to establish a new connection: [Errno 111] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.001s]
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/conda/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/conda/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5c329a7978>: Failed to establish a new connection: [Errno 111] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.001s]
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/conda/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/conda/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5c329a7ac8>: Failed to establish a new connection: [Errno 111] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.001s]
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 172, in perform_request
response = self.pool.urlopen(method, url, body, retries=Retry(False), headers=request_headers, **kw)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/conda/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/conda/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/conda/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/opt/conda/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5c32452550>: Failed to establish a new connection: [Errno 111] Connection refused
Traceback (most recent call last):
File "eval_f1.py", line 27, in
eval_f1(opt, print_parser=parser)
File "/workspace/ParlAI/projects/convai2/eval_f1.py", line 27, in eval_f1
report = eval_model(opt, print_parser)
File "/workspace/ParlAI/parlai/scripts/eval_model.py", line 106, in eval_model
agent = create_agent(opt, requireModelExists=True)
File "/workspace/ParlAI/parlai/core/agents.py", line 639, in create_agent
model = model_class(opt)
File "/workspace/agent.py", line 80, in init
split_into_sentences=self.opt['split_into_sentences'])
File "/workspace/model/postprocessing.py", line 79, in init
self._retrieval = RetrievalBot()
File "/workspace/model/retrieval.py", line 65, in init
raise ValueError('Connection to retrieval server is failed.')
ValueError: Connection to retrieval server is failed.

What I realized is,
the function "self.es.ping()" (line 64 in model/retrieval.py) returns False and it raises value error.

my environment is something wrong?
Is there anyone who can help me?

Thank you for reading this!

Thanks for releasing the code!

I'm wondering whether you have any thoughts on why your Hits@1 is substantially lower than many of the other teams in the ConvAI2 competition, while the scores from the human evaluation are the highest overall.

From the slides you presented, it looks like you didn't actually optimize a classification objective (just PPL, smoothed PPL, and an F1 risk objective). Do you think adding a classification objective might bump your Hits@1 up significantly?

Hi, as one of the top 7 teams in the automatic evaluations (and you may yet be the winner in human evaluations) we can offer 1 ticket for one of you to attend the NIPS workshops (as we understand you still have to buy the ticket).

We invite you to present at the competition workshop, either with a full talk or a short spotlight + poster, with the length of the talk depending on what happens in the human evaluations (whether you win or not).

What we need from you is:
(1) let us know you will be coming; and
(2) the relevant name and NIPS.CC email address so that we can tell the NIPS organizers to offer a ticket to that address. We need this pretty soon.

Thanks!

Hi! A team of collaborators at Facebook (Ryan Lowe, Koustuv Sinha) and I are looking at incorporating outputs of your model into a labelled personachat dataset.

Is it a generative model?

My understanding is that all convai2 submissions were supposed to be structured such that if directly placed in https://github.com/facebookresearch/ParlAI/tree/master/projects/convai2/baselines, I could run eval_f1.py (which is how I've set up the log collection).

How can I run eval_f1.py on this model without setting up docker?

Fairly urgent - I'm hoping to get this working by tomorrow EOD.

May I ask how BPE vocabulary is generated？ and how to do it on my own data? thanks

I want to fine-tune from openai's model, but the file parameters_names.json is reqiured, and it is not listed in https://github.com/openai/finetune-transformer-lm/tree/master/model. How can I get it?

Hi I have tried downloading the DailyDialogue dataset from parlai. The format seems to not match with the data configuration and dataloader in the code. Is there way to get the dataset other than from parlai?

Also, your slide (page 3) mentions reddit comments dataset but it is not included in the dataset configuration. For training, are all three datasets (convai2, daily dialogue, and reddit comments) needed to achieve the score you stated in the slide (page 10)?

Thank you,

Hello!
Thank you for your great work. I have been trying to understand and use this model for some experimentation.

The model trained for over 17hours on V100, the loss has been decreasing and everything seems to be fine. I used a small dataset of 28k lines of conversation data.

However, when using: python interactive.py the model predictions don't reflect my small dataset. It seems like everything is overridden by the previous checkpoint.

Is it possible to do transfer learning on top of your checkpoint? Should I train for longer?

Hi,

After running prepare_environment.sh and attempting one of the docker eval scripts, the script is unable to connect to the port: 9200.
Should I make my own docker network, or should be able to work on host or bridge?

Thanks.

Hello atselousov:
I want to train my own model with my own data. So I would very appreciate it if you introduce the format of training data and the steps of training process.
Thank you very much!
ghostyoona

Thanks for your submission! There may be some delay in evaluation as we have received a large number of submissions in the last couple of days, but I'm writing to confirm that we have received your submission and will be evaluating it. If we have any issues performing the evaluation we will comment here.

config's path of the data:
'train_datasets': ['./datasets/ConvAI2/train_self_revised_no_cands.txt',
'./datasets/ConvAI2/train_self_original_no_cands.txt',
'./datasets/DailyDialog/train_dailydialog.txt'],
'test_datasets': ['./datasets/ConvAI2/valid_self_revised_no_cands.txt',
'./datasets/ConvAI2/valid_self_original_no_cands.txt',
'./datasets/DailyDialog/valid_dailydialog.txt']

Congratulations! Your team made it to the next round (along with 6 other teams)! At this point, your automatic metrics are now discarded, as the winning team will be decided by human evaluations. Please make sure your GitHub repo contains an interactive script (see here for example) that allows your model to function in interactive mode. (To be clear, this must be the same model that we evaluated for the leaderboard.)

Reach out with any questions by opening an issue on your GitHub.

Sincerely,

The ConvAI2 Team

Dear participants,

We are going to start wild evaluation process soon, so we expect you to make ready your solution for it until Friday, 19th of October. Please make sure that you've tested it with our test server. Please have a look at info about wild evaluation at http://convai.io/wild . It would be helpful if you provide an instruction to run your solution.

If you have any questions/issues we will be glad to help you.