thu-coai / cotk Goto Github PK

Write a script that push results to dashboard

Command:
'''
cotk-report [--result result.json] [--only-upload] [--entry main] [other parameter]
'''
result: indicates the test results.
only-upload: indicates push results without running model
entry: means the entry point of models

If running in only upload, the result should be comparable
If runing in full mode, the result can reproducible

Provide a list of api for dashboard

Refer to SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

• pump up to a newer tensorflow version
• fix all the warnings in test

Ubuntu Dialog Corpus

refer to http://dataset.cs.mcgill.ca/ubuntu-corpus-1.0/

• hred model need refactor

https://github.com/thu-coai/contk/blob/e6e3d641766e4ae2111f41e742be206cc8684d2c/tests/dataloader/test_metric.py#L61-L76

• make the length of sentence different in different turns.
• make the length of turn different in different batches

TODO: Some small test

change sentence to sent?

change all sen to sent

Need recheck

waiting #107

BaseLanguageGeneration is easily confused with LanguageGeneration
Change LanguageGeneration -> LanguageModeling ?

• Build framework

Describe the bug
hred test is wrong.

Why the turn of generated sentences > turn of reference ???

Metric forward function shouldn't change any inputs.

Add asserts into unittest.

Give a url and return a local cached path.

Put in cotk.downloader instead of _utils

Description:

Now dataloader have added new attributes: valid vocabs and invalid vocabs
valid vocabs mean the vocabularies used by models
all vocabs(== valid vocabs + invalid vocabs) mean the vocabularies used by metrics.
If a word is not any kind of all vocabs, it is unkown vocabs, which are ignored by metrics.

Metric unittest must be adapted for new metrics.

Requirements:

• Pull invalid_vocab branch
• FakeDataloader should have new attributes like all_vocab_size, ...
• Bleu & Recorder metrics have to use all vocabs
• Perplexity used a smoothing algorithm (You can see the code in PerlplexityMetric as reference):
  • If models predict valid vocabs, perplexity is calculated as it was
  • If models predict UNK, the probability is divided evenly to invalid vocabs
  • If the reference is UNK, the word is ignored.
    So, you have to write tests for the new PerplexityMetric and MultiturnPerplexityMetric
    Try to cover the 3 conditions above.

Refer to Deep Reinforcement Learning for Dialogue Generation.

Requirements:

• Download data from net (Use cache)
• Import data from local system (label it by hash value)
• Config in json type
• Put it in ./contk/_utils/file_utils.py

Code refer to
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/file_utils.py

Pay attention to Apache LICENSE

Refer to Incorporating Copying Mechanism in Sequence-to-Sequence Learning.

split it to multiple files

Requirements:

Turn models README.md to a rst file in docs

Refer to Building end-to-end dialogue systems using generative hierarchical neural network models

Refer to Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders

Describe the bug
trim_index will get error when wasn't showed up:

IndexError: list index out of range

Expected behavior
don't trim when is not met

Additional context
https://github.com/thu-coai/contk/blob/7e41e43d5eb4af4881bb5e61a338025ab9f77858/contk/dataloader/language_generation.py#L198-L220

and in other dataloaders.

Consider modify index_to_sen behavior. Because there won't be but pad sometimes

Add hints for invalid input in metric.py

For example, missing start token or end token.

Reorganize docs for metric.py

Metric may have some information to reference. To make sure it is unique, put it in hash.

Eg: self-bleu need whole test set for unique.

Describe the bug
https://github.com/thu-coai/contk/blob/e6e3d641766e4ae2111f41e742be206cc8684d2c/contk/metric/metric.py#L147-L150

Use multiturn as batch_size in sub_metric

Expected behavior
Add some comments to explain this.

Refactor to eliminate duplicate

Gather the download links of data, make a 'dataset_config.json' in ./contk/dataloader

{
"MSCOCO": "https://XXXX"
}

It is best reference from the original link, can use gzip or other compressed format.

For implemention of #8 copynet, dataloader should change behaviours.

In our mind, there should be 3 vocab list:

• For model trainning, smallest. Only include words from train set. Call it set $V.
• For metric, bigger. The model will be evaluated on this vocab list, including words from train set and test set. Call it set $M. But almostly all models can't generate words from $V-$M, because they haven't seen these. Howerver, copyNet can gen words from $V-$M by copy mechanism. It's necessary to take these words into accounts when we implement metrics. $V-$M can be expressed as UNK token for some models. Dataloader have to tranlate them into a uniform distribution on $V-$M.
• The whole space of word, include not seen in all the data. Call it set $N. The words in $N-$M, we don't care about them, ignore them in evaluating models, as #37 . $N-$M is the TRUE UNK.

Require:

• Change the behavior of dataloader, metric.

• _build_vocab has to use multi_ref data
• renamed to inference metric. embedding should have a default realization (use wordvec from Glove)
• add unittest for unique feature of SwitchBoard

• add hashvalue

Requirement

• Run models test only in cpu mode
• Just check the arguments and the connection with the main library
• Don't need to check performance
• make the test standalone, because it may need packages like tensorflow or pytorch.

User may use same id to download same data from different sources:

like “glove” default from github
"glove~github" explicit from github
"glove~tsinghua" explicit download from coai.tsinghua

Describe the bug
BleuMetric will crashed when len(hypothesis) == 1?
possible because of smoothingFunction?

It's an upstreaming bug, just comment and give up

To Reproduce

checked

Problems

It may be hard to evaluate 2 models using the same test data in the same way.
So it's important to make the metrics be able to telling which data is used.

Proposal A

Make metrics binding the dataloader. Data must be processed in the same order.

Drawback:

• must be in same order

Proposal B

Make a hash value of data. It's able to tell the differences.

Drawback:

• hard to find bugs

Use seed in tests for debugging

Describe the bug
PerlplexityMetric ->PerplexityMetric

Move ./tests/dataloader/test_metric to ./tests/metric/test_metric

TODO:

• MultiTurnDialog
• _load_data

Some metrics need models evaluating or training, we have to build a framework for them:

• CVAE - dialog act classifier
• GAN - forward perplexity & backward perplexity & wordvec embedding similarity

wordvec refer to https://github.com/wlin12/wang2vec

Requirements:

modify ./docs/source/ to add description for models

Describe the bug
https://github.com/thu-coai/contk/blob/02e980498b534a174baea59dfbe9f42291233010/contk/dataloader/multi_turn_dialog.py#L199

refer to MSCOCO ?

Requirement:

remove <eot> from multiturn dialog, use <eos> instead. Don't using any mark between sentences.

• mscoco
• OpenSubtitles
• UbuntuCorpus

Check if we can redistribution the data.

Write a model for Language Generation Dataloader. Either in tensorflow or pytorch.

If you write in tensorflow, please use a newer version of tensorflow like 1.13.

Tests are required.

perplexity sometimes should ignore UNK.

waiting #38 (over)

If there is no best checkpoint (which means it restore from the previous run), load the best checkpoint will be an error.