A recent ICLR 2018 submission (https://arxiv.org/pdf/1711.03953.pdf) claims to achieve sub-50 perplexities in language modelling on Penn Treebank.

Does anyone happen to know the state-of-the-art for the popular 20 News Groups dataset? (And what's the most common train/dev/test splits people use?)

Hi, thanks for maintaining this list, it's awesome!

Just wonder if you are aware of nlpprogress.com/ which are doing similar things but focus on NLP. It would be nice to work together with them.

Time Series Classification is a very popular machine learning problem.
You can find a full survey and empirical study (link to paper) on 85 datasets that can be found here.
More recently in our paper we showed that deep learning can also reach state if the art performance for Time Series Classification.

• Research paper name: The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances (link) & Deep learning for time series classification: a review (link)
• Dataset: UEA archive (link)
• Metric: Accuracy + average rank comparison over the datasets (reference)
• Source code: Time Series Classification (link) & Deep learning for time series classification (link)
• Year: 2017 for Time Series Classification & 2018 for Deep learning for time series classification

https://github.com/kondiz/fraternal-dropout

paper: Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets https://arxiv.org/abs/1703.04887
year: 2107
average BLEU score in chinese - english NIST03 to NIST05 task up to 42.6 points
implementation: https://github.com/ngohoanhkhoa/GAN-NMT

Hi, I'd like to add several models implementing Mask R-CNN.
First one is Facebook Detectron in Caffe2. Works good.
Another one is in Tensorflow with custom Slim library. This one is not supported by author, but works.
Last one is MXNet

About the link in the description - it is Keras on top of TensorFlow, not pure TensorFlow.
Hope it helps.

P.S. May you add guidance in what format people should add pull requests?

I had just sent an email to [email protected] however it seems not work

Address not found
--
Your message wasn't delivered to [email protected] because the address couldn't be found, or is unable to receive mail.

The response was:The email account that you tried to reach does not exist. Please try double-checking the recipient's email address for typos or unnecessary spaces. Learn more at https://support.google.com/mail/?p=NoSuchUser r20-v6sor96536itb.73 - gsmtp

So I just post my email at here ;)

Hi there,

I'm a machine learning newbie with 20 years of programming experience. I love ML and this year I'll go for my Ph.D. study for ChatBot(NLU, ChatUI) in Beijing.

I'm willing to help as a collaborator because I love your idea of making one stop for all types of machine learning problems state of the art result, it helps me much.

Please feel free to let me know what I could do for you at any time.

My GitHub: https://github.com/zixia
My LinkedIn: https://linkedin.com/in/zixia
My WeChat: 918999

Have a nice day!

Huan LI
[email protected]

How to restore IP address

Nice work on the repo.

Maybe there should be some (unranked) honorary mentions for non quantitative / semi-quantitative results?

e.g. for image completion/inpainting
http://hi.cs.waseda.ac.jp/~iizuka/projects/completion/data/completion_sig2017.pdf

image translation
https://arxiv.org/abs/1703.10593

generative models
http://research.nvidia.com/sites/default/files/publications/karras2017gan-paper-v2.pdf

Please fix

source code：https://github.com/salesforce/nonauto-nmt

It's an excellent job in the repo.

For computer vision, some of the tasks will be important.
I will provide some topics and references that I am familiar with.

Instance Segmentation:
MASK-RCNN
The dataset used for evaluation is COCO

Bounding-Box Object Detection
MASK-RCNN

Some other metrics for evaluation might be important, such as fps for detection.
YOLO2
SSD

For bounding-box object detection, there are some other datasets:
ImageNet DET
Pascal VOC
UA-DETRAC

I have not looked at the development in speed for a while, so might be something new.
MASK-RCNN provides the best accuracy now for sure.

I find a website on which there are some state of the art objects classification results. I hope it will be useful. The link is:
http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html

According to Speech SOTA, the new state of the art on Switchboard Hub5'00 is The CAPIO 2017 Conversational Speech Recognition System, at WER of 5.0%, published in Dec., 2017.

1. Source code URL for Learning Structured Text Representations
2. Source code URL for Attentive Convolution

NLG, the other end of NLP, is important in many fields where AI is being applied. Please include the latest NLG research as well as imo it would be very helpful.

Very interested in machine learning solutions to any form of dynamic pricing including but not limited to:

Formulations

• Base case Known supply, known demand
• Retail Known supply, stochastic demand
• Consignment Stochastic supply, stochastic demand

Industries

• E-commerce
• Brick and mortar
• Airlines
• Hotels

Research Paper:
Unsupervised Neural Machine Translation with Weight Sharing
Datasets:

1.WMT14+16(en-fr fr-en de-en en-de)
2.LDC(zh-en)
Metric:
1.BLEU:(16.97 15.58 14.62 10.86)
2.BLEU:(14.52)
Source Code:
https://github.com/ZhenYangIACAS/unsupervised-NMT
Year:
2018

There is a new paper out with a faster learning and bit bit better BLEU score for the transformer architecture called Weighted Transformer Network for Machine Translation:
https://arxiv.org/abs/1711.02132

but i think there is no open source implementation available

NASNet, the paper is here,
The code is here.
The top-1 Error is 17.3 at ImageNet-1k.

Here is an update to the list for language modelling. The WikiText-103 dataset is not currently listed, but it has been a popular dataset for large vocabulary (not covered by ptb/wt2) and long time dependencies (not covered by billion word benchmark). It seems relevant for this list.

Paper title: Fast Parametric Learning with Activation Memorization
Dataset: WikiText-103
Metric: 29.2 (Perplexity)
Year: 2018
Link: https://arxiv.org/abs/1803.10049

There are a few different metrics and many datasets. I'll collect results later.

General

• "Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison", 2017

Supervised

• "Word Sense Disambiguation using a Bidirectional LSTM", 2016
• "Neural Sequence Learning Models for Word Sense Disambiguation", 2017
• "Embeddings for Word Sense Disambiguation: An Evaluation Study", 2016

Semi-Supervised

• "Semi-supervised Word Sense Disambiguation with Neural Models", 2016

paper: Squeeze-and-Excitation Networks
dataset: ImageNet 1K
metric: Top-1 Error 18.68
source code: https://github.com/hujie-frank/SENet
year: 2017

It will be good if a track of SoTA can be provided, listing the path of how the tech is developed.
I may be able to help in some of the areas

The Giant AmoebaNet claims SoTA results on ImageNet of 84.3% top-1 accuracy, beating the 2017 ImageNet winner, Squeeze-and-Excitation Networks. Should these be listed along with the CIFAR-10 and CIFAR-100 results?

(also, Gaint is misspelled. It should be Giant)

These papers are already exploring this concept
Neural Turing Machines
Neural Random Access Machines
Neural Programmer

strong baselines from openai for reinforcement learning:
https://github.com/openai/baselines
blog post: https://blog.openai.com/openai-baselines-dqn/

In the transfer learning section this Model should be mentioned:

https://arxiv.org/abs/1706.05137
But i dont know how to compare it to other approaches.

This a request to add state of the art results for sarcasm detection in text available online (like tweets, movie reviews, news etc).

There is an existing repo for speech SOTAs: https://github.com/syhw/wer_are_we. Perhaps, you want to reference them and/or join forces with them.

Concerning the Switchboard number, you need to mention that it used 2000h set for training and the Switchboard Hub5'00 for testing (not Call Home subset).

SNLI Leaderboard
https://nlp.stanford.edu/projects/snli/

MultiNLI Leaderboard
https://www.kaggle.com/c/multinli-matched-open-evaluation/leaderboard

SoTA: Densely Interactive Inference Network
Paper: https://arxiv.org/pdf/1709.04348.pdf
Code: https://github.com/YichenGong/Densely-Interactive-Inference-Network

It would be interesting to keep the list of previous state-of-start results and related papers.
That would help understanding the evolution of methods to address every single problem.

For readibility, I suggest to list them on a new page.
If you think this idea is worthwile, I'll start collecting information on this topic and will submit a PR.

Thanks for this wonderful resource! 👏

• Paper : Fixing the train-test resolution discrepancy (https://arxiv.org/abs/1906.06423)
• Code : https://github.com/facebookresearch/FixRes
• Year : 2019
• Metric : Top-1 Accuracy / Top-5 Accuracy

This a request to add state of the art materials (research papers, data-set and code-sources) for sentiment classification/analysis in text available online (like tweets, movie and product reviews, news, domain specific text etc)

This repo is fairly popular. Is it still being updated?

Hi Yudong,

I didn't see anything about object detection. Is there any reason for that or simply saying you forgot to add?

Thanks