State of the Art in ML

It is difficult to keep track of the current state of the art (SotA). Also, it might not be directly clear which datasets are relevant. The following list should help. If you think some datasets / problems / SotA results are missing, let me know in the comments or via E-mail ([email protected]). I will update it.

Papers and blog posts which summarize a topic or give a good introduction are always welcome.

In the following, a + will indicate "higher is better" and a - will indicate "lower is better".

Computer Vision

Image Classification

State of the art in this category are CNN models which use skip connections in the form of residual connections or dense connections.

The evaluation metrics are straight-forward:

• Accuracy: Count how many elements of the test dataset you got right, divided by the total number of elements in the test dataset. The accuracy is in \([0, 1]\). Higher is better.
• Error = 1 - accuracy. The error is in \([0, 1]\). Lower is better.
• Top-k accuracy: Sometimes, there are either extremely similar classes or the application allows having multiple guesses. Hence not the Top-1 guess of the network has to be right, but the correct label has to be within the top \(k\) guesses. The top-\(k\) accuracy is in \([0, 1]\). Higher is better.

Detection (Images)

Face recognition is a special case of detection.

Common metrics are:

• mAP (Mean Average Precision): A detection is successfull, if the bounding box prediction and the true bounding box $\frac{intersection}{union}$ (IU, IoU) ratio is at least 0.5. Then the average precision = $\frac{TP}{TP + FP}$ is calculated for each class and the mean is calculated of those (see Explanation, What does the notation mAP@[.5:.95] mean?)
• MR (miss rate)

Detection (Videos)

Person Re-Identitification

Person Re-ID is the task of identifying a person again which was already seen in a video stream. Person following and MTMCT seems to be very similar if not identical.

Semantic Segmentation

A summary of classical methods for semantic segmentation, more information to several datasets and metrics for evaluation can be found in A Survey of Semantic Segmentation.

Instance Segmentation

See [DHS15]

Action Recognition

Action recognition is a classification problem over a short video clip.

Super Resolution

See github.com/huangzehao

I'm not sure how super resolution is benchmarked. One way to do it would be to get high resolution images, scale them down, feed them to the network and measure the mean squared error for each pixel:

However, this might be sensitive to the way the images were downsampled.

Lip Reading

Other Datasets

For the following datasets, I was not able to find where to download them

• Mapping global urban areas using MODIS 500-m data: New methods and datasets based on urban ecoregions
• TorontoCity: Seeing the World with a Million Eyes

ASR

Automatic Speech Recognition (ASR).

Sentence-Level

See Word Error Rate (WER) for an explanation of the metric.

Relevant papers might be

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Phoneme-Level

Language

Natural Language Processing (NLP) deals with how to represent language. It is related and often a part of ASR.

NLP benchmarks use perplexity to measure how good a result is.

Translation

The BLEU score is used to measure how good a translation system is.

Another score is the Translation Edit Rate (TER) introduced by Snover et al., 2006.

Matrix completion

Collaborative filtering is an application of matrix completion. More datasets are on entaroadun/gist:1653794.

Reinforcment Learning

The OpenAI Gym offers many environments for testing RL algorithms.

Control

See also

• Are we there yet ?
• Some state-of-the-arts in natural language processing and their discussion
• aclweb.org: State of the art - NLP tasks
• wer_are_we: SotA in ASR
• github.com/michalwols/ml-sota

More datasets

• List of datasets for machine learning research
• traffic-signs-dataset
• Stanford Dogs
• Awesome Public Datasets
• archive.ics.uci.edu/ml/datasets.html
• Tiny ImageNet Visual Recognition Challenge