Martin Thoma A blog about Code, the Web and Cyberculture

NIPS 2016

· by

Contents

The Conference and Workshop on Neural Information Processing Systems (NIPS) is probably the biggest conference with machine learning / deep learning as a main topic. This year, about 6000 people attended it. My friend Marvin and me were supported by Begabtenstiftung Informatik Karlsruhe.

The complete program can be found in the Conference Book, but I would like to point out some of my highlights.

Hot Topics

To get an idea what NIPS 2016 was about, I generated a word cloud from the titles of the accepted papers:

Wordcloud of NIPS titles of 2016

The organization team made something similar, but they had access to the information how the papers were tagged:

NIPS 2016 areas submitted

(Image source: www.tml.cs.uni-tuebingen.de)

The top 10 topics were:

  1. 01: Deep Learning or Neural Networks
  2. 42: (Application) Computer Vision
  3. 02: Large Scale Learning and Big Data
  4. 05: Learning Theory
  5. 53: (Other) Optimization
  6. 08: Sparsity and Feature Selection
  7. 51: (Other) Classification
  8. 03: Convex Optimization
  9. 54: (Other) Probabilistic Models and Methods
  10. 56: (Other) Unsupervised Learning Methods

I would say the top five hot topics (first is hottest) are:

  1. GANs
  2. Reinforcment Learning: Look for "bandit" in the paper titles
  3. unsupervised learning
  4. alternative ways to train DNNs
  5. reducing the need for data (transfer learning, domain adaptation, semi-supervised learning)

GANs

Generative Adverserial Networks (short: GANs) were one hot topic at NIPS. The idea is to train two networks: A generator $G$ and a discriminator $D$. The generator creates content (e.g. images) and the discriminator has to decide if the content is of the natural distribution (the training set) or made by the generator.

An introduction can be found at blog.evjang.com

Noteworthy papers and ideas are:

Applications of GANs are (according to Eric Jang):

  • reinforcement learning: paper
  • domain adaptation
  • security ML
  • compression

Bayesian Deep Learning

Combinding deep learning with graphical models like CRFs / Markov Random Fields has been done for semantic segmentation for a while now. It seems like the combination of those two is called "bayesian deep learning". If you look for the keyword "variational" it seems to belong in this category.

I don't really know this area, so I leav it to Eric Jang to point out important papers.

Nut's and Bolts of ML

Andrew Ng gave a talk in which he summarized what he thinks are some of the most important topics when training machine learning systems. Most of it is probably also in his book Machine Learning Yearning or in his Coursera course.

Here are some of the things he talked about:

  • When you design a speech recognition system, you can measure 3 types of errors: Human error, training set error and test set error. The difference between the human error and the training error is "avoidable error" (bias), the difference between training and test error is "variance".
  • Human level performance is ambiguous: In a medical application, is an amateur, a doctor, an experienced doctor or a team of (experienced) doctors the "human level performance"?
  • Role of an "AI Product Manager"

More Papers

Clustering:

Optimization:

Theory:

Topology learning:

Analysis of ML models:

Content creation:

Labeling:

Content based Image Retrival (CBIR):

Misc:

Lessons learned for conferences

  • Bring a camera: The information comes very fast. Too fast to take notes, but you can shoot a photo of the slides. In fact, quite a lot of people do so.
  • Shoot a photo of the first slide, so that you know what the talk was about when you look at your slides.
  • If you give a talk / poster...
    • ... let the first slide be there long enough, so that people can take a photo of it.
    • ... or have a URL / the title of the paper on every single slide
    • ... let every slide be visible long enough, so that people can take photos
    • ... don't use QR-codes only, but also (shortened) URLs
    • ... make it available online as PDF
    • ... answer key questions: (1) Which problem did you tackle? (2) How did you test your results? (3) To what is your "solution" similar?
  • As a session organizer...
    • ... make sure there is a schedule at the door (outside)
    • ... make sure the schedule is online
    • ... make sure the schedule is changed everywhere, if it is actually changed

Miscallenious