Reinforcement learning is a sub-field of mathematics and computer science. It deals with the following kind of problems: You're given a set of states \(\mathcal{X} \subseteq \mathbb{R}^n\) and a starting state \(x_0 \in \mathcal{X}\). For every time step \(k = 0, 1, 2, \dots\) you have a set of possible actions, depending on your current state:

Depending on what your action and your current state is, the new state is

So the transition from state \(x_k\) to state \(x_{k+1}\) with action \(a_k\) is stochastic.

For some states \(x_k\), actions \(a_k\) at time \(k\), you receive rewards:

Your goal is to maximize

where \(\gamma in (0, 1)\) is a discounting factor which makes sure we don't get infinite rewards. \(\gamma = 0.99\) is a typical choice.

## Applications

This very general problem description can be applied in almost any scenario:

- Learning automatically to play games
- Learning to control robots

## Why RL is difficult

- Credit assignment: In chess, you only get a reward (positiv or negative) at the end of the game. How to you tell which move was good or bad?
- Exploration vs. exploitation: When should you stick to what you know and when should you try something new?
- State equivalence: Typically, your state is very high-dimensional. For example
when learning very old computer games from raw pixels you have
$$210 \cdot 160 \cdot 3 = 100800$$dimensions in your feature vector. But the relevant game states might be much less.

## Approaches

A **policy network** gets the state as input and outputs the action. It learns
by executing many episodes (e.g. a complete game; from start until you reach a
final state or at least a state with reward) and labels all actions before with
the received reward. There might be many which were good even in a lost game,
but in average you expect to punish bad decisions and encourage good decisions.

## Resources

If you are a student at KIT, I can recommend to visit the lecture Probabilistic Planning.

Other resources you might want to have a look at:

- Deep Reinforcement Learning: Pong from Pixels
- Guest Post (Part I): Demystifying Deep Reinforcement Learning
- Human-level control through deep reinforcement learning by V. Mnih et al.
- Deterministic Policy Gradient Algorithms