RL Agents

Twitter-Style Explanations

DQN

Like Q-Learning, but represent the current q-function by a neural network as function approximator.

SARSA

Initialize the Q-Function $Q: \mathcal{X} \times \mathcal{A} \rightarrow \mathbb{R}$ randomly, adjust it by time. See [Pseudocode](https://martin-thoma.com/probabilistische-planung/#sarsa)

DDPG

?

NAF

?

Comparisons

The Code is on Github.

CartPole-v0

The CartPole-v0 environemnt has 2 actions: move the paddle to the right or to the left. A reward of +1 is given is the pole is upright. The episode is finished when the pole is more than 15 degrees from vertical or moves more than 2.4 units from the center.

CartPole-v0 defines "solving" as getting average reward of 195.0 over 100 consecutive trials.

The bigger NN is

model = Sequential()
model.add(Flatten(input_shape=input_shape))
model.add(Dense(16))
model.add(Activation("relu"))
model.add(Dense(16))
model.add(Activation("relu"))
model.add(Dense(16))
model.add(Activation("relu"))
model.add(Dense(nb_actions))
model.add(Activation("softmax"))

Ok, so the bigger network is important. Also, 1000 training steps are not enough, but 10000 are. Let's see if we can reduce the episode memory. The episode memory is what is used for training.

CartPole-v1

All environments

You can list all environments with

#!/usr/bin/env python

"""Print OpenAI Gym Environment data."""

import gym
from gym import envs

envids = [spec.id for spec in envs.registry.all()]
print('<table class="table">')
for i, envid in enumerate(sorted(envids), start=1):
    try:
        env = gym.make(envid)
        observations = env.observation_space
        actions = env.action_space
    except:
        observations = "Error"
        actions = "Error"
    print(
        '<tr><td id="env-{i}">{i}</td>'
        '<td><a href="https://gym.openai.com/envs/{envid}/" id="{envid}">'
        "{envid}</a></td>"
        "<td>{observations}</td>"
        "<td>{actions}</td></tr>".format(
            i=i, envid=envid, observations=str(observations), actions=str(actions)
        )
    )
print("</table>")

which gives

To summarize it:

• Most environment have an continous observation space and a discrete action space
• The action space often has about 20 possibilities. The maximum is KellyCoinflip-v0 with 25000 actions.
• The observation space has at least one input (OneRoundNondeterministicReward-v0) and never more than 500 (Taxi-v2)
• The observation space shape of (210, 160, 3) is so common (264 times!) because it is the screen of an Atari game.

Other

• Comparison of DQN performance with linear function approximator