Refactor reward signals into separate class (#2144)

* Create new class (RewardSignal) that represents a reward signal. 
* Add value heads for each reward signal in the PPO model.
* Make summaries agnostic to the type of reward signals, and log weighted rewards per reward signal. 
* Move extrinsic and curiosity rewards into this new structure.
* Allow defining multiple reward signals in YAML file. Add documentation for this new structure.

Author: Ervin T

config/trainer_config.yaml

@@ -4,7 +4,6 @@ default:
     beta: 5.0e-3
     buffer_size: 10240
     epsilon: 0.2
-    gamma: 0.99
     hidden_units: 128
     lambd: 0.95
     learning_rate: 3.0e-4
@@ -17,14 +16,15 @@ default:
     sequence_length: 64
     summary_freq: 1000
     use_recurrent: false
-    use_curiosity: false
-    curiosity_strength: 0.01
-    curiosity_enc_size: 128
+    reward_signals: 
+        extrinsic:
+            strength: 1.0
+            gamma: 0.99
 
 BananaLearning:
     normalize: false
-    batch_size: 1024
     beta: 5.0e-3
+    batch_size: 1024
     buffer_size: 10240
     max_steps: 1.0e5
 
@@ -93,10 +93,7 @@ GoalieLearning:
     normalize: false
 
 PyramidsLearning:
-    use_curiosity: true
     summary_freq: 2000
-    curiosity_strength: 0.01
-    curiosity_enc_size: 256
     time_horizon: 128
     batch_size: 128
     buffer_size: 2048
@@ -105,11 +102,16 @@ PyramidsLearning:
     beta: 1.0e-2
     max_steps: 5.0e5
     num_epoch: 3
-
+    reward_signals:
+        extrinsic: 
+            strength: 1.0
+            gamma: 0.99
+        curiosity:
+            strength: 0.02
+            gamma: 0.99
+            encoding_size: 256
+            
 VisualPyramidsLearning:
-    use_curiosity: true
-    curiosity_strength: 0.01
-    curiosity_enc_size: 256
     time_horizon: 128
     batch_size: 64
     buffer_size: 2024
@@ -118,6 +120,14 @@ VisualPyramidsLearning:
     beta: 1.0e-2
     max_steps: 5.0e5
     num_epoch: 3
+    reward_signals:
+        extrinsic: 
+            strength: 1.0
+            gamma: 0.99
+        curiosity:
+            strength: 0.01
+            gamma: 0.99
+            encoding_size: 256
 
 3DBallLearning:
     normalize: true
@@ -126,7 +136,6 @@ VisualPyramidsLearning:
     summary_freq: 1000
     time_horizon: 1000
     lambd: 0.99
-    gamma: 0.995
     beta: 0.001
 
 3DBallHardLearning:
@@ -136,8 +145,11 @@ VisualPyramidsLearning:
     summary_freq: 1000
     time_horizon: 1000
     max_steps: 5.0e5
-    gamma: 0.995
     beta: 0.001
+    reward_signals:
+        extrinsic: 
+            strength: 1.0
+            gamma: 0.995
 
 TennisLearning:
     normalize: true
@@ -149,35 +161,44 @@ CrawlerStaticLearning:
     time_horizon: 1000
     batch_size: 2024
     buffer_size: 20240
-    gamma: 0.995
     max_steps: 1e6
     summary_freq: 3000
     num_layers: 3
     hidden_units: 512
+    reward_signals:
+        extrinsic: 
+            strength: 1.0
+            gamma: 0.995
 
 CrawlerDynamicLearning:
     normalize: true
     num_epoch: 3
     time_horizon: 1000
     batch_size: 2024
     buffer_size: 20240
-    gamma: 0.995
     max_steps: 1e6
     summary_freq: 3000
     num_layers: 3
     hidden_units: 512
+    reward_signals:
+        extrinsic: 
+            strength: 1.0
+            gamma: 0.995
 
 WalkerLearning:
     normalize: true
     num_epoch: 3
     time_horizon: 1000
     batch_size: 2048
     buffer_size: 20480
-    gamma: 0.995
     max_steps: 2e6
     summary_freq: 3000
     num_layers: 3
     hidden_units: 512
+    reward_signals:
+        extrinsic: 
+            strength: 1.0
+            gamma: 0.995
 
 ReacherLearning:
     normalize: true
@@ -196,7 +217,6 @@ HallwayLearning:
     hidden_units: 128
     memory_size: 256
     beta: 1.0e-2
-    gamma: 0.99
     num_epoch: 3
     buffer_size: 1024
     batch_size: 128

docs/Training-PPO.md

@@ -7,6 +7,10 @@ observations to the best action an agent can take in a given state. The
 ML-Agents PPO algorithm is implemented in TensorFlow and runs in a separate
 Python process (communicating with the running Unity application over a socket).
 
+To train an agent, you will need to provide the agent one or more reward signals which
+the agent should attempt to maximize. See [Reward Signals](Training-RewardSignals.md)
+for the available reward signals and the corresponding hyperparameters.
+
 See [Training ML-Agents](Training-ML-Agents.md) for instructions on running the
 training program, `learn.py`.
 
@@ -31,15 +35,18 @@ of performance you would like.
 
 ## Hyperparameters
 
-### Gamma
+### Reward Signals
 
-`gamma` corresponds to the discount factor for future rewards. This can be
-thought of as how far into the future the agent should care about possible
-rewards. In situations when the agent should be acting in the present in order
-to prepare for rewards in the distant future, this value should be large. In
-cases when rewards are more immediate, it can be smaller.
+In reinforcement learning, the goal is to learn a Policy that maximizes reward.
+At a base level, the reward is given by the environment. However, we could imagine
+rewarding the agent for various different behaviors. For instance, we could reward
+the agent for exploring new states, rather than just when an explicit reward is given.
+Furthermore, we could mix reward signals to help the learning process.
 
-Typical Range: `0.8` - `0.995`
+`reward_signals` provides a section to define [reward signals.](Training-RewardSignals.md)
+ML-Agents provides two reward signals by default, the Extrinsic (environment) reward, and the
+Curiosity reward, which can be used to encourage exploration in sparse extrinsic reward
+environments. 
 
 ### Lambda
 
@@ -184,30 +191,6 @@ the agent will need to remember in order to successfully complete the task.
 
 Typical Range: `64` - `512`
 
-## (Optional) Intrinsic Curiosity Module Hyperparameters
-
-The below hyperparameters are only used when `use_curiosity` is set to true.
-
-### Curiosity Encoding Size
-
-`curiosity_enc_size` corresponds to the size of the hidden layer used to encode
-the observations within the intrinsic curiosity module. This value should be
-small enough to encourage the curiosity module to compress the original
-observation, but also not too small to prevent it from learning the dynamics of
-the environment.
-
-Typical Range: `64` - `256`
-
-### Curiosity Strength
-
-`curiosity_strength` corresponds to the magnitude of the intrinsic reward
-generated by the intrinsic curiosity module. This should be scaled in order to
-ensure it is large enough to not be overwhelmed by extrinsic reward signals in
-the environment. Likewise it should not be too large to overwhelm the extrinsic
-reward signal.
-
-Typical Range: `0.1` - `0.001`
-
 ## Training Statistics
 
 To view training statistics, use TensorBoard. For information on launching and

docs/Training-RewardSignals.md

@@ -0,0 +1,114 @@
+# Reward Signals
+
+In reinforcement learning, the end goal for the Agent is to discover a behavior (a Policy)
+that maximizes a reward. Typically, a reward is defined by your environment, and corresponds
+to reaching some goal. These are what we refer to as "extrinsic" rewards, as they are defined
+external of the learning algorithm.
+
+Rewards, however, can be defined outside of the enviroment as well, to encourage the agent to
+behave in certain ways, or to aid the learning of the true extrinsic reward. We refer to these
+rewards as "intrinsic" reward signals. The total reward that the agent will learn to maximize can
+be a mix of extrinsic and intrinsic reward signals.
+
+ML-Agents allows reward signals to be defined in a modular way, and we provide three reward
+signals that can the mixed and matched to help shape your agent's behavior. The `extrinsic` Reward
+Signal represents the rewards defined in your environment, and is enabled by default.
+The `curiosity` reward signal helps your agent explore when extrinsic rewards are sparse.
+
+## Enabling Reward Signals
+
+Reward signals, like other hyperparameters, are defined in the trainer config `.yaml` file. An
+example is provided in `config/trainer_config.yaml`. To enable a reward signal, add it to the
+`reward_signals:` section under the brain name. For instance, to enable the extrinsic signal
+in addition to a small curiosity reward, you would define your `reward_signals` as follows:
+
+```yaml
+reward_signals:
+    extrinsic:
+        strength: 1.0
+        gamma: 0.99
+    curiosity:
+        strength: 0.01
+        gamma: 0.99
+        encoding_size: 128
+```
+
+Each reward signal should define at least two parameters, `strength` and `gamma`, in addition
+to any class-specific hyperparameters. Note that to remove a reward signal, you should delete
+its entry entirely from `reward_signals`. At least one reward signal should be left defined
+at all times.
+
+## Reward Signal Types
+
+### The Extrinsic Reward Signal
+
+The `extrinsic` reward signal is simply the reward given by the
+[environment](Learning-Environment-Design.md). Remove it to force the agent
+to ignore the environment reward.
+
+#### Strength
+
+`strength` is the factor by which to multiply the raw
+reward. Typical ranges will vary depending on the reward signal.
+
+Typical Range: `1.0`
+
+#### Gamma
+
+`gamma` corresponds to the discount factor for future rewards. This can be
+thought of as how far into the future the agent should care about possible
+rewards. In situations when the agent should be acting in the present in order
+to prepare for rewards in the distant future, this value should be large. In
+cases when rewards are more immediate, it can be smaller.
+
+Typical Range: `0.8` - `0.995`
+
+### The Curiosity Reward Signal
+
+The `curiosity` Reward Signal enables the Intrinsic Curiosity Module. This is an implementation 
+of the approach described in "Curiosity-driven Exploration by Self-supervised Prediction" 
+by Pathak, et al. It trains two networks:
+* an inverse model, which takes the current and next obersvation of the agent, encodes them, and
+uses the encoding to predict the action that was taken between the observations
+* a forward model, which takes the encoded current obseravation and action, and predicts the
+next encoded observation.
+
+The loss of the forward model (the difference between the predicted and actual encoded observations) is used as the intrinsic reward, so the more surprised the model is, the larger the reward will be.
+
+For more information, see
+* https://arxiv.org/abs/1705.05363
+* https://pathak22.github.io/noreward-rl/
+* https://blogs.unity3d.com/2018/06/26/solving-sparse-reward-tasks-with-curiosity/
+
+#### Strength 
+
+In this case, `strength` corresponds to the magnitude of the curiosity reward generated 
+by the intrinsic curiosity module. This should be scaled in order to ensure it is large enough 
+to not be overwhelmed by extrinsic reward signals in the environment. 
+Likewise it should not be too large to overwhelm the extrinsic reward signal.
+
+Typical Range: `0.001` - `0.1`
+
+#### Gamma
+
+`gamma` corresponds to the discount factor for future rewards.
+
+Typical Range: `0.8` - `0.995`
+
+#### Encoding Size
+
+`encoding_size` corresponds to the size of the encoding used by the intrinsic curiosity model.
+This value should be small enough to encourage the ICM to compress the original
+observation, but also not too small to prevent it from learning to differentiate between
+demonstrated and actual behavior.
+
+Default Value: 64
+Typical Range: `64` - `256`
+
+#### Learning Rate
+
+`learning_rate` is the learning rate used to update the intrinsic curiosity module. 
+This should typically be decreased if training is unstable, and the curiosity loss is unstable.
+
+Default Value: `3e-4`
+Typical Range: `1e-5` - `1e-3`  

ml-agents/mlagents/trainers/bc/policy.py

@@ -57,7 +57,7 @@ def evaluate(self, brain_info):
             self.model.sequence_length: 1,
         }
 
-        feed_dict = self._fill_eval_dict(feed_dict, brain_info)
+        feed_dict = self.fill_eval_dict(feed_dict, brain_info)
         if self.use_recurrent:
             if brain_info.memories.shape[1] == 0:
                 brain_info.memories = self.make_empty_memory(len(brain_info.agents))

ml-agents/mlagents/trainers/components/__init__.py

@@ -0,0 +1 @@
+from .reward_signal import *

ml-agents/mlagents/trainers/components/reward_signals/__init__.py

@@ -0,0 +1 @@
+from .signal import CuriosityRewardSignal