Merge branch 'develop-barracuda' into develop-barracuda-test

* develop-barracuda:
  Backup and restore fixedDeltaTime and maximumDeltaTime on Academy init / shutdown
  Restore global gravity value when Academy gets destroyed
  deleted dead meta file and added a note on the OpenGLCore Graphics API
  Barracuda : Updating the documentation (#1607)
  Remove env creation logic from TrainerController (#1562)
  Fix In editor Docker training (#1582)
  Only using multiprocess when --num-runs>1 (#1583)
  Replace AddVectorObs(float[]) and AddVectorObs(List<float>) with a more generic AddVectorObs(IEnumerable<float>) (#1540)
  fixed the windows ctrl-c bug (#1558)
  Improve Gym wrapper compatibility and add Dopamine documentation (#1541)
  Fix typo in documentation (#1516)
  Update curricula brain names for 0.6
  Addressing #1537
  Fix for divide-by-zero error with Discrete Actions  (#1520)
  Documentation tweaks and updates (#1479)

Author: mantasp

UnitySDK/Assets/ML-Agents/Plugins/Barracuda.Core/Tools.meta

@@ -755,21 +755,11 @@ protected void AddVectorObs(Vector2 observation)
         }
 
         /// <summary>
-        /// Adds a float array observation to the vector observations of the agent.
-        /// Increases the size of the agents vector observation by size of array.
+        /// Adds a collection of float observations to the vector observations of the agent.
+        /// Increases the size of the agents vector observation by size of the collection.
         /// </summary>
         /// <param name="observation">Observation.</param>
-        protected void AddVectorObs(float[] observation)
-        {
-            info.vectorObservation.AddRange(observation);
-        }
-
-        /// <summary>
-        /// Adds a float list observation to the vector observations of the agent.
-        /// Increases the size of the agents vector observation by size of list.
-        /// </summary>
-        /// <param name="observation">Observation.</param>
-        protected void AddVectorObs(List<float> observation)
+        protected void AddVectorObs(IEnumerable<float> observation)
         {
             info.vectorObservation.AddRange(observation);
         }

UnitySDK/Assets/ML-Agents/Scripts/Agent.cs

@@ -411,8 +411,8 @@ private string CheckVisualObsShape(Tensor tensor, int visObsIndex)
             var widthBp = resolutionBp.width;
             var heightBp = resolutionBp.height;
             var pixelBp = resolutionBp.blackAndWhite ? 1 : 3;
-            var widthT = tensor.Shape[1];
-            var heightT = tensor.Shape[2];
+            var heightT = tensor.Shape[1];
+            var widthT = tensor.Shape[2];
             var pixelT = tensor.Shape[3];
             if  ((widthBp != widthT) || (heightBp != heightT) || (pixelBp != pixelT))
             {

UnitySDK/Assets/ML-Agents/Scripts/InferenceBrain/ModelParamLoader.cs

@@ -16,7 +16,7 @@ to TensorFlow-related tools that we leverage within the ML-Agents toolkit.
 performing computations using data flow graphs, the underlying representation of
 deep learning models. It facilitates training and inference on CPUs and GPUs in
 a desktop, server, or mobile device. Within the ML-Agents toolkit, when you
-train the behavior of an agent, the output is a TensorFlow model (.bytes) file
+train the behavior of an agent, the output is a TensorFlow model (.nn) file
 that you can then embed within a Learning Brain. Unless you implement a new
 algorithm, the use of TensorFlow is mostly abstracted away and behind the
 scenes.
@@ -36,18 +36,3 @@ documentation, but, in the meantime, if you are unfamiliar with TensorBoard we
 recommend our guide on [using Tensorboard with ML-Agents](Using-Tensorboard.md) or
 this [tutorial](https://github.com/dandelionmane/tf-dev-summit-tensorboard-tutorial).
 
-## TensorflowSharp
-
-One of the drawbacks of TensorFlow is that it does not provide a native C# API.
-This means that the Learning Brain is not natively supported since Unity scripts
-are written in C#. Consequently, to enable the Learning Brain, we leverage a
-third-party library
-[TensorFlowSharp](https://github.com/migueldeicaza/TensorFlowSharp) which
-provides .NET bindings to TensorFlow. Thus, when a Unity environment that
-contains a Learning Brain is built, inference is performed via TensorFlowSharp.
-We provide an additional in-depth overview of how to leverage
-[TensorFlowSharp within Unity](Using-TensorFlow-Sharp-in-Unity.md)
-which will become more
-relevant once you install and start training behaviors within the ML-Agents
-toolkit. Given the reliance on TensorFlowSharp, the Learning Brain is currently
-marked as experimental.

config/curricula/push-block/PushBlockBrain.json

@@ -25,32 +25,12 @@ Unity settings.
        Equivalent or .NET 4.x Equivalent)**
 6. Go to **File** > **Save Project**
 
-## Setting up TensorFlowSharp
-
-We provide pre-trained models (`.bytes` files) for all the agents 
-in all our demo environments. To be able to run those models, you'll 
-first need to set-up TensorFlowSharp support. Consequently, you need to install 
-the TensorFlowSharp plugin to be able to run these models within the Unity 
-Editor. 
-
-1. Download the [TensorFlowSharp Plugin](https://s3.amazonaws.com/unity-ml-agents/0.5/TFSharpPlugin.unitypackage)
-2. Import it into Unity by double clicking the downloaded file. You can check 
-if it was successfully imported by checking the
-TensorFlow files in the Project window under **Assets** > **ML-Agents** >
-**Plugins** > **Computer**.
-3. Go to **Edit** > **Project Settings** > **Player** and add `ENABLE_TENSORFLOW`
-to the `Scripting Define Symbols` for each type of device you want to use 
-(**`PC, Mac and Linux Standalone`**, **`iOS`** or **`Android`**).
-
-   ![Project Settings](images/project-settings.png)
-
-   **Note**: If you don't see anything under **Assets**, drag the
-   `UnitySDK/Assets/ML-Agents` folder under **Assets** within Project window.
-
-   ![Imported TensorFlowsharp](images/imported-tensorflowsharp.png)
-
 ## Running a Pre-trained Model
-We've included pre-trained models for the 3D Ball example.
+
+We include pre-trained models for our agents (`.nn` files) and we use the 
+[Unity Inference Engine](Unity-Inference-Engine.md) to run these models 
+inside Unity. In this section, we will use the pre-trained model for the 
+3D Ball example.
 
 1. In the **Project** window, go to the `Assets/ML-Agents/Examples/3DBall/Scenes` folder
    and open the `3DBall` scene file.
@@ -74,7 +54,9 @@ We've included pre-trained models for the 3D Ball example.
    folder.
 7. Drag the `3DBallLearning` model file from the `Assets/ML-Agents/Examples/3DBall/TFModels` 
    folder to the **Model** field of the **3DBallLearning** Brain in the **Inspector** window. __Note__ : All of the brains should now have `3DBallLearning` as the TensorFlow model in the `Model` property 
-8. Click the **Play** button and you will see the platforms balance the balls
+8. Select the **InferenceDevice** to use for this model (CPU or GPU). 
+   _Note: CPU is faster for the majority of ML-Agents toolkit generated models_
+9. Click the **Play** button and you will see the platforms balance the balls
    using the pretrained model.
 
    ![Running a pretrained model](images/running-a-pretrained-model.gif)
@@ -93,7 +75,7 @@ More information and documentation is provided in the
 
 ## Training the Brain with Reinforcement Learning
 
-### Setting up the enviornment for training
+### Setting up the environment for training
 
 To set up the environment for training, you will need to specify which agents are contributing
 to the training and which Brain is being trained. You can only perform training with
@@ -240,7 +222,7 @@ INFO:mlagents.trainers: first-run-0: 3DBallLearning: Step: 10000. Mean Reward: 2
 ### After training
 
 You can press Ctrl+C to stop the training, and your trained model will be at
-`models/<run-identifier>/<brain_name>.bytes` where
+`models/<run-identifier>/<brain_name>.nn` where
 `<brain_name>` is the name of the Brain corresponding to the model.
 (**Note:** There is a known bug on Windows that causes the saving of the model to
 fail when you early terminate the training, it's recommended to wait until Step
@@ -254,7 +236,7 @@ the steps described
    `UnitySDK/Assets/ML-Agents/Examples/3DBall/TFModels/`.
 2. Open the Unity Editor, and select the **3DBall** scene as described above.
 3. Select the  **3DBallLearning** Learning Brain from the Scene hierarchy.
-5. Drag the `<brain_name>.bytes` file from the Project window of
+5. Drag the `<brain_name>.nn` file from the Project window of
    the Editor to the **Model** placeholder in the **3DBallLearning**
    inspector window.
 6. Press the :arrow_forward: button at the top of the Editor.

config/curricula/wall-jump/BigWallBrain.json renamed to config/curricula/wall-jump/BigWallJumpLearning.json

@@ -224,6 +224,10 @@ The `--train` flag tells the ML-Agents toolkit to run in training mode.
 follow the instructions in
 [Using an Executable](Learning-Environment-Executable.md).
 
+**Note**: Re-running this command will start training from scratch again. To resume
+a previous training run, append the `--load` flag and give the same `--run-id` as the
+run you want to resume.
+
 ### Observing Training Progress
 
 Once you start training using `mlagents-learn` in the way described in the
@@ -269,19 +273,9 @@ Once the training process completes, and the training process saves the model
 use it with Agents having a **Learning Brain**.
 __Note:__ Do not just close the Unity Window once the `Saved Model` message appears. 
 Either wait for the training process to close the window or press Ctrl+C at the 
-command-line prompt. If you close the window manually, the `.bytes` file 
+command-line prompt. If you close the window manually, the `.nn` file 
 containing the trained model is not exported into the ml-agents folder.
 
-### Setting up TensorFlowSharp
-
-Because TensorFlowSharp support is still experimental, it is disabled by
-default. Please note that the `Learning` Brain inference can only be used with
-TensorFlowSharp.
- 
-To set up the TensorFlowSharp Support, follow [Setting up ML-Agents Toolkit
-within Unity](Basic-Guide.md#setting-up-ml-agents-within-unity) section. of the
-Basic Guide page.
-
 ### Embedding the trained model into Unity
 
 To embed the trained model into Unity, follow the later part of [Training the

config/curricula/wall-jump/SmallWallBrain.json renamed to config/curricula/wall-jump/SmallWallJumpLearning.json

@@ -154,7 +154,8 @@ public class RollerAcademy : Academy { }
 
 The default settings for the Academy properties are also fine for this
 environment, so we don't need to change anything for the RollerAcademy component
-in the Inspector window.
+in the Inspector window. You may not have the RollerBrain in the Broadcast Hub yet, 
+more on that later. 
 
 ![The Academy properties](images/mlagents-NewTutAcademy.png)
 
@@ -547,6 +548,44 @@ you pass to the `mlagents-learn` command for each training run. If you use
 the same id value, the statistics for multiple runs are combined and become 
 difficult to interpret.
 
+## Optional: Multiple Training Areas within the Same Scene
+
+In many of the [example environments](Learning-Environment-Examples.md), many copies of 
+the training area are instantiated in the scene. This generally speeds up training,
+allowing the environment to gather many experiences in parallel. This can be achieved
+simply by instantiating many Agents which share the same Brain. Use the following steps to
+parallelize your RollerBall environment.  
+
+### Instantiating Multiple Training Areas
+
+1. Right-click on your Project Hierarchy and create a new empty GameObject. 
+   Name it TrainingArea. 
+2. Reset the TrainingArea’s Transform so that it is at (0,0,0) with Rotation (0,0,0) 
+   and Scale (1,1,1). 
+3. Drag the Floor, Target, and RollerAgent GameObjects in the Hierarchy into the 
+   TrainingArea GameObject. 
+4. Drag the TrainingArea GameObject, along with its attached GameObjects, into your 
+   Assets browser, turning it into a prefab.
+5. You can now instantiate copies of the TrainingArea prefab. Drag them into your scene, 
+   positioning them so that they do not overlap. 
+
+### Editing the Scripts 
+
+You will notice that in the previous section, we wrote our scripts assuming that our 
+TrainingArea was at (0,0,0), performing checks such as `this.transform.position.y < 0` 
+to determine whether our agent has fallen off the platform. We will need to change 
+this if we are to use multiple TrainingAreas throughout the scene. 
+
+A quick way to adapt our current code is to use 
+localPosition rather than position, so that our position reference is in reference 
+to the prefab TrainingArea's location, and not global coordinates. 
+
+1. Replace all references of `this.transform.position` in RollerAgent.cs with `this.transform.localPosition`.
+2. Replace all references of `Target.position` in RollerAgent.cs with `Target.localPosition`.
+
+This is only one way to achieve this objective. Refer to the 
+[example environments](Learning-Environment-Examples.md) for other ways we can achieve relative positioning.
+
 ## Review: Scene Layout
 
 This section briefly reviews how to organize your scene when using Agents in

docs/Background-TensorFlow.md

@@ -475,6 +475,10 @@ if ((ball.transform.position.y - gameObject.transform.position.y) < -2f ||
 The `Ball3DAgent` also assigns a negative penalty when the ball falls off the
 platform.
 
+Note that all of these environments make use of the `Done()` method, which manually
+terminates an episode when a termination condition is reached. This can be 
+called independently of the `Max Step` property.
+
 ## Agent Properties
 
 ![Agent Inspector](images/agent.png)

docs/Basic-Guide.md

@@ -43,8 +43,6 @@ model.
 To use a graph model:
 
 1. Select the **Learning Brain** asset in the **Project** window of the Unity Editor.
- **Note:** In order to use the **Learning** Brain with inference, you need to have
- TensorFlowSharp enabled. Refer to [this section](Basic-Guide.md#setting-up-ml-agents-within-unity) for more information.
 2. Import the `model_name` file produced by the PPO training
    program. (Where `model_name` is the name of the model file, which is
    constructed from the name of your Unity environment executable and the run-id
@@ -54,7 +52,7 @@ To use a graph model:
    [import assets into Unity](https://docs.unity3d.com/Manual/ImportingAssets.html)
    in various ways. The easiest way is to simply drag the file into the
    **Project** window and drop it into an appropriate folder.
-3. Once the `model_name.bytes` file is imported, drag it from the **Project**
+3. Once the `model_name.nn` file is imported, drag it from the **Project**
    window to the **Model** field of the Brain component.
 
 If you are using a model produced by the ML-Agents `mlagents-learn` command, use
@@ -65,9 +63,9 @@ the default values for the other Learning Brain parameters.
 The default values of the TensorFlow graph parameters work with the model
 produced by the PPO and BC training code in the ML-Agents SDK. To use a default
 ML-Agents model, the only parameter that you need to set is the `Model`,
-which must be set to the `.bytes` file containing the trained model itself.
+which must be set to the `.nn` file containing the trained model itself.
 
-* `Model` : This must be the `.bytes` file corresponding to the pre-trained
+* `Model` : This must be the `.nn` file corresponding to the pre-trained
    TensorFlow graph. (You must first drag this file into your Project window 
    and then from the Resources folder into the inspector)
 

docs/Getting-Started-with-Balance-Ball.md

@@ -158,15 +158,16 @@ Brain assigned to this Agent must be set.
 
 You must also determine how an Agent finishes its task or times out. You can
 manually set an Agent to done in your `AgentAction()` function when the Agent
-has finished (or irrevocably failed) its task. You can also set the Agent's `Max
-Steps` property to a positive value and the Agent will consider itself done
-after it has taken that many steps. When the Academy reaches its own `Max Steps`
-count, it starts the next episode. If you set an Agent's `ResetOnDone` property
-to true, then the Agent can attempt its task several times in one episode. (Use
-the `Agent.AgentReset()` function to prepare the Agent to start again.)
+has finished (or irrevocably failed) its task by calling the `Done()` function. 
+You can also set the Agent's `Max Steps` property to a positive value and the 
+Agent will consider itself done after it has taken that many steps. When the 
+Academy reaches its own `Max Steps` count, it starts the next episode. If you 
+set an Agent's `ResetOnDone` property to true, then the Agent can attempt its 
+task several times in one episode. (Use the `Agent.AgentReset()` function to 
+prepare the Agent to start again.)
 
 See [Agents](Learning-Environment-Design-Agents.md) for detailed information
-about programing your own Agents.
+about programming your own Agents.
 
 ## Environments
 

docs/Learning-Environment-Create-New.md

@@ -201,7 +201,7 @@ INFO:mlagents.trainers: first-run-0: Ball3DLearning: Step: 10000. Mean Reward: 2
 ```
 
 You can press Ctrl+C to stop the training, and your trained model will be at
-`models/<run-identifier>/<brain_name>.bytes`, which corresponds
+`models/<run-identifier>/<brain_name>.nn`, which corresponds
 to your model's latest checkpoint. (**Note:** There is a known bug on Windows
 that causes the saving of the model to fail when you early terminate the
 training, it's recommended to wait until Step has reached the max_steps
@@ -212,7 +212,7 @@ into your Learning Brain by following the steps below:
    `UnitySDK/Assets/ML-Agents/Examples/3DBall/TFModels/`.
 2. Open the Unity Editor, and select the **3DBall** scene as described above.
 3. Select the **Ball3DLearning** object from the Project window.
-5. Drag the `<brain_name>.bytes` file from the Project window of
+5. Drag the `<brain_name>.nn` file from the Project window of
    the Editor to the **Model** placeholder in the **Ball3DLearning**
    inspector window.
 6. Remove the **Ball3DLearning** from the Academy's `Broadcast Hub`

docs/Learning-Environment-Design-Agents.md

@@ -246,10 +246,6 @@ training the Python API uses the observations it receives to learn a TensorFlow
 model. This model is then embedded within the Learning Brain during inference to
 generate the optimal actions for all Agents linked to that Brain. 
 
-**Note that our Learning Brain is currently experimental as it is limited to TensorFlow
-models and leverages the third-party
-[TensorFlowSharp](https://github.com/migueldeicaza/TensorFlowSharp) library.**
-
 The
 [Getting Started with the 3D Balance Ball Example](Getting-Started-with-Balance-Ball.md)
 tutorial covers this training mode with the **3D Balance Ball** sample environment.

docs/Learning-Environment-Design-Learning-Brains.md

@@ -42,7 +42,8 @@
 * [Using TensorBoard to Observe Training](Using-Tensorboard.md)
 
 ## Inference
-* [TensorFlowSharp in Unity (Experimental)](Using-TensorFlow-Sharp-in-Unity.md)
+
+* [Unity Inference Engine](Unity-Inference-Engine.md)
 
 ## Help
 
@@ -55,4 +56,4 @@
 
 * [API Reference](API-Reference.md)
 * [How to use the Python API](Python-API.md)
-* [Wrapping Learning Environment as a Gym](../gym-unity/README.md)
+* [Wrapping Learning Environment as a Gym (+Baselines/Dopamine Integration)](../gym-unity/README.md)