Simple Maze Experiment – Part 1

Overview

Now that I’ve got a basic understanding of how to use Unity’s reinforcement learning, the next step is to put it into practice. To do so, I’m going to go with a very simple concept. An agent will be placed in a maze and will need to find the quickest route to the exit.

Normally I’d use some form of pathfinding for this, most likely the built in Unity pathfinding tool. The main reason for this is later on I plan to add hazards and puzzles to the maze for the agent to solve, and I want it to handle it like a player would, rather than needing to write explicit code to handle each puzzle. I’m also curious as to how well ML will handle visual inputs from a camera to navigate, rather than a raycast system.

So here’s my current battle plan:

  • Create a simple maze with an entrance and an exit.
  • The agent starts at one end, and must reach the other.
  • For randomisation, the start and exit points will be swapped every so often.
  • The agent will navigate using a camera, and four buttons, W, A, S and D. W and S will move the agent forward and back, while A and S will rotate left and right.

I’m toying with the idea of giving the capsule the ability to strafe, and then rotate seperately, but I’ll probably revisit that later once the basics work as expected.

Unity Setup

To keep it simple, I’m using Unity 2018.4.2f2 and trying to keep the extras to a minimum, the only extra items I have are:

  • Probuilder (for rapid prototyping of the maze).
  • JSON .NET (Required for TensorflowSharp functionality, though I only needed the Newtonsoft.json.dll from the Standalone folder.)
  • The Unity TensorSharp pack (availaible from here)
  • The following folders from the ML-Agents project:
    • unity-environment\assets\ML-Agents\
      • Scripts (Contains the parent ML scripts that we will need to extend upon).
      • Editor (Contains the editor tweaks to better display options on the ML scripts from the Scripts folder).

In addition to these two, I’ve also set the build settings for PC, Mac & Linux to use Experimental .NET 4.6 support (Edit > Project Settings > Player > Configuration > Scripting Runtime Version)  and added ENABLE_TENSORFLOW to the Scripting Define Symbols* field.

With this done, I prototyped a quick maze in Blender, and imported it into the game world, then I spawned a basic capsule collider, gave it a rigidbody, and positioned it at the start of the maze.

With the editor set up, the next step is to configure the Academy, Brain and Agent scripts ready for use. I’ll cover those off in part 2.