A little agent starts out knowing nothing. Each cell of the maze is one of Williams' stochastic units. After every run, the single reinforcement signal it gets nudges the weights of every unit it used. Eventually it finds the way.
Click a cell to cycle it between empty, wall and pit. The agent adapts as it goes.
| # | r | α | curio. | b | maze | 1st | 50% | 90% | best |
|---|
Every Reset brain starts a new attempt, so change settings first and then reset. 1st = first goal, best = first run along the shortest route. * marks a milestone reached after a setting or the maze changed, so it doesn't reflect the settings shown.
Williams (1992) named REINFORCE after its own update rule: REward Increment = Nonnegative Factor × Offset Reinforcement × Characteristic Eligibility. Everything on this page maps onto those pieces.
Williams leaves the choice of r to the task, and it matters more than anything else on this page. The reinforcement r switch has three versions, each fixing the trap the one before it falls into:
Try it: load the Pit field, pick naive, and press Reset brain. Or click cells to build a winding path through a sea of pits and compare time-aware with warmer/colder. A "Looks stuck" note appears when the signal has gone flat.
One addition: a small entropy bonus that nudges each unit's distribution back toward even. It isn't in the 1992 paper. It's the entropy-maximization strategy Williams and Peng explored in a companion paper. With warmer/colder scoring the default maze is solved without it, because the slope toward the goal does the exploring. With the other two scorings this maze is almost never solved without it: pits are easy to stumble into, so the units learn to fear them long before a random walk ever reaches the goal, and then the agent stops exploring.
Because every unit gets the same end-of-episode r, credit is spread across the whole trajectory, detours included. That's why the agent's runs are wobbly even once it reliably succeeds. Later variants sharpen this with discounting, so steps taken closer to the outcome get more of the credit, but that goes beyond the original paper.
Papers: Ronald J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning 8, 229–256 (1992). Ronald J. Williams and Jing Peng, “Function optimization using connectionist reinforcement learning algorithms”, Connection Science (1991). Everything here runs in your browser in a few hundred lines of plain JavaScript.