I wanna design a self-play capable environment that’s as simple as possible, where the outcome of agent actions are affected by the other agent(s)’s decisions, but is a control system environment rather than the usual discrete-time game, and preferably doesn’t need to use a physics engine.
My mind immediately jumped to Bennett Foddy’s Get On Top, a 2D wrestling game with only 3-button control(per player). As I was pondering how to make it even simpler by not needing a physics engine, I came up with the idea of 2 agents that each control one wheel of a differential drive robot. Each agent wants to bring the robot to its corresponding half of the arena, where it gains reward per timestep spent there by the robot’s center. I’d also like to make the edges wrap around like a torus, to give losing agents a fighting chance, while challenging the winner not to stray too far to its side. This might become an adjustable setting.
To make the environment playable the “normal” RL way, I might include a simple “kinesis AI” agent which moves forward or backward when in its opponent’s area, and stays put otherwise. A possible issue was the loss of one player’s agency when the robot sits in the middle of its opponent’s scoring area, when said opponent only needs to output a motor torque/speed of 0. To remedy this, I can set the wheelbase to be wider than the narrowest part of the scoring zone, so the “sit in place” strategy can easily be defeated.
An alternative version would be two agents that each control the horizontal or vertical thruster of a block, then try to move said block to its target quadrant.
No code or pictures for now, just wanna put the idea out there.
Exactly What It Says On The Tin 