Was using an AABB physics engine a good idea?

Totally. https://ldjam.com/events/ludum-dare/39/wayside-a-little-robots-journey-in-search-of-an-end