OK, the game detects when a row is selected and close enough to be removed.
Yet, it’s a damn bunch of bugs. I guess my setup is to complex, and 3D physics is not really suitable for this

Anyway, here’s an updated binary :

http://astrofra.com/posts/ld15/binary/