After a couple more hours, it is basically playable!

https://youtube.com/watch?v=n_kaBXRwFx4

IMO the coolest thing about this right now is the fact that the visual and audio feedback when the ball bounces of something is literally the exact same thing/data/oscillation. (The whole thing doesn't flash like in the video to the human eye btw.)

The patch has gotten quite big by now, but I am really happy with how well PureData has been working for me, especially considering the very little experience I have had with it before yesterday.

I think I got everything I did together in this screenshot (click for fullscreen):