Okay, writing this here to get down some thoughts for later (after a shower), and also in case it might help someone else down the line.
A ladder is a bit more complex a scenario than I originally anticipated. It’s not enough to just say so and so object is ‘nearLadder’ and allow them to float around on the Y axis. There are a few different scenarios that happen when a player gets near a ladder, which all need their own behaviors.
1: Player walks next to bottom of ladder

In this most basic scenario, the player could have some kind of variable set that it is over a ladder and can climb. Being able to climb should allow the player to modify their Y location via input. The basic climbing stuff.
2. Player climbs ladder

For this scenario, some basic climbing animation could play. In my case, I am setting the gravity scale for the player to 0, but that has it’s own issues.
3. Player walks next to top of ladder

And here is the issue. If you set the gravity scale to 0 when near a ladder, coming from the top makes the character float! Instead, I think a variable isClimbing could get set if the player presses up or down while over a ladder, otherwise, the player falls as normal.
4. Player climbs partially up ladder then jumps
This should set isClimbing to false, and since I’m using grounded checks to allow climbing, ladders should ground the player the whole way.
So a state machine would be something like:

Well that’s a start anyway.