If you don't want us to be able to climb certain places, then make a specific invisi-wall where you need to, but every time I try to jump up onto a rock that isn't even that high, it's like an invisible force pushes me back and over to the side, sometimes causing me to even fall or take damage, or just frustrates the hell out of me. I mean, I know it can be better. The only thing I liked about New World was literally the climbing. They had no-no areas and invisible walls as well, but you could still actually jump up and your character would grab hold of the rock / ledge and you would pull yourself up onto it, instead of being pushed back and to the side in extremely unfavorable and frustrating ways. This is just how it is, unless they manually put invisi-walls on top of literally every single rock in the game (low ones and high ones alike).
-rant for the day over-