Yeah it does that if you did a travel to location recently like a home then try to do it again a second time. The fix is to zoom in or out to break the dragging.
It could be due to drift in your controller joystick. Does your character also move in the same direction or the camera move in the same direction all by itself? I didn’t have issues like this on PS4 but my PS5 controller does.