I have this problem except with the left analog stick. This has happened to two of my controllers so far on Xbox and started before TESOU so it is a controller problem, not a game problem. Xbox support on twitter wasn't too helpful (no surprise there), but they did give me a link for a survey. Yay for surveys...
What about when you go to the map, it spins my character around when i view the map so i have no idea which way i was facing.