I didn't answer this at first as I don't have an Xbox so I can't help with specifics. You likely have already tried all of these things but.. if I list something you haven't tried - for sure give it a go.

Power Cycle your console (don't just close the game and reopen, totally power off the console)
Try a different controller (obviously if it's working for other games.. controller isn't broken, but hey, I've seen weirder thigns happen)
Not sure if this applies to Xbox - but it has been mentioned if people encounter errors on the PS4 - that they should delete the saved file on the PS4 (not the whole game, just a little saved file).

Sorry I can't really help you, but hopefully somebody stops by who can.