It is more than a Xbox live issue. Other games work fine. Live via Twitter has only reported party chat issues. It is once again in Zenimax hands. This happens nearly every time.
Delete and Reinstall the game, it has happened to me before and that its my fix on the xbox one. I Also think it was an xbox issue this weekend because its not happening again.