Maintenance for the week of March 16:
• PC/Mac: No maintenance – March 16
• ESO Store and Account System for maintenance – March 18, 6:00AM EDT (10:00 UTC) - 4:00PM EDT (20:00 UTC)
This happens on PS4 sometimes as well. The only fix I have found is to delete that person from your friends list then add them again. I don't know if it works on X1, though.