I had this problem with only one wireless controller. I was fairly confident that it was just an issue with that controller so I just returned it and have not had the issue since. I have gone through about a half dozen controllers playing ESO as I always end up breaking the RB and/or LB after a while.