So I spoke to one of the devs real quick on Twitch. They said the big culprit for us PC users is outdated add-ons. Also try deleting your shader cache file in Elder Scrolls Online > Live folder. I'll give the add-ons more time to update before really judging my game's current performance.
As for the issue still persisting after disabling all add-ons, I'm not sure. I didn't ask them about that since I hadn't tried that myself yet and didn't think of it.