That is a good time to send it some feedback through the game itself. At times ESO can and will drop in performance depending what's going on. It doesn't necessarily mean that your hardware is to blame. This kind of issue happened to me in a certain part of the game where it severely slowed down my performance. I sent in the feedback and eventually one of the updates fixed the issue. I would suggest doing the same.