If you want a prime example of that: I used to have a GTX 980 Ti, but had to sell it due to some unexpected bills. I went back to a GTX 760 I had laying around. There was no different in game performance at all. There wasn't a single noticeable difference in performance -- at least not in ESO. Every other game I had suffered, but not ESO.I know how your feeling OP and its so frustrating. its like the game hates high end systems. My specs below.
Moving to SSDs can improve performance, but only under certain circumstances (e.g. when you enter a new area and the game has to dynamically load textures/designs). It's not something that will be a noticeable improvement for most unless they have a really slow drive.HDD or SSD ?
After switching to SSD, game runs now with 90-100 fps, before with crappy 30-40.