Played GOW for an hour last night and didn't notice, but apparently my controller has a very small dead zone that is causing the unresponsiveness in ESO. Only noticed now when I fired up Fallout 4. Couldn't even tell anything was wrong when I switched to Elite Dangerous. Played GOW again after and still couldn't notice any impact on controller responsiveness there. Same stopping and starting in 3rd person Fallout, though.
Plugged in the Xbox One controller and ESO is fixed.
Stuttering and fps drops remain, but it was my broken controller that was "game breaking". Apologies. Please close thread.