Yes, ESO missed the boat.
As I was basically suggesting, (and seems like some have agreed), there should be marginal advantages, and *significant* disadvantages. That's how they've worked in previous ES games. Currently (at least with Vampirism) there are significant advantages. And neither of them really have significant disadvantages.