The difference between the "meta" or optimal races and off-races for a class spec isn't much, but in PvE endgame content and PvP it can and often will matter, but endgame is a small percent of the actual content.
I used to play a Stamina Templar with Altmer and I totally advise against it. I'm full Magicka now and barely need potions. If you want Stamina characters go Redguard, Orc or Imperial.