Will this gimp you later in the game? I know many people still just think of these 2 stats as resources for using your abilities and skills. The truth of the matter is that they both seem to significantly affect your damage. It seems like optimizing your damage output would mean focusing in either Magicka, or Stamina... but not both.
Using the tool at esohead
http://esohead.com/skills/35596-impale you can see the effects of these stats (in this case its magicka). The difference between spending no points in Magicka, vs maxing out the stat, is an additional 252 damage (at rank 4, level 50) Thats almost a 50% damage increase.
My character is only level 16 right now, so i have no idea how things pan out later in the game.