The biggest problem is it's a resource you're not using. When stamina runs out you can stand there looking stupid waiting for it to recover or switch to magic based skills until it does.

I guess I can always respec, that is true. I was hoping to just put points into stamina, and a few in health. I'm trying to build a dual wielding, medium armor wearing, high crit, melee type character. But with no magic. Not sure it's going to work out so great in the end though. But it's fun trying

I think the biggest problem you will encounter is a lack of an Ultimate ability. The only skill lines I know of that are Stamina based and still have an Ultimate are the Fighter Guild and Werewolf skill lines. There are no Ultimates for weapon skills. I don't think the single-resource should be too much of an issue. Plenty of mages go single-resource on Magika, especially if they use staves. I suggest you focus on Medium armor (some stamina-based passives are in there) and decided if you wanna go werewolf or take the fighter guild Ultimate. Beyond that, you have your weapon skills and fighter guild skills for your active abilities.

The biggest advantage of avoiding magic is that you are less predictable. If I see someone with the dragonknight armor buff up or a mage familiar I have a good idea what their abilities will be like. A guy in armor with just a weapon would be a lot harder to predict. Especially since you can switch weapons and that would totally switch what skills you are using.

Just think of magickia as energy, willpower, consitution, whatever you wanna call it. All games have it in some shape just because they call it energy or runes doesnt make it not magic.

Now going heavy stamina build, wearing medium armor, and using class skills to compliment your weapon skills is fine and can be great.

My main (level 25 Nord DK (twohanded/bow) ) is a basically pure stamina user. The only class skill I'm using is stonefist as it looks more or less like tossing a rock at someone to knock him over with a blow to the head.
Fights can be tough and I did have severe troubles with several bosses in a few questlines, but it generally works out so far. I may have to skill dragonblood for the heal though, as the character lacks staying power in a prolonged fight. Some of these fights I could handle well enough by overleveling slightly.