I assume you are talking about avoiding the Main Quest? It'll mostly be okay as long as you don't mind being soulless for no reason, don't want prior relationships with MQ NPCs when you do their DLC content, and aren't bothered by the lack of explanation for why you can waltz around all three alliances. You also won't level up the Soul Magic skiil line.
If those details won't bother you, go for it. I have a number of characters who haven't done the MQ, but since I know what happens from the characters who have, it's not been a problem.