Not enough to worry about it outside of vet dlc dungeons or vet trials. I don't use potions in randoms, pledges, or farm runs and I still pull good dps (35-40k group buffed).
You are going to lose a few things if you run inner light on a bar and switch it out for entropy you will lose the crit buff.
You will be changing your rotation depending on how strict you are on it.
The amount of sustain you have will be lower the pots really help with sustain than you notice.
With that being said, unless I'm raiding a trial or dlc dungeons with people I know or my trials guilds, I dont bother using them. Ill use surge on my sorc, or entropy/sap essence with my magblade.