It all depends on what your race is. If you're a race with zero magicka passives then I would say yeah it would be worth it. If you have some racial passives that help with magicka reduction/regeneration, then probably not. Just my opinion. As an example,
my Blood Paladin is always at stage 4 vampirism and I pretty much am "ok" with it. Great magicka because of my Altmer but I plan to remove the vampirism eventually when I get another alt (a Dunmer) rolling because in delves/trials with lots of fiery things it ends to get crispy.
its really up to you.