The damage loss would be marginal but for min/max it adds up.
Since you are in PvP the potion passive is worth it since every time you use a potion you gain a bit of all three stats. Since you are a magplar the increased healing done and received also help. The fast escape across a slaughter fish infested river is just a small plus in PvP.
You know what would be sweet. If zos gave us the opportunity to select ONE racial passive in addition to the ones we have. For example. I'm a nord and have all more passive. But they let me choose the max magic passive from altmer. Or they let altmer choose the % damage redux from nord passives. ....just a thought XD