Do you have Undaunted at 9 or 10? Or are you planning on chasing that to get the Passive?

Is this a character your leveling? If so, you need to wear a mix of armors and level them all to 50 anyway. You never know what changes will come in the future, and armor is the one thing you don't want to grind up end game.

To answer your question, maybe. But, it all depends on how you play, what other magic buffs you have, how often you die, how much you run around and or block during combat.

Personally, it's only 2% gain, not enough to worry about whether you have it or not. It's not going to matter all that much. Then again, Molag Kena drops in Medium too, so it makes wearing an off armor shoulder palatable to have the passive.