It pretty much goes without saying that there are some imbalances within the game and they are being worked on.

One thing that ESO is suppose to be about is build diversity but there is an armor passive that goes against this grain.

Prodigy for light armor.
Agility for medium armor.
Bracing for heavy armor.

Those passives are enough to severely hamper mixing and matching armor. I believe that the set limit is too high. Rather than 5 it should be 3.

With 7 armor slots, you can have two light, two medium, and two heavy with one slot left over. Which one you choose would provide the third piece.

But this introduces something...

With 7 slots I could equip 3 lights and 3 heavy and gain access to both passives. Well, if this is an issue and we are only suppose to receive the benefit of just one, then there is a simple solution. Spending that first point to unlock one, will permalock the other two so that you don't have the option to put a point into them.

Alternatively, these passives allow the allocation of 2 points. While allocation of the points is gated behind armor level, these particular passives could be gated behind quantity equipped. So with the benefit given to 3 pieces, if you go with 3 light and 3 heavy, you can only invest 1 point in each. To invest the second point, you will have to equip 6 of a single armor type.