How exactly does it have nothing to do with privacy? That's a very valid concern. It violates my privacy if people can look at exactly what i'm wearing, exactly my build etc. So yes. It does apply. It's a violation of privacy that LEADS to elitism. It's all related.
Providing a way to "turn it off" is also not an appropriate solution. It will lead to exactly the same elitism. People will camp out in town recruiting for trials etc. and they will refuse to let you join unless you turn your inspect on so they can judge your gear. It will happen, it's guaranteed.