Nope, hop on in! Sometimes I like to try to solo WBs, but if someone joins me, I'm more than happy to wait for a respawn and try again. These are meant to be group events, so I'm not going to get mad if someone is doing just that.
I enjoy soloing world bosses but if someone jumps in to help I don't get angry at them. ESO is a multiplayer game and sometimes I actually feel a bit rude soloing a world boss in a zone with daily quests since it causes people to wait for the boss to respawn.