That's a "standard problem" in ESO, the party system is pretty broken in many ways, including sometimes only remembering who was in the party and hardly ever putting the party characters into the same server node upon login. So much for "the game keeps track of your friends", etc., I mean.. seriously: Having to "travel to player" and run from the nearest wayshrine to your friends to actually see their characters and being able to team up with them clearly is a major hassle and makes everyone waste a lot of their time. I wonder why this crap never got fixed since beta - it's not that this is new by any means !
The leader not properly being defined is another key issue which happens often for no reason, the only way to get out of this being to disband the party by letting everyone leave it, then relogging. But even this along with creating a new party oftentimes doesn't fix the issue. Again, this is as old as the beta and ZOS never cared to fix it despite their mouthy advertising of ESO being a most social game.
Yeah sure... but party system not working properly clearly contradicts this advertisement. Get your sh!t together, ZOS ?