Although I don't know the answer to your question, perhaps players who have joined the group at your invitation are able to invite them. If that is the case, then I don't know whether that ability is a feature or a bug.

Regardless, I suggest that you read the following OP and add more information to your inquiry:

https://forums.elderscrollsonline.com/en/discussion/294964/guidelines-for-bug-report-forum#latest

By the way, there is at least one add-on available (from ESOUI) which, when the user activates it, enables anyone who wants to join the user's group to enter a "password" in a chat channel (usually the guild chat channel for its members). The add-on detects the entry and adds them to the group (the user is the first member, and designated by "wearing the crown"). It is a popular add-on for people who organize pre-scheduled groups, usually through their guild Message of the Day.

Anyone who observes other players entering the password in the chat channel can do the same thing to join the group without an explicit invitation from the organizer (you, in this case).