One workaround is not to exit the group menu on the party role selection or any other line that has an info panel pop up and/or that changes selection via the left axis stick. Sometimes in gamepad mode the game takes down the UI for a menu but it is still active behind the scenes, so moving your left axis stick can cause an info panel to pop up or a rotation through certain selections even while the menu looks closed. You can get the whole group population list to cover your screen this way, and I believe it’s the same bug that causes housing permissions to be changed when the menu is supposedly closed, as those perms cycle through based on the left axis stick.

You can bring up the group menu, navigate to a line without a pop-up panel, and then close the menu.