In my experience, keep accepting and it will stick sooner or later, but almost always sooner.
Invite comes. I accept. Then "someone declined" message displayed, i get in the top of the queue. Second later the invite comes again and it repeats infinitely.