If you get the prompt to accept your dungeon queue whilst you're switching maps, it refuses to work. You can't accept or decline it, you just get in 'queue limbo' and have to wait for the timer to run out. Then you get the timer restriction for having not accepted or having queued too recently.