You have to go out the door (which is patrolled by guards) to get out of it again.
It's inconvenient, but it's part of the idea.
You go into the refuge to hide from the guards. You have to go out the door (which is patrolled by guards) to get out of it again.
No easy way out if you have a bounty. It makes sense that way.