Probably all the artillery is set as some invisible untargetable enemy that has your number as soon as you enter the hold proper and that is what is pulling you out. That would be my guess.
You can evade a good amount of them by going around them but I don’t think you can sneak by enemies in any dungeon.