its because the separate zones of IC are individual pvp instances.
so every time you use a ladder you are actually doing the same thing as if you port into Cyro.
so there is a pvp instance travel delay, like when you log into Cyro and still have the loading into instance notification going, though normally in Cyro it does not amount to much, its far more noticeable in IC when those transitions are literal doors and ladders that you can spam interact with.
of note its just the sewers and the district that are separate instances, inter district travel does not have this issue, just sewer to district.