I agree. Or better yet, put it just inside the door, then you don't have to even load the outside zone first.This reminds me. I wanted to make a suggestion to ZoS:
Can we please have a (miniature) wayshrine outside the harborage where you can interact with as any regular wayshrine. (Specially in EP), you can easily travel to the harborage for free, but have to make a 2 minute horseride to get a wayshrine when you are done and want to get back to the map for your level.