I think the question is: can you travel to Coldharbour freely? And the answer is no, not at the moment. As noted you will be taken there on quests that arise during your travels in addition to starting there at the beginning.
I've been wondering the same thing. I have a treasure map for this place and I don't see how I would actually go and dig the treasure up if I can't access it. I'm pretty sure the loot is low level anyways but if thats the case how do they expect an alt to be able to pull it out of the bank and dig it up?
When you first start a new character if you have never played before you will need to play through a small part of Coldharbour that gets you out of it. After that as you level up the Prophet will call you back to his hideout in Tamriel, which is near your first city after the starting areas are complete, and he will send you on missions to Coldharbour.