If I remember correctly, you use a ladder to get off, so then there should be a small boat on the beach that will take you back. You might also be able to catch a ride back in Skywatch.
Yes, that quest.So the quest you want to have is "The Great Tree."
If you dropped the quest so you don't have it anymore, you may need to receive it again from Queen Ayrenn. She might be in Firsthold still where you finished the last quest, or I believe she moves to Skywatch palace at some point.
If you have the quest, you'll meet Jimila beside the stage near the Mercantile. Then she'll take you to Vulkhel Guard, from whence you'll head out to Grahtwood.
Does Jimila in Vulkhel Guard or Ayrenn in Skywatch offer you any dialogue for it?
If not, you may have been shunted into the alternate, mutually exclusive version of that quest: https://en.uesp.net/wiki/Online:Trouble_at_the_Tree
In order to get that quest, you need to do "Unsafe Haven", which you would otherwise get from Captain Jimila. Instead, you'll need to travel to the wayshrine outside Haven and get the quest from an NPC near the sailors on the beach: https://en.uesp.net/wiki/Online:Unsafe_Haven
Both "The Great Tree" and "Trouble at the Tree" are transitional quests designed to send your character headed off towards Elden Root, so it won't matter which one you do.