There is a bug whereas a cancelled fast travel or recall can come back and still complete when you really don't want it to. And since this happened to me a couple of times recently I decided to look closer into it.
Simply put, if you cancel a Fast Travel or Recall action, then at any time walk through a door, the fast travel action will kick back in and finish the job. Resulting in not only traveling when you decided you didn't want to, but also deducting the gold coin cost for the travel. I clipped a video demonstrating it below..
https://youtu.be/CEgELiaJGGA