Assuming you've logged in your character which had the horse then it should be in your Collections: press U and select Mounts and you should be able to double-click it to make it active.
I have to admit I don't know how it's delivered on a brand new account or one which has had the IE added to it after 2.0 landed.
No, not at all. In the game as it is now, you buy it on one character and it goes into your Collections where it is available to all characters.
The thing is, if you're talking about a character that existed before 2.0 then the mount would be in that character's inventory and won't be in your Collections for other characters until you log in the character holding the mount after 2.0; the mount is transferred from their inventory to Collections the first login after 2.0.
So, if you had a pre-2.0 character with the mount the game 'knows' you have one so won't offer it to you again at the Stables, BUT, until you log the character holding the mount the mount won't be in Collections for other characters to use.