You must tell us which system that you are playing the game with: PC/MAC, PlayStation4, or XBoxOne.
I doubt that you're playing with a PC/Mac unless you are using an add-on that allows the player to change the default location of the Map. In that case, there could be a flaw in the add-on, or perhaps a configuration option that you need to disable or enable.
Regardless, if you press the same "button" that you pressed to display the Map, then pressing it again should either close the Map and show the normal display screen, or return the Map to whatever was displayed before. At least it works that way on a PC/Mac.
--- Shadowshire .......... ESO Plus on PC NA with Windows 7 Pro SP1