I don't believe this is a fixable issue given the limiting factor is the controller. It looks to me several features were combined in order to facilitate the fewer user input options the controller provides.
I don't see a frame rate drop as much as I see the console struggling to move the assets as fast as we're trying to get. I know PC players can zoom the map while moving locations easily enough, but that's something that struggles on the console.
I now just do one "command" at a time, rather than trying to use the map as intended. Zoom first, then move, or vice versa. It seems to help.
Of course, a solution ZoS could provide is to put a map on the HUD instead, so I wouldn't need to use the map option so much.