To sum up.
I log in some major city, max settings keeping 60 fps no problem (vsync locked).
I go out, quest around for a while and return to the city. Now I have 45-55 fps and notable choppyness and horrible performance.
Restarting the game and logging back in seems to solve it completely.
I read that there has been a lot of memory leaks wit this game, is that what it is?
Specs: MSI gaming X gtx1070 - i7-4790k. SSD evo850 samsung