It's strange that no one has already asked the "Natch Potes"
Keep in mind that when sneaking into an adjacent zone through gaps in the map that the zone that is entered is not really an active game zone, regardless of whether it is in development or finished. At least, the ones I have been to or seen pictures of.
There are no players there. The zones aren't finished. Not everything is there. Nothing works. The terrain doesn't even always match. They are placeholders only. It looks like every zone is a separate map that really only contains the finished map for that zone plus some placeholder maps for adjacent zones. I figure it is some copy-paste artifact from how they generate new zones so that they get placement right and things appear to line up along the borders.
Going from Auridon to Summerset is probably going to show the Summerset placeholder from when they created Auridon. It is possible that this has not changed since 2012 or 2013, and might bear no resemblance to anything they are working on today. There is no reason for them to update the Auridon map to include the latest Summerset development. It could be that they originally planned to do a Summerset Isle years ago and stopped.
One of the things I commented on during Morrowind PTS: If you stand on the far southern end of Vivec City and look towards Stonefalls, you will not see it. You will see land, but you will not see the Stonefalls that you can go to. For whatever reason, when they made Vvardenfell they did not create a copy of Stonefalls so that it appeared to be there across the water. Likewise, the Vvardenfell that we can see from Davon's Watch is not the Vvardenfell that actually exists. This serves as a reminder that the adjacent zones are just placeholders. Facades. Not the real zone.
(edit: and this lack of surrounding zones in Vvardenfell might also be why Solsteim is not there)