I dont think its ever been explained in game. But my best guess is that youre right that they are from the Aldmer. Likely these locations fell out of use at some point and werent maintained or restored for numerous reasons. With groups like the Chimer, Orsimer, Aylieds and Direnni leaving the Summerset Isles it would be inevitable that areas on the islands would become abandoned and desolate. And being that the elves are well known for being long lived but not able to reproduce at high rates like humans. The major losses of population that occurred because of the schism and the immigrations to mainland Tamriel left areas uninhabited far longer than they normally would have been if it had been humans. So by the time the population of the elves bounced back up to normal levels on the islands the areas that are now ruins were too degraded to waste resources fixing or rebuilding.

Don't forget that the aldmer -> altmers have had enless wars against both the sea elfs and sloads. So cities that have been under heavy attack would need to be rebuild. Or just left as is depending how the situation evolved.

While we were all chasing around Summerset with him looking for eye balls, the previous lore master did mention that these architectural features were in the style of "elegant decay".

So, in my opinion, it is intended that most of these "ruins" are a fashion statement on the part of the Altmer. Many buildings, regardless of whether they are new, or old, or even long abandoned, have been deliberately worked over to make them seem elegantly ruinous.

In a similar way in IRL, in Europe, it was fashionable at one time for the royals and nobles to build follies (decorative garden features) that looked like ancient ruins on their estates.

For the most part the ruins we see in Summerset are not the remains of earlier Aldmer settlements. They are building that are deliberately built as ruins. This is especially so regarding the ruins inside the cities. The "ancient" pillars on top of the not-so-ancient crafting hub are a bit of give away.

The paving on the bridge leading to the Royal Palace is a wreck because it has been deliberately made to look like that. Those guys hammering away at the walls in Alinor are not fixing them, they are distressing the architecture to make it look elegantly ancient.

It's all a matter of style - Altmeri Grandeur.