Because they are located here and there where inspiration struck, rather than at points of convenience, these sites are scattered across the land, and have come to be known as "wayshrines."
https://en.uesp.net/wiki/Online:Wayshrines_of_Tamriel
Yes the holy spots comes first and the structures are built around them for convience. This is the nature of men & mer, so why this there design against their nature?