Sadly, since houses are instanced this is what we can expect to have to live with, and since these are often connected to now very popular areas the load times are correspondingly long.
No other game I play regularly experiences 30+ second zone loads, yet this is par for the course in ESO.