When you buy a house unfurnished, you do get all the exterior stuff. But as stated, you can move it or remove it, up to a point. There will be some "landscape", such as grass and scrub plants that aren't furnishing items and will persist.
Serenity Falls has a bridge over the stream on the property that is not listed as a "furnishing", so is fixed. There's a simple Khajiit Archway, however, that you can take down and move.