Try a different door. Once its damaged you've already "lost" and have to start that door over but there's a respawn before the door repairs itself and the daedra appear again. Ultimately you will have to defend all four doors to complete the quest even though it starts you off with defending the south. If you go to a different door it will update to "defend the east door" etc. for the quest.

FYI doing this without a group is kinda tough unless you have great crowd control because the door really can't take many hits and starting in the second wave they just rush the door and ignore you.