Siege camp only adding to it would bork the progression and story. Why bother doing the other quests?
So now the Stirk Fellowship have to battle the Worm Cult and Daedra with no equipment and supplies?
A lot of players are gathering Vitrified Souls for that quest by defending the siege camps even after they completed the siege camp quests, so they are still contributing.