The bonus from having the stacks is only active when you have 5pc siroria active. You can keep the stacks, you just won’t keep the spell damage on backbar.
Also, siroria makes more sense to keep active over other 5pc-active only sets like mothers sorrow or Julianos because its 5pc bonus is considerably stronger than those of these other sets.
if you only front bar siroria, you will lose the entire stack because it is a buff