Since I tend to take pretty much all passives I am not sure I would ever respec just those.
In fact, I would like a Respec option that just lets us respec our Active Skills. But, that being said, if your going from say 1H/S to 2H or DW, then you would want to respec the passives you spent in 1H/S
So, what we have is pretty good really.
Or, we could just respec one skill line, both passives and actives and leave the rest alone. That might be worthwhile.