Sorry if it's a dumb question, but do you only get the benefit from any passives you have "bought" when you have a skill from that skill line slotted? As in, do I only get the benefit of refreshing shadows when I have a shadow ability slotted?
Not necessarily. The passive description tells you whether or not you need to have an ability from the skill line, or the appropriate weapon slotted. Many are effective for your character generally, whatever weapon/skills you are using at the time.