Having 5/5 Wise mage set equipped I began to notice, even attacking one mob, the visuals for the debuff would appear and disappear. According to set description this should only happen if I attack a different target. I could not find a similar post to what I assume is a problem/bug. So I made sure to turn off all addons and capture a video of it in action.
Basic weapon attacks appear to toggle the debuff on/off.
https://www.youtube.com/watch?v=FFdjWBh_7Y4