I've hit people that were visible when it was cast and then went hidden. I've also had people go hidden and it not hit. I assume there's a very small window in which the projectile will hit or not. Unless you're saying you were always hidden and just randomly got hit by crystal fragments.