Could be influenced by the skills you have on your bar, or a buff running out just at the time.
Are you a Sorcerer perchance?
@Aletheion
Could it be this?
KNOWN ISSUES
Gameplay
•Your stats for Spell Damage, Weapon Damage, Spell Resistance and Physical Resistance while in battle leveled areas (Cyrodiil, Imperial City, Wrothgar and all dungeons) are not currently being displayed accurately in your Character UI; it will appear lower than the stats actually are. This will be resolved in a future PTS patch.
Did you actually test the damage and not just the character sheet?