The only time this happened to me is when I made a mistake and dropped the quest, then retook it right after: somehow it messed up the timers.
If you do drop the quest (and get ported back to where the quest giver is) give that character 15 minutes to reset.