I was running back and forth across the fire for a while with the buff and I noticed it wasn't clocking up. So I started checking before each time I'd strike an orb... and it only clocked up every few trips and it seemed like the number was highly variable despite the route I was taking being the same.
There is definitely an issue with how it's clocking the travel distance. It doesn't seem to be recording a lot of the travel time. Is there a cooldown time? There wasn't when I was doing the hookshot achievement.