First, you get a base score. For trials, each enemy you kill grants you points, and each boss grants you lots of points. Hard mode in a trial grants additional bonus points. For the arenas, each completed stage grants points, and in vMA, each unused sigil grants points.
Second, you get a vitality bonus based on how many you have left at the end.
Base score plus vitality is your unadjusted score.
The score is then adjusted based on your time to get the final score. Are you sure you are looking at the time-adjusted final score and not the "final" unadjusted score?
If your time is shorter than the target time, this multiplier will increase your score. If your time is longer than the target time, this multiplier will decrease your score. The target time for vAA/vHRC is 15 minutes (ZOS didn't change this after the vet rescaling). For vSO, it's 25 minutes. For vMoL, it's 45 minutes. For vDSA, it's 60 minutes, and for vMA, it's 90 minutes. If you complete it at exactly the target time, then your unadjusted score will be your final score.
So, for example, if you complete vMA in 60 minutes, then your base score will be multiplied by a bonus factor of 1.18. If you completed it in 45 minutes, the bonus factor would be 1.27. On the other hand, if you complete vMA in 2 hours, your base score will be multiplied by 0.82 (18% penalty). For a 4-hour run, your base score will be multiplied by 0.1 (90% penalty). Since it's multiplicative, it means things like the final cost of a sigil depends on what time you get: In a 90-minute run, a sigil will cost 750 points in the end, but in a 45-minute run, that sigil's worth 952 points, and in a 4-hour run, that sigil will cost you only 75 points.
In any case, a 45-minute vHRC run will be adjusted down by 18% over the final unadjusted score. A 90-minute run will be adjusted down by 55%. You should see a pretty substantial score difference between 45 and 90 minutes, which leads me to suspect that you were probably look at the "final" unadjusted score rather than the real final time-adjusted score.
Take Hel Rar for example. A guild I'm in, with our strongest team we run through it and we get an alright score. Today the guild took a few low levels in so they could get the title and a bit of experience, they almost wiped on all bosses but got pretty much the same score we normally do and it took them about 90 minutes.
How? I'm so confused lol