I recently noticed that if your bounty decays during a fast travel or loading screen it does not factor in the swiftly forgotten skill. I have it at max and my bounty decays at a rate of 115 every 3 minute but if the 3 minute tick happens during a loading screen my bounty will only go down by the default amount of about 20 or so. This is consistently repeatable. You just need swiftly forgotten maxed out, get a bounty, sync up the 3 minute tick timer and fast travel right before it goes off. So that it does during the load screen.
I don't think it matters but just in case this is on Xbox series S NA server.