I end up taking photographs with my phone to remember where I spent in each category. I have every skyshard in the game but 7 in the imperial city. It takes me ages to redo skill points if I need to. Now thankfully I have enough that I shouldn't need to redo. But still. It's aggravating