This, you can probably cut it a bit not putting points into skill you never use.You need a minimum of 120 Skill Points to have an effective combat character that uses one weapon, does no PvP, no Crafting, no Vampire or Werewolf, and minimal guild skills (Persuasion and Intimidation). This is the minimum.
It really does no good to be at L50 or CP160 and only have 64 skill points. Even if you can wear end game gear, your running a gimped character.
Thing is, the activities that you are going to do to get the skill points will gain you the levels. If you just gain the levels like with Dolmen Zerging, then you still have to do the activities to get the skill points. So you have wasted your time really if you don't have the skill points to do something with.
Recommend doing an combination as in stuff who give lots of xp and a skill point.
the main quest, delves , collect overland skyshards and mage guild books in area.
then you are able to do decent dps and survive: public dungeons don't do them if you struggle a lot.
At this time you can also do group dungeons, Random dungeon + dungeon quest give a lot of xp and an skill point.