For best results, you would also need to get some weapon/spell damage from set bonuses and jewelry enchants, since otherwise it will be extremely low if you don't equip any weapon.Assuming you're not trying to clear too much group based or end game content.
You could take something like a Templar and never equip a weapon. Focus on skills which would buff you damage or look monk like, then just weave in punches.