Penetration is the direct opposite of resistance. If you hit an enemy with 18k resist while you have 13k pen, it would be the same as hitting an enemy with 5k resist with 0 pen. If you have more penetration than the enemy has resistances, the extra penetration doesn't give you any damage increase, that's why there is a penetration cap in PvE (18200).
Also, how much you increase your damage depends on whether you hit a player or NPC. For NPCs it's 500 pen/1%, for players 660/1%.
So with the values you've given...
PvE: 18k = 36% mitigation. Meaning the NPC takes 64% of potential damage. With 13k resist you bring it down to 5k = 10%, meaning you do 90% of the damage at cap. You gained 26% damage as a proportion of an unmitigated hit, but because it's an additive value, it would be a 100/64*90-100 = 40.625% increase compared to the value you got with 0 penetration.
Same principle for PvP, except 18k is now only 27% mitigation instead of 36%.