It's your Extraction Passive that sets how much you get back from refining, not your material level. In fact, it is a waste to invest in material levels unless you can use them (or need to farm for VR15/16 mats in Wrothgar/Cyrodiil).
So, if you have the extraction passives you will get tempers from any material. I have pulled Golds from Jute and Iron and Maple.