If you are at L40 in Blacksmithing, then you can make any material level in the game. So, your maxed out effectively.

You do want to spend points in Temper usage as that really cuts down the number of Tempers you need to improve something. Like 2 Greens, 3 Blues and 4 Purples at L3 in that Passive.

You also need to be researching Traits so you can make special Crafted Sets. You are required to know a certain number of traits (2 at a minimum for each piece you want to craft) in order to make the Crafted Sets.

So, spend points on the passives, and do your Trait Research. It only takes a week or two to make the 2 Trait sets, it can take a month to make the 5/6 Trait sets, and it can take many months to get to 8 or 9 traits. So, plan accordingly.