You spend them in the crown crates selection. You can turn in poisons, xp scrolls, food and other consumables to crown gems, or if you get some furniture or a duplicate item from your crown crates. With the crown gems you may be able to afford one of the things you truly want from the crown crates. But you do need a lot of gems to be able to buy something other than consumables.