In my opinion the problem of queuing system maybe come from the priority for group vs group and solo player vs solo player. Then it make the script become complicated and uncontrollable.
For hardcore player. I suggest that ZoS should allow ppl who want play as group to make a team. Then They can queue as a team ( when all member of thier team are online ). Then we can give them a ranking system, group with low rank will not face with high rank group. It's very similar to another Moba game ( LoL, Dota ...)
Forcansual player. Now you will not need to worry about the problem of group player will detroy solo player any more. The only thing you need to focus is make sure that every single match must have enough 12 player. I think the principle is similar to what you do with DF
Basically make 3 queuing type
-Team ranked queue ( must have team and make a group of 4 to queue)
-Solo ranked queue ( you only can queue solo, group will not be allowed )
-Casual queue ( let them do whatever they want, queue as group or solo, even group player vs a buch of solo player is not problem, because it's casual, no rule no priority , just make sure we have enough 12 player each match ).
And about the reward for BG.
Don't spam player mail by their unwanted gear.
Give them skin, costume, furniture, mount ( just for appearance ). Not specific one but a chest and key. Then the reward will be random ( Similar to Dota and Lol )