It's worse than that.

Even if you join an xbox party, the game will still pick up your voice chat. It can be pretty hillarious in city areas when this bug happens to people. I don't think it happens all the time to everyone, but it happens often enough that you can hear some pretty odd things.