Yes, eventually they will leave if you push their Rapport low enough.
They've got their Rapport meter that spans from red (hate your guts) to green (BFFs) -- and it moves pretty slowly. People are trying to get maxed out on the green side (for achievements, related bonuses, etc), and because progress is so slow no one really loves the idea of having minor stuff like harvesting bugs or crafting mead negate their efforts.