There are four quests at Silent Mire. The Ravaged Village, The Saving of Silent Mire, and Cold-Blooded Vengeance need to be completed to turn the icon white (Strange Guard Beasts is a side quest that turns up after).

Cold-Blooded Vengeance might be the one you're missing; the quest-giver is Azeenus, who can be found to the northeast of the village, near the ship called The Crafty Monkey (see map)

Silent Mire isn't required for Cadwell's Almanac though. This screenie's in French, but it shows that for Cadwell's Almanac in Deshaan, you need Obsidian Gorge, Mournhold, Tribunal Temple, Shrine of Saint Veloth, and Eidolon's Hollow.