This is just a theory of mine and isn't confirmed. For context, Druids of Galen is a retroactive term used for the proto-Bretons (the Nedes of High Rock), as well as later on for the Bretons that emerged and took what would be the practice of modern Druidism. Galen is the origination of the "True Way", the start of the druidic faith, which was not fully formulated until the Bretons came about. This is not a thread about the Systres island of Galen, which was named after the original Galen.
People will
probably point out that Wyress Ileana says the following to form a counter arguement:
"If you must know, the guardians are the Ehlnofey. They've been a part of the land since the dawn of Tamriel.
The guardians have been with us since we first found the Wyrd Tree. They protect the land and guide us in our duties."
The bit that says "The guardians have been with us since we first found the Wyrd Tree." does not necessarily contradict, as we now know that the Druids and the Wyrd used to be one of the same, in that they originated from the same people, the Nedes of High Rock thanks to
Wyrd and Druid. This discovery could have been back when they were one of the same. My focal point is the Wyrd Tree, which is what the present day Beldama Wyrd congregate around.
This is only a theory, but here it is:
In the
Loremaster's Archive - The Druid Circles of Galen, we learned that the term Galen has an unknown and possibly contradicting origin for the group. According to some ancient tablets, it is the name of an island in a lake, the name of a spirit, or the name of the first Druid Queen.
"Galen, it's said, was the name of a spirit that walked from the heart of the forest into the midst of an ancient proto-Breton settlement. They were the one to sing the songs of Y'ffre to our ancient forebearers. They were the one to teach us the ways of the Green.
Or, if you believe another tale, Galen was the name of the first Archdruid, the first Druid Queen. She was one of the very first Nedes to channel the songs of the Singer, and rose to lead the first circle in the deep woods of High Rock.
Or, in yet another telling, Galen was the name of an island in a lake. A place of sacred worship where Y'ffre's heart could be felt beating even here in the realm of mortals. It was there, on that island, where our forebearers heard the songs and learned to tend the groves."
Here is why I believe neither of these stories don't necessarily contradict, but rather complement each other. Just like the Galen is a spirit story, the Wyrd Tree is described as a spirit by a
wyress. While it can't walk, wandering into a town wouldn't really be an issue either despite being a tree, as it is
said that its roots stretch through ALL of Nirn. It doesn't contradict the island story either, as the Wyrd Tree is quite literally in an island on the lake. Image below

The Wyrd Tree being Galen will contradict the story of the Nedic Druid King origin though, unless the Wyrd Tree itself was worshipped as the first Druid Queen. Wouldn't be too hard to see a female spirit as its representation, considering the
Guardians of the Air, Water, and
Earth are female spirits. It is also possible that the first Druid Queen was just a female Nede who took on the name Galen to honor the Wyrd Tree, and fully representative it.
There's also the emphasis on the first druid circle rising from the deep woods of High Rock, which is what the Wyrd Tree is
emphasized as being located at.
Another connection is the term druid itself. According to wikipedia, druid irl means something like "oak-knower". The Wyrd Tree itself is
described as a "a colossal specimen of oak with a branch canopy that covered the firmament". So essentially, Druids of Galen are Oak-knowers because they follow the teachings of an oak tree spirit named Galen.
As for why modern druids would forget why the Wyrd Tree is Galen? Many of the physical records were destroyed by the Sinestral Elves during
their assault of 1E 660, and as Laurel
said, druids have almost been wiped out several times throughout history so that is less people to tell oral histories from. Combine this with the fact that druids had been
absent on High Rock for centuries (the Druids left High Rock in 1E 330, ESO takes place in 2E 582, so 3172 years have passed), people will probably forget a place that haven't been to in that amount of time. Doesn't help that they named a Systres isle Galen, as that would probably lead to misunderstandings or assumptions. I can imagine that overtime druids just assumed the term Galen was for the Systres isle.
Ok so if its the Wyrd Tree, why would they willingly leave it behind? I can only speculate, but Druid Ryvana notes that the Wyrd
stayed behind to safeguard the forests of High Rock, and the druids probably thought they could take care of it. Combined with the fact that the Direnni and the Alessians were at the Druid's doorstep, the Direnni in particular disliked them for attempting to
assert druidic governance of the province. As Emeric
notes, the Wyrd were pretty under the radar compared to the Druids who actively tried to meddle, so the Wyrd would likely be good caretakers for the Wyrd Tree when the Druids left.