Yeah I always have to restart game after switching alts (inventory management and crafting), game gets choppy if I don't. i72600k @4.6GHz gtx780sc acx and 8 gigs DDR3 1600 RAM.Pretty much this.
Its not dumping ram after you log off a character.
Usually things are held for a while, with a ttl timer on it(time to live). When ttl expires, memory is supposed to be freed and the data purged. Its holding it indefinitely.
If I hop around playing inventory management online a bit I have to reload the game before actually playing again.
I decided to check how much memory was being used about 2 hours ago (after being on ESO for an hour or more already) and it was around 2.6GB with just over 1,000,000 Private(KB) in resource manager (sorry can't remember the exact number). However, at the time of writing this now, with ESO in the background, I'm currently using 4.71GB of memory with ESO using 2,094,688 Private(KB) in the resource manager.