As @KhajitFurTrader said. Also, I have the same setup and it's about average though I monitor my memory via Activity Monitor (select eso process in Memory and double click). When Real Memory gets close to 2.7GB or Virtual Memory to 3.7GB I reload the game to clear memory.

Also, set Texture Quality to Medium will make a big difference though I find I prefer the above mostly) Setting Shadow Quality down from Ultra to Medium will give the best boost to FPS and not much impact to shadows; though generally Ultra-High works fine most of the time it's worth tweaking it.

In PTS I never crashed due to memory mainly because the number of other players was lower than this launch period. So I expect it will also improve over time in questing etc, together with changes as they come.

The Mac client did have some changes to help memory use since the last beta which reduces the frequency of crashes. In PvP it i's more frequent because of the much higher demands of big battles. The development team is continuing to monitor and work on the issue.

Ultimately a 64-bit client will alleviate the problem when it comes.