I agree...I think sometimes lag is server side or even on your ISP's network, but make sure you have a decent video card as well, or run dual cards like I do. The other system specs you mentioned sound fine.
Also agreeing with above comments, upgrade all drives(always). Go right to nVidia if you have a card that uses their drivers. They just released drivers not long ago that greatly improve performance for many games.
Another thing has has helped me with many games is a solid state drive.
Also, some people never seem to like to turn off their computers. A reboot in those cases fix MANY issues.