EU Server Performance - Update from Matt Firor

ZOS_JessicaFolsom

Hi everyone,

Today, members of the ESO team are participating in a Reddit AMA over on the ESO subreddit. We have seen your questions and concerns about EU server performance, server capacity, and issues with the Activity Finder, and wanted to make sure we addressed them in the AMA and here. Below is a response from Game Director Matt Firor.

Hello! I suspect we'll get questions about performance more than a few times today, so I prepared a description of what we have going on to address performance concerns - on a number of fronts.

Even though there is very little new information here, we definitely could be doing a better job of keeping everyone appraised of how things are going while we are in process. We will be better messengers in the future keeping everyone up to date on longer-term fixes and updates.

First, server capacity. As we all know, we had server capacity issues a few months back, especially on the PC EU servers. We were forced to temporarily cap the number of concurrent users on PC EU and put up a queuing system to enforce it. We got the new capacity online for PC EU in a week, and added that same capacity for PC NA (which was very close to experiencing the same problems). We took the queuing system down, and we can now support the concurrency demand we are seeing.

Second, the looking for group system. As we posted last week, we are working on a 100% revamp of this system, which is mostly used for Dungeon and Battleground queuing. We have a version done that we are testing in-house, although it will be a while before it goes live. We want to make sure that the solution we have scales correctly and will support the now massive numbers of players that are using the system. We are planning for revamped LFG system to be in Update 24, depending on how testing is going. Until then, we will continue with the only solution we have (which we posted about last week in the LFG update) - cap the number of players in the queue. We know it is extremely frustrating to not be able to queue for dungeons and BGs and get "cannot join queue" when you try, but at least now once you get into a queue successfully, you should be able to join the dungeon or BG.

Third: overall performance. Rich talked about this in his end of year message six months ago: we are currently implementing sweeping optimization and memory management client architecture changes that should improve client performance and especially client stability. These optimizations will be most noticeable on lower-spec PCs, base Xbox, and base PS4, but overall everyone should notice better stability and performance. This is scheduled for Update 24. Additionally, we have another set of optimizations scheduled for Q1 next year that will reduce the client "footprint" requirements and should speed up loading times.

Fourth: Longer term, we are in the beginning stages of combat system performance improvements that should improve overall game performance in Cyrodiil and Trials. Stay tuned for updates on this.

Obviously if anything else crops up, we'll investigate. I know that it can seem like it takes an inordinate amount of time to address performance problems, but we need to take our time and make sure that we don't introduce new issues while we do this. Thanks for your patience, and we will keep you updated.

Source: https://www.reddit.com/r/elderscrollsonline/comments/bwdb0r/welcome_to_the_elsweyr_update_22_aua/epwuxj0/

You'll find more details in our PC-EU Server Update thread from April here.

MartiniDaniels

ZOS_JessicaFolsom wrote: »

Hi everyone,

Today, members of the ESO team are participating in a Reddit AMA over on the ESO subreddit. We have seen your questions and concerns about EU server performance, server capacity, and issues with the Activity Finder, and wanted to make sure we addressed them in the AMA and here. Below is a response from Game Director Matt Firor.

Hello! I suspect we'll get questions about performance more than a few times today, so I prepared a description of what we have going on to address performance concerns - on a number of fronts.



Third: overall performance. Rich talked about this in his end of year message six months ago: we are currently implementing sweeping optimization and memory management client architecture changes that should improve client performance and especially client stability. These optimizations will be most noticeable on lower-spec PCs, base Xbox, and base PS4, but overall everyone should notice better stability and performance. This is scheduled for Update 24. Additionally, we have another set of optimizations scheduled for Q1 next year that will reduce the client "footprint" requirements and should speed up loading times.

Source: https://www.reddit.com/r/elderscrollsonline/comments/bwdb0r/welcome_to_the_elsweyr_update_22_aua/epwuxj0/

Thanks for information!

But p.3 made me worried:
"another set of optimizations in Q1 next year"

(shrugging) So if there won't be players churn and server load won't decrease we will be kicked to login screen on regular basis at least until Q1 2020?

Qwazzy

Staggering player base growth one mistake at a time.

FierceSam

Not entirely sure players are going to be happy to stick around until U24 (End 2019). Or maybe they'll just not bother doing dungeons. Maybe ZOS should think about not adding new loot or stuff there to encourage people to go there. Or are they hoping that after everyone stops levelling necros traffic will drop and they can get back to ignoring the issue again.

But then if ZOS's solution to increased numbers of players is to implement queues for everything, the issues will soon be resolved as people leave as quickly as they arrived.

Loving the fact that you don't even mention the Mac. Really reassuring.

Apparently Rich talked about performance 6 months ago. That was enough. It's all been downhill from then.

Haenk

I smell BS.
EU has *server side* capacity problems. This is easy to prove - list items in main time and in low population times in your favorite Guild store. Notice the difference? Just a few minutes ago, the answer to a search result took no less than 80 seconds. (Yes, that is *eighty* *seconds*.) During low population times, this takes about 100ms (100 *milli*seconds).
So - improving the client does *nothing*.

code65536

Next dungeon DLC should be Update 23, if the release pattern holds. It's unfortunate, then, that you won't be done testing the new LFG system until Update 24...

casparian

We will be better messengers in the future keeping everyone up to date on longer-term fixes and updates.

I hope so, but if I had a nickel for every time y'all have said this, I would be able to subscribe to Plus for the remaining lifetime of the game.

Chaos2088

Well we got communication about what is going on at least. Got to give zos the credit for that.

Do we want to sorted now, yeah course we do. But I would rather them take their time and get it right rather than the same convo/updates over and over again

Bloodraven187

The way I read it is that they should have improvements before then, as long as they don't encounter new or bigger problems by introducing the fix. It would suck to get to a point where they fix it so you can't get kicked out...but also find you can't get back in all, or even worse not be able to get in in the first place. If they are doing things like what they suggest here that is a lot of code to rewrite or replace. The bigger the change the more testing that needs to be done...and the more than can go wrong. I'm not a coder myself but I've talked to enough to know and worked the tech support side to know how one fix can lead to 10 new problems. Be patient, at least they are acknowledging it and working on it.

Bloodraven187

Of course improvements doesn't mean resolution. In reference to my comment above

daemonios

Haenk wrote: »

I smell BS.
EU has *server side* capacity problems. This is easy to prove - list items in main time and in low population times in your favorite Guild store. Notice the difference? Just a few minutes ago, the answer to a search result took no less than 80 seconds. (Yes, that is *eighty* *seconds*.) During low population times, this takes about 100ms (100 *milli*seconds).
So - improving the client does *nothing*.

This exactly. I don't think the server issues were or are limited to "number of concurrent users". The LFG tool issues look suspiciously like server-side capacity issues (inability to spawn the required number of instances). And as @Haenk stated, other problems like laggy systems during prime time also point to server-side issues. I would love to see these specific situations addressed.

Suddwrath

The new LFG system won’t be released until U24? Wouldn’t it make more sense to release it with the next dungeon dlc (U23)? Four months seems like a long time to wait for the LFG system to be fixed, especially since we have been hearing for years that it was being worked on.

Putting a cap on the number of players in the queue seems like a bandaid that many players will not tolerate for 4 months, especially during prime time.

Keewo

Third: overall performance. Rich talked about this in his end of year message six months ago: we are currently implementing sweeping optimization and memory management client architecture changes that should improve client performance and especially client stability.

Will these optimizations help with the FPS Drops over Time problem? Overall, espacially in Trials are they my biggest Problem.

Jolipinator

Jolipinator wrote: »

There's the communication we are looking for.

Mitrenga

Sorry but that schedule has many ifs and buts. The number of ifs and buts stating that we will see a timeplan exceeding mid 2020, considering the exact definition of which implementations will go live and how the testing will be done (probably by us PC EU lab rats?).

Personally, I would have expected a more expedited plan since PC EU really is in a pathetic state and it is a major issue but looks like ZoS has other priorities.

Jamdarius

ZOS_JessicaFolsom wrote: »

Hi everyone,
First, server capacity. As we all know, we had server capacity issues a few months back, especially on the PC EU servers.

So basically you have been aware of EU PC servers issues for few months and you are telling us that "you may" (or may not at all) solve those issues at the end of this year? (another few months).

Sorry but at this point do you guys take us for fools or something?! How long do you intend on making us wait till you fix the damn thing?! 1 year? 2 years? till the closure of EU server? We have been promised better servers like month ago, it didn't happen, we got same if not worse issues right now...

Instead of working on new dungeon DLC I bet whole player base of PC EU at the very least would rather have you work on fixing the issues we have, game is unplayable as it is now on peak hours/weekends and those are times when players want to relax after work/school/[insert whatever you fancy here] and not getting annoyed by some broken things in a game they pay for.

Make a community poll on what you should fix first on new dung DLC or fixing issues if you dare ...

Gythral

A very large amount of info on the wrong part of the issue.

The servers (location/code/capacity/support is in adequate) not the client & I see nothing about fixing these issues in the next year...

Please hire a whole team that know about MMO servers and what they need & listen to them!

Peekachu99

Thank you for the update, @ZOS_JessicaFolsom This was exactly the info that I was looking for in the AMA and I know that architecture changes like this aren’t easy to implement so I appreciate you keeping us in the loop on larger scale development plans. 6 months isn’t too long of a wait for better tools and performance.

Verbal_Earthworm

8.36pm on Monday 3rd of June 2019.

Whilst trying to play the laggiest pvp ever with skills not working and surreal things happening then the server just crashed and kicked everyone to log in screen.

Thanks, Matt.

Good to know you have things under control.

NirnChaos

[quote=".... I know that architecture changes like this aren’t easy to implement so I appreciate you keeping us in the loop on larger scale development plans. 6 months isn’t too long of a wait for better tools and performance. [/quote]

LOOOOOOOOOOOOL 6 months isn’t too long of a wait for better tools and performance. YES YES GOOD NO PROBLEM sir

everyone is entitled to his opinion, we must respect. but frankly I can not find this reasonable, it is an eternity, it is not an independent developer this is Bethesda!

it's literally a machine to print cash.

That should not even happen, they do not want to invest that's all.

Jamdarius

Peekachu99 wrote: »

Thank you for the update, @ZOS_JessicaFolsom This was exactly the info that I was looking for in the AMA and I know that architecture changes like this aren’t easy to implement so I appreciate you keeping us in the loop on larger scale development plans. 6 months isn’t too long of a wait for better tools and performance.

You have totally forgotten PC EU was already suffering from those issues for 6 months at least (around before Murkmire release if you ask me but whatever lets say it was just 6 months and not 9 QQ).

meaning they tell us that we should wait another 6 months to a total of at 12 months IF THEY WILL ACTUALLY FIX THE DAMN THING. So far I see nothing but empty promises...

Hyzock

Fourth: Longer term, we are in the beginning stages of combat system performance improvements that should improve overall game performance in Cyrodiil and Trials. Stay tuned for updates on this.

You're in the "beginning stages" of fixing an issue of something that has been a major issue for about 5 years? Matt is telling us they've been doing literally nothing about cyro performance for the past 5 years? CP pvp has been unplayable during prime time for the past 5 years and only now they're in the "beginning stages"? I don't even know what to say anymore.

Royaji

casparian wrote: »

We will be better messengers in the future keeping everyone up to date on longer-term fixes and updates.

I hope so, but if I had a nickel for every time y'all have said this, I would be able to subscribe to Plus for the remaining lifetime of the game.

Well that's objectively not true. They only pop in to reiterate how they are trying to be better at communication once in a month at best.

Peekachu99

Jamdarius wrote: »

Peekachu99 wrote: »

Thank you for the update, @ZOS_JessicaFolsom This was exactly the info that I was looking for in the AMA and I know that architecture changes like this aren’t easy to implement so I appreciate you keeping us in the loop on larger scale development plans. 6 months isn’t too long of a wait for better tools and performance.

You have totally forgotten PC EU was already suffering from those issues for 6 months at least (around before Murkmire release if you ask me but whatever lets say it was just 6 months and not 9 QQ).

meaning they tell us that we should wait another 6 months to a total of at 12 months IF THEY WILL ACTUALLY FIX THE DAMN THING. So far I see nothing but empty promises...

You realize that a vast majority of players experience next to no issues at all, right? I’ve crashed maybe once in the last six months in 4-6h of playing across weekends and peak times. Anecdotally, performance has gotten better (PS4 Pro). So your experience is not empirical. If you’re playing on a base console or crap PC that’s your first issue. Also data server location is another, but that will affect you in every major online game depending on where you live.

frostz417

So basically we have to wait 6 months for fixes that may or may not actually come? Nice

Gilvoth

what about that you are working on a new MMO?
isnt that going to mean ESO is now on the Back Burner?
kinda does feel that way.

Wolfpaw

Cool stuff.

I will continue not to sub, buy expansions/dlc's, & crown store purchases until the fix is live.

Thanks for the update.

Peekachu99

Gilvoth wrote: »

what about that you are working on a new MMO?
isnt that going to mean ESO is now on the Back Burner?
kinda does feel that way.

Don’t think it’s a new MMO, necessarily. Could be a GaaS.

MentalxHammer

Huzzah! 100 more years!

Jamdarius

Peekachu99 wrote: »

Jamdarius wrote: »

Peekachu99 wrote: »

Thank you for the update, @ZOS_JessicaFolsom This was exactly the info that I was looking for in the AMA and I know that architecture changes like this aren’t easy to implement so I appreciate you keeping us in the loop on larger scale development plans. 6 months isn’t too long of a wait for better tools and performance.

You have totally forgotten PC EU was already suffering from those issues for 6 months at least (around before Murkmire release if you ask me but whatever lets say it was just 6 months and not 9 QQ).

meaning they tell us that we should wait another 6 months to a total of at 12 months IF THEY WILL ACTUALLY FIX THE DAMN THING. So far I see nothing but empty promises...

You realize that a vast majority of players experience next to no issues at all, right? I’ve crashed maybe once in the last six months in 4-6h of playing across weekends and peak times. Anecdotally, performance has gotten better (PS4 Pro). So your experience is not empirical. If you’re playing on a base console or crap PC that’s your first issue. Also data server location is another, but that will affect you in every major online game depending on where you live.

Witcher 3 ultra settings is my PC...

Majority of players experience no issues? pure bs, say that to steam players, say that to PC EU players say that to my guildies, say that to my friends, say that to players that play only on weekends to relax or play on peak hours due to work/real life not allowing them to play in different times. Sure you may have no issues, sure your friends can have no issues, it doesn't change the fact that this issue exist for a long time.

PS: Just noticed you are console player, what are you even complaining about when I spoke of PC EU issue ? Do you even play on PC EU at peak hours ?

coletas

"and make sure that we don't introduce new issues while we do this."

like always lol

I will never put a dollar in this game again until I can play PVP EU. Every month is worse. No matter the queues or the computer, conection or ISP I have. Please, stop wasting time on things that no one is asking for and solve the historical bugs. There is a big list, if you dont have it, I can compile it for you.