EU Server Performance - Update from Matt Firor

wavingblue

And a free play weekend! So with load issues heavily present lets kick off an influx of new players, because reasons. FYI I get 33ms pings to the American server on iRacing... about 110 to Europe. For ESO its normally 80ish roaming and 110+ when doing things.

Jagdkommando

@ZOS_MattFiror hey Matt, take a cup of coffee, sit in front of computer and just read last 10 pages of this thread. Thank you!

burglar

lordrichter wrote: »

MartiniDaniels wrote: »

Why not? They can split them and allow one-time account transfer. People who are satisfied with current performance or can't transfer as entire guild can remain on current server. People and guilds for whom "game is unplayable" move to new server. Everyone is happy, problem solved. There will be some mess and drama with separation of guilds, but better start a new with good performance then struggle with lag and crashes.

There are a myriad of different ways to do MMO servers, and honestly, which one is used is more personal preference regarding features. A major advantage of a megaserver over multiple discrete servers in a geography is that I don't have to care what server I am on. I find that annoying in other MMO games. I should not have to care what server I am on relative to my friends, and I should never have to realize I picked wrong. ESO would be better if they had one megaserver for both EU and NA, but that is a technology challenge well above what we have today.

The problem we have is not the megaserver idea, or the fact that there are only two per platform. (Unless you in Asia or down under... but that is a different story) The problem is that they really only made their server capable of handling a certain number of players before it could no longer scale, and we are above that number of players. Short sighted? Too focused on what is currently needed vs what will be needed? Dunno. Anyway, this is a combination of the success of the game, and probably also due to gradual decline in the number of players that the game can actually support.

When software performance is the limitation, throwing hardware at the problem only makes for a lot of hardware sitting around. If the software can only handle X player transactions per second, it does not matter that they are running on hardware that can handle twice that much. In this case, doubling the hardware performance does not double software performance. The diminishing returns means that, eventually, a hardware performance cap is hit It is no longer feasible to double the hardware performance in order to squeeze out a fractional improvement in the software performance.

Properly implemented, the megaserver should perform very well for the type of game that ZOS is running.

This is spot on, I think. I tried to get an idea of what megaservers are and came onto clustered computing, which was described as having large overhead for lots of equipment, challenges in maintaining it and as you mentioned, difficulty in scaling it due to the previous two. If it's anything like parallel programming, then it's almost impossible to predict it's behavior... kinda of like trying to predict how an exponential function will behave, given an input.

Even though I probably don't know what I'm looking at (lambda calculus?), I checked out some papers and found something that suggests that 'high-dimensional' data inputs - or those with many potential outcomes - would lead to unpredictable, or nonreproducible results. So, they padded the input with some predictions of what they expected some of the inputs to be, and got more reliable outputs. Which, I guess was an answer to something a lot of clustering algorithms struggle with.

I thought it was interesting to at least see the kind of work they need to do to run this game, and it's nothing to sneeze at. This article walks through choosing settings for a clustering algorithm, which I assume ZoS has had to do, or has been trying to optimize when setting up or attempting to optimize these servers. As a side note, I imagine it's hard to find people that are willing to do this kind of stuff.

https://advances.sciencemag.org/content/1/7/e1500163.full

Also, I don't know if ZoS equipment would comparable to this, or possible even better, but here are some of the hardware specs for the clusters or 'megaservers' used by princeton for research.

https://researchcomputing.princeton.edu/systems-and-services/available-systems

red_emu

The complete lack of communication or effort to figure out some improvements is now causing PvP to die off.

Remember when a year ago you tried to log into Cyro at prime time? (Queue position 90-250 for most of the evening)... Compare that to now. Hardly ever any faction is on pop lock on the most popular campaigns and that's just PC. I hear on XBox EU there are factions with 5-15 players max at prime time in 30day campaigns while the most populated alliance is on 2bars.

Congratulations ZOS. You killed PvP. Now the snowball is in effect and good luck getting any money off your players to actually invest in anything.

And your Free-To-Play event? Have you gone mad? Why would you entice new players to try out your product in such horrible state?

Amazing game, totally addicted but it has now become an absolute joke.

(Yes, yes, I know. PvE crowd is still gonna call PvPers a minority but for some of us PvP is all that we play the game for)

Jagdkommando

red_emu wrote: »

The complete lack of communication or effort to figure out some improvements is now causing PvP to die off.

Remember when a year ago you tried to log into Cyro at prime time? (Queue position 90-250 for most of the evening)... Compare that to now. Hardly ever any faction is on pop lock on the most popular campaigns and that's just PC. I hear on XBox EU there are factions with 5-15 players max at prime time in 30day campaigns while the most populated alliance is on 2bars.

Congratulations ZOS. You killed PvP. Now the snowball is in effect and good luck getting any money off your players to actually invest in anything.

And your Free-To-Play event? Have you gone mad? Why would you entice new players to try out your product in such horrible state?

Amazing game, totally addicted but it has now become an absolute joke.

(Yes, yes, I know. PvE crowd is still gonna call PvPers a minority but for some of us PvP is all that we play the game for)

Oh mate, you are right...
About free to play event: they opened event despite of *** up state of server because of WOW Classic and not for anything else, like a *** beggar who begs for a players...

About pvp/pve: After Wrathstone game have been totally ruined. Why they changed established skills, nobody knows, but why they changed racial passives even Jesus doesnt know...

I have feeling this game will be closed soon...

Delparis

Better switch to other mmo as performance wont get any better before Q2 2020.

FF14 is by far a better mmo, best main story an mmo can have and best performance on every single mmo available.

PvP isn't a thing in FF14 however as the game focus much on PvE. Still tournament are made and a ranking team group is also available (ESO sent have a 2v2, 3v3 or 4v4 team pvp ranking or tournament)

Davor

everlastingodeb17_ESO wrote: »

Zos uk had profits of 50+ million euros in 2017 and 40+ in 2016( dont have 2018 data) not to mention Zos France and Zos Germany, yet they cant fix the server... Get skilled people you clearly have the financial resources needed.

I know I am later here. Like the old saying, why fix something if it isn't broke?

For Zenimax why spend money to fix something if they are still getting the 20+ millions? So in the higher up eyes, nothing is broken and working as planned.

commodore64

You can't complain about the lack of comms here, they're probably stuck in a loading screen, just like us.

ThirdHorseman

Delparis wrote: »

Better switch to other mmo as performance wont get any better before Q2 2020.

FF14 is by far a better mmo, best main story an mmo can have and best performance on every single mmo available.

PvP isn't a thing in FF14 however as the game focus much on PvE. Still tournament are made and a ranking team group is also available (ESO sent have a 2v2, 3v3 or 4v4 team pvp ranking or tournament)

You are on the wrong forum. You are looking for FF14 forum. Good luck and have fun. If you want to talk more about FF, please go to their subreddit and forum. Thank you

888and888

ThirdHorseman wrote: »

Delparis wrote: »

Better switch to other mmo as performance wont get any better before Q2 2020.

FF14 is by far a better mmo, best main story an mmo can have and best performance on every single mmo available.

PvP isn't a thing in FF14 however as the game focus much on PvE. Still tournament are made and a ranking team group is also available (ESO sent have a 2v2, 3v3 or 4v4 team pvp ranking or tournament)

You are on the wrong forum. You are looking for FF14 forum. Good luck and have fun. If you want to talk more about FF, please go to their subreddit and forum. Thank you

I think it will be quite cool if they set up a sub forum here, actually, named “Alternatives to ESO”.
It would cost way less than fix the performance.

Somber97866

Hey they can fix it by making all the grumbling dirty elite endgamers quit by nerfing damage and.............shazzam! Less ppl on the servers! Lol, they got their money out of them, who needs them anymore! Hahahahahahaha

Delparis

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

Elsonso

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

No one else will be playing?

daemonios

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

People who call for "fixing" this, do you realize ESO's combat is what sets it apart from 99% of other MMOs? It may not have been intended, but that's the thing about emergent gameplay - it never is intended by the developers. It's not exactly rocket science, it's relatively easy to do with the barest minimum of practice, there really isn't any reason to take it out. At most I think ZOS should reinforce the ways it teaches new players about this and other combat mechanics that end up stumping them e.g. in group content.

As for performance, I have no idea what impact it would have. I mean, it's double the actions per second, but I don't have the information to say whether performance issues are linked to combat calculations, and what impact reducing the number of actions per second would have.

Delparis

daemonios wrote: »

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

People who call for "fixing" this, do you realize ESO's combat is what sets it apart from 99% of other MMOs? It may not have been intended, but that's the thing about emergent gameplay - it never is intended by the developers. It's not exactly rocket science, it's relatively easy to do with the barest minimum of practice, there really isn't any reason to take it out. At most I think ZOS should reinforce the ways it teaches new players about this and other combat mechanics that end up stumping them e.g. in group content.

As for performance, I have no idea what impact it would have. I mean, it's double the actions per second, but I don't have the information to say whether performance issues are linked to combat calculations, and what impact reducing the number of actions per second would have.

Yeah go tell ESO devs you have to spend 1 week doing animation for a skill that will be cancelled because of an intended mechanic.
In that case why not remove completely skill animations... complete BS.

Elsonso

Delparis wrote: »

daemonios wrote: »

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

People who call for "fixing" this, do you realize ESO's combat is what sets it apart from 99% of other MMOs? It may not have been intended, but that's the thing about emergent gameplay - it never is intended by the developers. It's not exactly rocket science, it's relatively easy to do with the barest minimum of practice, there really isn't any reason to take it out. At most I think ZOS should reinforce the ways it teaches new players about this and other combat mechanics that end up stumping them e.g. in group content.

As for performance, I have no idea what impact it would have. I mean, it's double the actions per second, but I don't have the information to say whether performance issues are linked to combat calculations, and what impact reducing the number of actions per second would have.

Yeah go tell ESO devs you have to spend 1 week doing animation for a skill that will be cancelled because of an intended mechanic.
In that case why not remove completely skill animations... complete BS.

Animation is one of the few game development skills that I have never had to do, but from what I understand, this is not the wasted effort that you try to portray it as. The animation is never fully canceled, so the challenge comes with trying to make the animations so they are canceled at the right time and can flow into the next action. ZOS did a lot of work a couple years ago doing animation prioritization, and I am not sure where that work ended up.

Delparis

lordrichter wrote: »

Delparis wrote: »

daemonios wrote: »

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

People who call for "fixing" this, do you realize ESO's combat is what sets it apart from 99% of other MMOs? It may not have been intended, but that's the thing about emergent gameplay - it never is intended by the developers. It's not exactly rocket science, it's relatively easy to do with the barest minimum of practice, there really isn't any reason to take it out. At most I think ZOS should reinforce the ways it teaches new players about this and other combat mechanics that end up stumping them e.g. in group content.

As for performance, I have no idea what impact it would have. I mean, it's double the actions per second, but I don't have the information to say whether performance issues are linked to combat calculations, and what impact reducing the number of actions per second would have.

Yeah go tell ESO devs you have to spend 1 week doing animation for a skill that will be cancelled because of an intended mechanic.
In that case why not remove completely skill animations... complete BS.

Animation is one of the few game development skills that I have never had to do, but from what I understand, this is not the wasted effort that you try to portray it as. The animation is never fully canceled, so the challenge comes with trying to make the animations so they are canceled at the right time and can flow into the next action. ZOS did a lot of work a couple years ago doing animation prioritization, and I am not sure where that work ended up.

So you're telling me that ZoS got the ressources to make animation but not fix a bug that has been here for 5 years.
just lol.

daemonios

Delparis wrote: »

lordrichter wrote: »

Delparis wrote: »

daemonios wrote: »

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

People who call for "fixing" this, do you realize ESO's combat is what sets it apart from 99% of other MMOs? It may not have been intended, but that's the thing about emergent gameplay - it never is intended by the developers. It's not exactly rocket science, it's relatively easy to do with the barest minimum of practice, there really isn't any reason to take it out. At most I think ZOS should reinforce the ways it teaches new players about this and other combat mechanics that end up stumping them e.g. in group content.

As for performance, I have no idea what impact it would have. I mean, it's double the actions per second, but I don't have the information to say whether performance issues are linked to combat calculations, and what impact reducing the number of actions per second would have.

Yeah go tell ESO devs you have to spend 1 week doing animation for a skill that will be cancelled because of an intended mechanic.
In that case why not remove completely skill animations... complete BS.

Animation is one of the few game development skills that I have never had to do, but from what I understand, this is not the wasted effort that you try to portray it as. The animation is never fully canceled, so the challenge comes with trying to make the animations so they are canceled at the right time and can flow into the next action. ZOS did a lot of work a couple years ago doing animation prioritization, and I am not sure where that work ended up.

So you're telling me that ZoS got the ressources to make animation but not fix a bug that has been here for 5 years.
just lol.

Animation cancelling is your personal vendetta. ZOS don't consider it a bug, YOU do. Why would they throw resources at removing something the community values as a distinctive feature of the game?

ZOS are supposedly working on improving performance for the next couple of updates. Their goal is to get max bang for the buck, i.e. expend the fewest resources for the biggest performance increase. If it were as simple as removing animation cancelling, that's the way they'd go. The fact that they're trying other stuff is a good indication that animation cancelling isn't high on the list for performance improvement.

IwakuraLain42

lordrichter wrote: »

Delparis wrote: »

daemonios wrote: »

Delparis wrote: »

Just fix the LA weaving / animation cancelling bug and you'll notice a huge improvement in performance.

People who call for "fixing" this, do you realize ESO's combat is what sets it apart from 99% of other MMOs? It may not have been intended, but that's the thing about emergent gameplay - it never is intended by the developers. It's not exactly rocket science, it's relatively easy to do with the barest minimum of practice, there really isn't any reason to take it out. At most I think ZOS should reinforce the ways it teaches new players about this and other combat mechanics that end up stumping them e.g. in group content.

As for performance, I have no idea what impact it would have. I mean, it's double the actions per second, but I don't have the information to say whether performance issues are linked to combat calculations, and what impact reducing the number of actions per second would have.

Yeah go tell ESO devs you have to spend 1 week doing animation for a skill that will be cancelled because of an intended mechanic.
In that case why not remove completely skill animations... complete BS.

Animation is one of the few game development skills that I have never had to do, but from what I understand, this is not the wasted effort that you try to portray it as. The animation is never fully canceled, so the challenge comes with trying to make the animations so they are canceled at the right time and can flow into the next action. ZOS did a lot of work a couple years ago doing animation prioritization, and I am not sure where that work ended up.

That was back in the days when the dev's actually communicated with players and provided real answers to their changes. I think that was somewhere around either the Thieves Guild or Drak Brotherhood release when they changed animations, there was a rather long post explaining it.

The basic thing is that skill animations are split into a (non cancel-able/weaving-able) front part and a (cancel-able) tail end. Light attacks are either (animation-wise) so fast or have no front part, that's the reason why you have to weave LA->skill to make that reliable. There is unfortunately no consistent rule to where the split is for each skill.

It's all part of the wonderful and fast gameplay the game now has (and that the vast majority of players is unable/unwilling/unknown to use ...)

regime211

WoppaBoem wrote: »

All the hate, I feel sorry for you guys it must be really bad at your mega server. You know you can stop playing the game?

ZOS thank you for the update. Keep it up, steady progress is much appreciated.

And funny all these people of the forums who know (assume and think) they know exactly was is going on and what needs to be done with no inside information. In Holland we call this the best sailors are in the dry docks, they know everything better from the side line....

Man please you're talking to people who have played since Beta take that white knight crap somewhere else. The game is in a poor state and has been for awhile.

marius_buys

Anyone else getting weird lag spikes this week particularly? Ranges from 300 to 500 to 999
and I asked in my guild, I'm not the only one affected across an international community of players in the guild.

FierceSam

EU server performance always dips during PTS weeks - just suck it up

ZOS still don’t care about EU server players. They don’t play it, they don’t stream it, they don’t promote it.

I would be delighted to be proved wrong, but 12 months of shite performance, and 2 pinned posts from the bosses have produced no noticeable improvements other than “performance is better when the population is lower”.

Delparis

FierceSam wrote: »

EU server performance always dips during PTS weeks - just suck it up

ZOS still don’t care about EU server players. They don’t play it, they don’t stream it, they don’t promote it.

I would be delighted to be proved wrong, but 12 months of shite performance, and 2 pinned posts from the bosses have produced no noticeable improvements other than “performance is better when the population is lower”.

agree, i've noticed many lag spikes (+600 ms).
we're back again to the server lag issue, nothing will ever be fixed i guess

KoultouraS

The game has gotten truly weird in terms of consistency of performance.
I get the incompetence of the devs is the main barrier here,
--(just reminding that they decided to address issues present since beta , with a yearly upgrade after 5 freaking years !
Kudos to them , they deserve every penny they try to squeeze in by forcing us to change gear and race/class meta every 2 months!!)--
...but strangely the game has been performing like bananas lately .

I DizzySwinged a guy yesterday and found him to be flying 5 meters or so above me for some few secs (and I mean 6-9 secs or something). For the record I was totaly sober.

Also the delay here has been expanding up to intergalactical levels . I ve still been waiting for an onslaught ulti casted last monday to land on the enemy...

What a disgrace...

Malmai

SOON better...

maddiniiLuna

Hello,

In the first post or head of the topic there was a comment "we definitely could be doing a better job of keeping everyone appraised of how things are going while we are in process. We will be better messengers in the future keeping everyone up to date on longer-term fixes and updates.".

What happened with this? There is no Information anywhere. The current state of the game is so terrible sometimes. When i play Cyrodiil it looks like 2 *** doing light attacks. I've prepared a video to demonstrate how bad the situation actually is.

https://youtu.be/69I7G6ZiNTs

Done my best to not show any names, but I'm not perfect. Anyway - Check how the opponent would use ultimate, then i cast 2 spells and THEN the damage and effects from the ultimate hit me like wtf. There was such a huge delay in animation and actual hit, that i was able to cast spells in that time. It was so laggy i could barely see what is going to happen and no - i did not press perma block (this happened from being laggy).
At the end you can see how the second ultimate hits me, then i move around and because i moved around and the damage was applied so late i fell off. If the damage would have been applied before that i wouldn't have fallen off (but i most likely was dead regardless).

It's horrendous laggs when attacking a castle. Unplayable like this.

Elsonso

Roadmap plus one update since they published the road map. Not specific to EU, though.

Delparis

maddiniiLuna wrote: »

Hello,

In the first post or head of the topic there was a comment "we definitely could be doing a better job of keeping everyone appraised of how things are going while we are in process. We will be better messengers in the future keeping everyone up to date on longer-term fixes and updates.".

What happened with this? There is no Information anywhere. The current state of the game is so terrible sometimes. When i play Cyrodiil it looks like 2 *** doing light attacks. I've prepared a video to demonstrate how bad the situation actually is.

https://youtu.be/69I7G6ZiNTs

Done my best to not show any names, but I'm not perfect. Anyway - Check how the opponent would use ultimate, then i cast 2 spells and THEN the damage and effects from the ultimate hit me like wtf. There was such a huge delay in animation and actual hit, that i was able to cast spells in that time. It was so laggy i could barely see what is going to happen and no - i did not press perma block (this happened from being laggy).
At the end you can see how the second ultimate hits me, then i move around and because i moved around and the damage was applied so late i fell off. If the damage would have been applied before that i wouldn't have fallen off (but i most likely was dead regardless).

It's horrendous laggs when attacking a castle. Unplayable like this.

I can confirm that servers are still broken

FierceSam

Update 24

We’ve done jack **** about it...

But hey... check out those dragons #killdragonsforserverupgrade

BrightOblivion

Or maybe you can check the article where they've laid out their gameplan and are posting monthly updates. They posted one earlier this week in fact, explaining where they are and what they're doing...

https://www.elderscrollsonline.com/en-us/news/post/56681

(I'm on my phone and don't feel like dancing through formatting hoops, so feel free to copy and paste.)

Granted, if you feel they're a bunch of lying, contemptuous, greedy, mustache-twirling Saturday-morning-cartoon villains, what they're saying is meaningless, but that part's on your end.