How do we determine if issues are game coding related or server capacity related?

TineaCruris · April 2020

Since I started playing ESO I've been amazed at how fantastic of a game it is. But I've also been stunned by how much the performance changes based on factors like the time of day we play, the day of the week we play, and the number of players in the zones we are playing in. To me, this feels more like server load issues than it does game coding issues.

As players, as end users, how do we determine if the issues we are having are sever load related vs. game coding related?

I'm assuming that all these issues are a combination of some degree of both coding and server load, but how do we as users determine with confidence what is causing our many in game issues? (sorry, I'm not the kind of person that takes companies on their word, I have to have verification when things delivered aren't as advertised. So how do we verify the lag isn't mainly just server capacity overload?)

Danikat · April 2020

Other than curiosity why do we need to determine the cause of the lag?

We can't do anything about it, even if we think we know exactly what the cause is and I doubt ZOS is going to change their approach to fixing server problems just because someone on the forum thinks they've worked it all out and knows better than ZOS does what's causing the problem.

TineaCruris · April 2020

They say they are working on fixing things, but things are not getting fixed. Things are degrading further, not improving. Knowing why allows the consumer to make informed purchasing/investment decisions. And, it's always nice to know if someone is being honest with you or not. "Trust but verify" is an expression for a reason.

I'm not just curious. I'm trying to make informed decisions.

Pinja · April 2020

Well if your on PC you can use task manager to gauge performance. I think server lag wont effect your CPU, where renders and client side code will.
On xbox I think the camera will tell you whether it's your client or the server. If you can pan smoothly while it's lagging it's the server, if you pan in frames it's client.
This is a guess.
How do you know if it's server code vs hardware capabilities? Well you have to do math... also run the servers diagnostic tests to make sure the system is running to it's design specs.
Chances are you'll find the sever allocation is low.

oXI_Viper_IXo · April 2020

The general lag in prime time PvP with zerg on zerg fights is mainly due to server load.

The very specific issues of skill delay, rubberbanding, blacked out skill bars, inability to barswap, positional desync, and gap closers not working all arose right after the Feb 5th patch on console and with update 25 on PC, which means these are coding related issues.

Elsonso · April 2020

oXI_Viper_IXo wrote: »

The general lag in prime time PvP with zerg on zerg fights is mainly due to server load.

The very specific issues of skill delay, rubberbanding, blacked out skill bars, inability to barswap, positional desync, and gap closers not working all arose right after the Feb 5th patch on console and with update 25 on PC, which means these are coding related issues.

For Cyrodiil... I label all of that "software/coding related issues", whether it is due to server load or not.

Nestor · April 2020

If we, who have never seen the code, could determine what the issues are; they would have been fixed long ago by those who know the code.

VaranisArano · April 2020

If it's server capacity;
A. When ZOS says " Oh, we're upgrading our capacity to address this" after months of issues
B. When those issues improve after the upgrade

If it's coding:
A. It happens suspiciously just after a new patch or update
B. ZOS tries to fix it in a future patch. Well, okay, it often takes multiple future patches.
C. If it's big enough, it winds up in the official performance improvements thread.
D. There are no fixes.

Lysette · April 2020

And there are reasons as well outside of hard- and software - a flawed design which is causing the issues in the first place, IMO a combat system which is so twitchy and focused on effects, which are prone to cause all kind of performance issues.

Gilvoth · April 2020

because it's Broke and they need to Fix it!
NOW!

Iskiab · April 2020

The two issues are interrelated.

A lot of code out there is poor and not optimized. Usually companies use cheap coders or outsource to an outside contract (pay top dollar for cheap coders, but it can pay off if you just want a quick project and not a full time employee).

That’s where software engineering comes into play. Have someone experienced come in and fix all the mistakes.

Back to your question. Poor code is everywhere but you can brute force most problems with processing power.

About ESO, it’s a bit of both. Bad code leading to poor performance as well as lack of capacity.

You can tell it’s part a capacity issue because during pvp events they devote more resources to pvp and performance improves, and PvErs complain about dungeons. Once the event is over pve goes back to normal and pvp declines with less players.

Code and architecture are also undoubtedly part of the issue too. ESO’s engine is third party, then ESO adapted it for their game (localization). Problem with that is depending on how it was done, updating the engine could be problematic. Plus ESO was designed with smaller servers in mind, they were amalgamated with the mega server. Who knows what the engine can handle.

Anyways, ESO is having problems from all directions.

redgreensunset · April 2020

We can't. We have access to nothing but what they've told us and what ZOS staff has said is that it's a coding issue and that there is no silver bullet (ie, there's no quick, simple, magic solution that'll fix everything).
In other words, we're in for the long haul. If a fix is possible at all it'll take months, probably the rest of the year at minimum, to fix and we'll get a ton of bugs and broken code in the attempt that'll take a while to fix too. So my guess is that we won't know until about this time next year if their fixes are working or not and the ride to get there will be extremely bumpy. Whether or not a player is willing to ride that out on the possibility that they can get things fixed is up to the individual.

MusCanus · April 2020

Nestor wrote: »

If we, who have never seen the code, could determine what the issues are; they would have been fixed long ago by those who know the code.

Agreed. Too bad these people don't work for ZOS.

Pinja · April 2020

Reguarding the math required to answer this question. I'd need to know:

The specs of the server?
How big is the compiled program?
The bandwidth?
The number of Max players/interactions?
And the programing language?

Get me those answers and we'll be closer to an answer.

ShawnLaRock · April 2020

The engine is ancient, and just can’t handle all of the calculations we throw at it. That’s it.

The “balancing” they are constantly reworking is only to try and keep us within the confines of what it can barely process & then represent in game.

It’s dumb-down the procs in the current engine, or create a new one.

S.

SeaGtGruff · April 2020

I think some of the problems are graphics/animation related.

What I mean is, sometimes when I'm fighting in PvE, my character will seem to freeze-- well, he's still "wagging his tail" (as I call it) while crouched in attack stance with his greatsword ready, but my skills and attacks don't seem to fire, and I can't move him-- yet all of a sudden the XP indicator will pop up in the bottom right corner of the screen to show much XP I've just gained by killing the MOB or boss, even though they still appear to be very much alive and in my face. Then, after the XP indicator has popped up and scrolled upward, suddenly my skills and attacks will appear to fire in rapid sequence and the MOB or boss will keel over dead, sometimes before my attacks finish firing.

To me, the popping up of the XP indicator shows that my skills and attacks were firing all along, and were doing damage all along, even though I couldn't see it happening because the animations of my character and the MOB or boss weren't being updated in time with what was actually happening. My actions were going to the server, and the server was processing them, but my client wasn't getting timely information about what to draw. I know everyone will say that that's the server's fault. But is it possible that the holdup was occurring somewhere between the server and my client-- that the data going from my client to the server was being sent in a reasonably timely fashion, but that the data coming back from the server to my client was getting buffered or otherwise held up for some reason even though the server was actually sending it out in a more or less timely manner? And is it possible that part of the lag is in my client and how quickly it draws the animations in response to the data it's receiving? Not saying, just asking.

JAwtunes · April 2020

IDK, but I'm only here for the crown crates. A+ game, thanks zos

SeaGtGruff · April 2020

To add just a bit more explanation/context to my question, I sometimes get these sorts of lags on the internet when I'm not even playing ESO, such as browsing a site that suddenly seems to stop responding to my clicks even though I can scroll the browser just fine (i.e., it apparently isn't my computer or tablet that's frozen), although maybe if I scroll the window there will be white space at the point where the browser hasn't received information yet about what to display; or watching a streamer on Twitch and suddenly the stream freezes and I have to go back to the list of streamers, choose a different streamer, then go back to the streamer I was watching (which is now working fine).

These occasional lags and freezes are happening to me outside of ESO, so clearly they can't be caused by ZOS's ESO servers. How much of the intermittent problems I experience in ESO are due to the servers, and how much is due to other things?

red_emu · April 2020

It doesn't help matters that prime time Pvp has now become full pop stack vs full pop stack. Throw in a few ball groups and it takes 20 seconds to swap bars.

This game is unplayable in it's current state.

And don't fool yourself. ZOS will never invest money into fixing this.

Shareholders can see the game is being played by enormous amount of people spending thousands of pounds/dollars/euro on shiny mounts and it's a simple business decision.

If it brings in money - don't fix it.

So all of us have to understand. This is the way Eso is going to be from now on. There is no fix for it and it's not the devs fault. They would love to give us a smooth gameplay but at the end of the day, business is business and money turns the wheels.

VaranisArano · April 2020

SeaGtGruff wrote: »

To add just a bit more explanation/context to my question, I sometimes get these sorts of lags on the internet when I'm not even playing ESO, such as browsing a site that suddenly seems to stop responding to my clicks even though I can scroll the browser just fine (i.e., it apparently isn't my computer or tablet that's frozen), although maybe if I scroll the window there will be white space at the point where the browser hasn't received information yet about what to display; or watching a streamer on Twitch and suddenly the stream freezes and I have to go back to the list of streamers, choose a different streamer, then go back to the streamer I was watching (which is now working fine).

These occasional lags and freezes are happening to me outside of ESO, so clearly they can't be caused by ZOS's ESO servers. How much of the intermittent problems I experience in ESO are due to the servers, and how much is due to other things?

If its just happening to you, its you. Your internet, your system, etc.

If its happening to other players on the forums/your guildmates/your discords, its ZOS.

If it happens to other players, but you seem to be hit first/worst, its probably a combination of you and ZOS.

The_Old_Goat · April 2020

This sounds like a job for Scooby and the gang!

Pinja · April 2020

SeaGtGruff wrote: »

I think some of the problems are graphics/animation related.

What I mean is, sometimes when I'm fighting in PvE, my character will seem to freeze-- well, he's still "wagging his tail" (as I call it) while crouched in attack stance with his greatsword ready, but my skills and attacks don't seem to fire, and I can't move him-- yet all of a sudden the XP indicator will pop up in the bottom right corner of the screen to show much XP I've just gained by killing the MOB or boss, even though they still appear to be very much alive and in my face. Then, after the XP indicator has popped up and scrolled upward, suddenly my skills and attacks will appear to fire in rapid sequence and the MOB or boss will keel over dead, sometimes before my attacks finish firing.

To me, the popping up of the XP indicator shows that my skills and attacks were firing all along, and were doing damage all along, even though I couldn't see it happening because the animations of my character and the MOB or boss weren't being updated in time with what was actually happening. My actions were going to the server, and the server was processing them, but my client wasn't getting timely information about what to draw. I know everyone will say that that's the server's fault. But is it possible that the holdup was occurring somewhere between the server and my client-- that the data going from my client to the server was being sent in a reasonably timely fashion, but that the data coming back from the server to my client was getting buffered or otherwise held up for some reason even though the server was actually sending it out in a more or less timely manner? And is it possible that part of the lag is in my client and how quickly it draws the animations in response to the data it's receiving? Not saying, just asking.

Took me a second to think about this but it could be and probably is that the server that handles the battle and the server that handles your character information is two different servers. No matter where you go whether it be Cyrodiil or Glenumbra there's a server that keeps track of your player xp and items, verifying and saving them.
What could happen is that the broadband for the server hosting the battle is too divided among all the connected players but it's local connection to the server that updates your Xp isn't crowded with interactions. Nor is the XP server processing as many complicated requests, so it gets to you faster.
This theory is supported by the idea that your charterer can change instances without needing to transfer and save data to that server. Where each server doesn't need to delete or save a copy of your character (sounds like it'd be a design for disaster otherwise.)

If it were an issue with client side code and animations you'd still hear sound and the enemy would drop when it's own animation is prompted. The fact that it's happening all at once means it's a backlog from the server. As animation canceling will teach you there is no que for an animation that could back up in the memory and fire all at once. It would mean you could hit block while lagging and cancel all the animations you had stored in que. Rather It waits on instructions from the server to fire animations.
When the client does lag you'll notice everything seize up, frame rate, tail wag, ect.
Once again if it were a glitch in the animation code other functions of combat would work like enemy position, your HP and resources, and new animations would be prompted over the old animation, because their isn't a local que that says one has to go after the other. That que or cool down timer is managed by the server.

I would think.
*reiteration the server may still be processing it on time but it's the broadband.

Thinking about it, this does happen when you hold block and then you're frozen walking around holding block till you your game crashes, so there very well may be a local que. But then again the server fails to respond in this instance and you do get a connection error.

Vaoh · April 2020

There’s no way to know for sure.

However, from my experience as a PS4 Pro player, there were noticeable (very subtle) decreases in performance every single patch beginning before Morrowind but especially with Morrowind. Truly subtle changes. I caught on only because I’d begin to notice those little things. A load screen taking longer, characters/NPCs taking longer to load, fishy behavior in PvP with my characters skills, tiny desyncs everywhere, odd visual bugs due to loading failure, NPCs no longer exhibiting perfect AI behavior, formerly perfect mechanics now feeling clunky or breaking (gap closers), etc.

If it was server capacity then I do not think they would willingly kill off their PvP population and allow for his massive storm of complaints/player loss. Plus that would not account for many of these issues.

My educated guess is that it is coding-related. The same way they have poor internal communication in terms of game balance/sets (Elsweyr to Scalebreaker to Dragonhold DoT buffs and nerfs), they may have poor organizational, maintenance, and whatever other skills involved with the coding side of the game are. Basically the code being put into the game is sloppy and causes internal conflicts in a plethora of ways (NB Incap silence changed not because of balancing reasons, but because it conflicted internally with the reworked block).

This is why I have zero faith in ESO ever playing or feeling even remotely how it used to.

ImmortalCX · April 2020

TineaCruris wrote: »

Since I started playing ESO I've been amazed at how fantastic of a game it is. But I've also been stunned by how much the performance changes based on factors like the time of day we play, the day of the week we play, and the number of players in the zones we are playing in. To me, this feels more like server load issues than it does game coding issues.

As players, as end users, how do we determine if the issues we are having are sever load related vs. game coding related?

I'm assuming that all these issues are a combination of some degree of both coding and server load, but how do we as users determine with confidence what is causing our many in game issues? (sorry, I'm not the kind of person that takes companies on their word, I have to have verification when things delivered aren't as advertised. So how do we verify the lag isn't mainly just server capacity overload?)

You're not being paid to test, so you don't find out.

Knowing the answer, won't help you in any way. Just play. Or not.

Leave this to the experts who are employed by zos.

Neoealth · April 2020

Maybe it's both their server and code that sucks.

Anhedonie · April 2020

Without seeing the code, we can't.
But I remember the times when things worked almost perfectly, due to a much different architecture.

Neoealth · April 2020

Anhedonie wrote: »

Without seeing the code, we can't.
But I remember the times when things worked almost perfectly, due to a much different architecture.

I can see the code....

TineaCruris · April 2020

red_emu wrote: »

It doesn't help matters that prime time Pvp has now become full pop stack vs full pop stack. Throw in a few ball groups and it takes 20 seconds to swap bars.

This game is unplayable in it's current state.

And don't fool yourself. ZOS will never invest money into fixing this.

Shareholders can see the game is being played by enormous amount of people spending thousands of pounds/dollars/euro on shiny mounts and it's a simple business decision.

If it brings in money - don't fix it.

So all of us have to understand. This is the way Eso is going to be from now on. There is no fix for it and it's not the devs fault. They would love to give us a smooth gameplay but at the end of the day, business is business and money turns the wheels.

I'm afraid you may be accurately describing the situation. This is why I'm trying to figure out if the issues are actually code related, or if they are more accounting related. It costs a lot of money to host a ton of servers for a huge gaming community. This is a recurring cost.

RefLiberty · April 2020

TineaCruris wrote: »

red_emu wrote: »

It doesn't help matters that prime time Pvp has now become full pop stack vs full pop stack. Throw in a few ball groups and it takes 20 seconds to swap bars.

This game is unplayable in it's current state.

And don't fool yourself. ZOS will never invest money into fixing this.

Shareholders can see the game is being played by enormous amount of people spending thousands of pounds/dollars/euro on shiny mounts and it's a simple business decision.

If it brings in money - don't fix it.

So all of us have to understand. This is the way Eso is going to be from now on. There is no fix for it and it's not the devs fault. They would love to give us a smooth gameplay but at the end of the day, business is business and money turns the wheels.

I'm afraid you may be accurately describing the situation. This is why I'm trying to figure out if the issues are actually code related, or if they are more accounting related. It costs a lot of money to host a ton of servers for a huge gaming community. This is a recurring cost.

Pretty simple.
If the issue is code related client side (example, client side mem leaks, bad CPU core utilisation etc).
Solution is even more simple:
https://www.nzxt.com/camapp
I never had high resource util when performance suffered. Enough said.

Which leaves us to your two questions.
Server capacity or code/engine itself.
Nowadays increasing shards and overall server capacity is not that expensive especially for big companies, they would solve it already.
If you are looking at 2020 performance improvement plans, where they talk about pet behaviour rewrite, better server multicore utilisation, fixture loading etc...
I can say that patch roadmap notes already answered your questions.

TineaCruris · April 2020

RefLiberty wrote: »

TineaCruris wrote: »

red_emu wrote: »

It doesn't help matters that prime time Pvp has now become full pop stack vs full pop stack. Throw in a few ball groups and it takes 20 seconds to swap bars.

This game is unplayable in it's current state.

And don't fool yourself. ZOS will never invest money into fixing this.

Shareholders can see the game is being played by enormous amount of people spending thousands of pounds/dollars/euro on shiny mounts and it's a simple business decision.

If it brings in money - don't fix it.

So all of us have to understand. This is the way Eso is going to be from now on. There is no fix for it and it's not the devs fault. They would love to give us a smooth gameplay but at the end of the day, business is business and money turns the wheels.

I'm afraid you may be accurately describing the situation. This is why I'm trying to figure out if the issues are actually code related, or if they are more accounting related. It costs a lot of money to host a ton of servers for a huge gaming community. This is a recurring cost.

Pretty simple.
If the issue is code related client side (example, client side mem leaks, bad CPU core utilisation etc).
Solution is even more simple:
https://www.nzxt.com/camapp
I never had high resource util when performance suffered. Enough said.

Which leaves us to your two questions.
Server capacity or code/engine itself.
Nowadays increasing shards and overall server capacity is not that expensive especially for big companies, they would solve it already.
If you are looking at 2020 performance improvement plans, where they talk about pet behaviour rewrite, better server multicore utilisation, fixture loading etc...
I can say that patch roadmap notes already answered your questions.

I am not understanding most of what you just posted, and I'm not sure you do either. The NZXT cam software is neat and informative, but it only tells us what resources our system is using and other PC specific information. It doesn't relate to how much server load there is or whether or not the lag is coding related or server capacity related.

Uryel · April 2020

TineaCruris wrote: »

As players, as end users, how do we determine if the issues we are having are sever load related vs. game coding related?

There really isn't a point. Either it's your rig, and it's up to you to fix it, or it's not, and then it's up to them to fix it.

My rig can run much more demanding games at much better performances ? Then it's not my rig. Problem solved. Or rather, not solved, it would be alot easier to fix my rig than talk some sense into ZOS.