Performance Theory: Reverse Desync, Repairing Band Aid Patches, And Cloud based solutions.
Maintenance for the week of April 14:
• PC/Mac: No maintenance – April 14
• Xbox: NA and EU megaservers for maintenance – April 16, 6:00AM EDT (10:00 UTC) - 9:00AM EDT (13:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – April 16, 6:00AM EDT (10:00 UTC) - 9:00AM EDT (13:00 UTC)
• PC/Mac: No maintenance – April 14
• Xbox: NA and EU megaservers for maintenance – April 16, 6:00AM EDT (10:00 UTC) - 9:00AM EDT (13:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – April 16, 6:00AM EDT (10:00 UTC) - 9:00AM EDT (13:00 UTC)
Pinja
✭✭✭✭
Hello everyone.
This thread is to explain a concept I’ve talked about all over the forums I call the reverse dysync.
A classic dysync comes about when things don’t line up with the server and you get things like the snipe glitch.
https://www.youtube.com/watch?v=rl-n9WwNlko
This glitch was my first run of the term desync and is what I classically think of. Even though it means much more like positioning glitches, *health bar glitches, ect.
Either way following a shoddy pattern of development, the snipe glitch and fellow desyncs have reappeared and disappeared over the patches. In other words, there were junk repairs made that might have never gotten taken out of the game that may negatively affect performance. A notable example block changes didn’t help at all and got stuck in the game, how much more meta code is just lying around because someone abandoned it.
Now the reverse dysync is something I saw in February when abilities were not firing. Instead of getting hit with abilities all at once, people were not allowed to fire abilities on targets. The inverse of a dysync like if one person or the server was laging, players were not allowed to submit abilities on targets. I’d seen a few posts to support this, one questions whether one person's lag would cause all to lag. It’s as if the server waits on clients before a processing, instead of spamming a lagging client with abilities all at once. Potentially done as a solution to fix desync, hence the name reverse desync.
I however started using the reverse dysync as an example and label to a larger problem plenty of people have observed. Bad patches. When it comes to some patches the Devs don’t know how to hit the Undo button. U25 was a bad one. Going as far back some still blame the lighting patch. Not getting too far into details, a team should go back and take the time to clean up those junk piles I was talking about, and patch the Band Aid. They should ask what were these changes for? Are they causing problems? And is there another way to achieve the same goal?
A big one server side Anti Cheat. Could you come up with a client side Anti-Cheat and would that work better?
Speaking on the topic of Anti-Cheat I came up with another realization and potential solution that may deserve it’s own thread if it has credence.
I was in a Discord talking to someone and they said pretty much because of the way the games latency and positioning checks work your pretty much playing a guessing game with your positioning when it comes to AoEs.
Then I thought about the way the game checks positioning and may have run across something overlooked. Because positioning checks are moved server side, there may be a double jeopardy running between them. It would make the game run smoother if you got rid of all the client side checks or server checks vice versa. As it may be now abilities can be canceled out by two positioning checks if people aren’t moving slow enough for server and client to be in sync.
No lie, it may be a better investment and solution to run the game like cloud based gaming, or get the process as streamlined to it as possible. As the game may be running now you have redundancies client side.
If Zos continues to fail at fixing performance they may want to hire a third party like Shadow to fix it and set them up. ESO puts a lot of pressure on systems and cloud based gaming may be the answer to all it’s problems.
This is a really good idea. Make me CEO for a week. I’ll write the contract. Let’s get it done.
Edited by Pinja on August 18, 2020 6:42PM This thread is to explain a concept I’ve talked about all over the forums I call the reverse dysync.
A classic dysync comes about when things don’t line up with the server and you get things like the snipe glitch.
https://www.youtube.com/watch?v=rl-n9WwNlkoThis glitch was my first run of the term desync and is what I classically think of. Even though it means much more like positioning glitches, *health bar glitches, ect.
Spoiler
* There is a health bar glitch where someone will be at 100% health but look like they're lower sometimes near death 3%. Since everyone can see it, it’s not a desync but a UI error.
Either way following a shoddy pattern of development, the snipe glitch and fellow desyncs have reappeared and disappeared over the patches. In other words, there were junk repairs made that might have never gotten taken out of the game that may negatively affect performance. A notable example block changes didn’t help at all and got stuck in the game, how much more meta code is just lying around because someone abandoned it.
Now the reverse dysync is something I saw in February when abilities were not firing. Instead of getting hit with abilities all at once, people were not allowed to fire abilities on targets. The inverse of a dysync like if one person or the server was laging, players were not allowed to submit abilities on targets. I’d seen a few posts to support this, one questions whether one person's lag would cause all to lag. It’s as if the server waits on clients before a processing, instead of spamming a lagging client with abilities all at once. Potentially done as a solution to fix desync, hence the name reverse desync.
I however started using the reverse dysync as an example and label to a larger problem plenty of people have observed. Bad patches. When it comes to some patches the Devs don’t know how to hit the Undo button. U25 was a bad one. Going as far back some still blame the lighting patch. Not getting too far into details, a team should go back and take the time to clean up those junk piles I was talking about, and patch the Band Aid. They should ask what were these changes for? Are they causing problems? And is there another way to achieve the same goal?
A big one server side Anti Cheat. Could you come up with a client side Anti-Cheat and would that work better?
Speaking on the topic of Anti-Cheat I came up with another realization and potential solution that may deserve it’s own thread if it has credence.
I was in a Discord talking to someone and they said pretty much because of the way the games latency and positioning checks work your pretty much playing a guessing game with your positioning when it comes to AoEs.
Then I thought about the way the game checks positioning and may have run across something overlooked. Because positioning checks are moved server side, there may be a double jeopardy running between them. It would make the game run smoother if you got rid of all the client side checks or server checks vice versa. As it may be now abilities can be canceled out by two positioning checks if people aren’t moving slow enough for server and client to be in sync.
No lie, it may be a better investment and solution to run the game like cloud based gaming, or get the process as streamlined to it as possible. As the game may be running now you have redundancies client side.
If Zos continues to fail at fixing performance they may want to hire a third party like Shadow to fix it and set them up. ESO puts a lot of pressure on systems and cloud based gaming may be the answer to all it’s problems.
- It would have the security of a closed system so you’d stop cheat engine and third party software from playing on Shadows systems.
- No cheats,
- No server side checks,
- No system/client latency,
- No bad performance,
This is a really good idea. Make me CEO for a week. I’ll write the contract. Let’s get it done.
3
-
idk✭✭✭✭✭
✭✭✭✭✭I suggest you read the very post you linked for the client-side anti-cheat. At least they say maybe we will have no cheats as the seem to understand they are just guessing at what would happen. However, you are speaking in absolute terms that we would have no cheats. Also that this would solve all latency problems and eliminate the bad performance issues when in fact we have had issues of latency and lag to various degrees as well as poor performance over the years, even when we had a trusted client model.
I understand that we think of all these amazing ideas to solve the problems but as armchair developers (and armchair CEOs) we tend to not take in the full picture.1 -
Pinja✭✭✭✭I suggest you read the very post you linked for the client-side anti-cheat. At least they say maybe we will have no cheats as the seem to understand they are just guessing at what would happen. However, you are speaking in absolute terms that we would have no cheats. Also that this would solve all latency problems and eliminate the bad performance issues when in fact we have had issues of latency and lag to various degrees as well as poor performance over the years, even when we had a trusted client model.
I understand that we think of all these amazing ideas to solve the problems but as armchair developers (and armchair CEOs) we tend to not take in the full picture.
Very fair,
I didn't say cure all latency as network latency still exist until networks stop throttling and encountering errors, even with 5G.
However, this idea nukes so many problems only network roaches will be left for the hamsters to feast on.
Let me explain why this idea is so awesome. Lag happens in a lot of different variables. The lag of individual CPUs, both client an server, and the network lag.
Looking at how it reduces network lag Shadow says they have protocols for the best routing from end to end. Unlike ESO which I hear uses TCP. It also reduces the number of network transactions per in game action, which may make a huge difference. Instead of sending all that clunky data like player IDs, attack stats, Current position, and ect. It receives one stream of data, and from what I would imagine sends one type of data, the input data. It potentially reduces packet sizes, relieving network burden.
Now that was actually the failing end of this concept. As it may turn out the packet streaming is bigger or smaller, probably depending on chosen resolution settings. The thing with network latency though is your going to have it either way, but with cloud based gaming you reduce it's foot print.
There's plenty of performance issues caused client side that hosting on a top end PC will fix, like crashing. This would be a band aid to replace a band aid. It also should eliminate frames, load screens, and plenty of other client struggles.
Now the big modification between this and the current set up is the Anti cheat. Because the game would be hosted on a Shadow PC, they'd control what can and can't be installed like Sony & Xbox. This brings out a world of application as you'd have to be a Paid Russian Federation hacker to illegally tap the services and install your mods. Salvage some stuff from the client-side anti-cheat for add-ons and your in business.
With checks gone, big performance increase. With "clients" hard wired to the servers, big performance increase. With "client" memory and CPU majoritively upgraded and controlled, big performance increase.
Check out this concept video:
https://www.youtube.com/watch?v=pHQw4cs8oRI
Supposedly can work on any device so perhaps even Stadia integration.
What this does for sure is get rid of third party mods with better protections than server side checks.
Gets rid of system/client latency because you'll have better systems that don't render graphics till they get to the user end.
And it just may be better than bad for a majority of people.Edited by Pinja on August 18, 2020 8:18PM1 -
Rungar✭✭✭✭✭
✭the "fixable" lag is eso is a simple equation of sequential inputs vs sequential outputs.
Other games have conservative limitations ( i.e cooldowns) on how often you can use a skill where eso has an extremely aggressive ( too aggressive if you ask me) model, no cooldowns, multiple global timers.
thus in other games you might be limited to one action per second, in eso its 3+ actions per second. light attack, skill, bash or block, and up to 3 proc sets, class group buffs.... Now the light attack has calculations for applying a condition, critical hit and applying an enchant. The skill itself could have any number of actions depending on if it was aoe etc, and the bash or block has its own actions.
combining all these things is the pinnacle of eso skilled gameplay. Our 3 actions per second might be closer to 30 or more server requests per second for one person depending on what is calculated serverside. Add in the sequential nature of how these requests are processed and you begin to understand the broken nature of eso combat. A whole bunch of requests might be processed before even one of yours is processed all within a second.
Sooner or later they will have to accept that their model is, was, and always will be too aggressive and will have to move to a more conservative model of 1 action (skill) per second +emergency DEFENSIVE actions ( dodge or block an attack). If they do not they will end up with all kinds of gutting like 3 second cooldowns on aoe etc.even the smallest flicker of light can overcome any amount of darkness
Rungar's Mystical Emporium of ESO REVAMP Ideas1 -
Pinja✭✭✭✭the "fixable" lag is eso is a simple equation of sequential inputs vs sequential outputs.
Other games have conservative limitations ( i.e cooldowns) on how often you can use a skill where eso has an extremely aggressive ( too aggressive if you ask me) model, no cooldowns, multiple global timers.
thus in other games you might be limited to one action per second, in eso its 3+ actions per second. light attack, skill, bash or block, and up to 3 proc sets, class group buffs.... Now the light attack has calculations for applying a condition, critical hit and applying an enchant. The skill itself could have any number of actions depending on if it was aoe etc, and the bash or block has its own actions.
combining all these things is the pinnacle of eso skilled gameplay. Our 3 actions per second might be closer to 30 or more server requests per second for one person depending on what is calculated serverside. Add in the sequential nature of how these requests are processed and you begin to understand the broken nature of eso combat. A whole bunch of requests might be processed before even one of yours is processed all within a second.
Sooner or later they will have to accept that their model is, was, and always will be too aggressive and will have to move to a more conservative model of 1 action (skill) per second +emergency DEFENSIVE actions ( dodge or block an attack). If they do not they will end up with all kinds of gutting like 3 second cooldowns on aoe etc.
Well let's look at other games and compare it. Battlefield as I like to use as an example does have kinda unresponsive combat where you are stuck watching your character, but it's servers calculate and handle thousands of objects. It's not a hit scan game so an automatic weapon produces several objects a second. This is not necessarily the same as several inputs a second, but fits in the same category as procs as it's just a server generated process. All the input does is send a signal initiating generation, but each input may grapple its own latency. The cloud solution removes latency from point of process. IE: the areas that do the processing.
Eso is a unique game. Closest coming to mind is fighting games on the quick input even they might not be as responsive.Edited by Pinja on August 18, 2020 10:11PM0 -
carlos424✭✭✭✭✭the "fixable" lag is eso is a simple equation of sequential inputs vs sequential outputs.
Other games have conservative limitations ( i.e cooldowns) on how often you can use a skill where eso has an extremely aggressive ( too aggressive if you ask me) model, no cooldowns, multiple global timers.
thus in other games you might be limited to one action per second, in eso its 3+ actions per second. light attack, skill, bash or block, and up to 3 proc sets, class group buffs.... Now the light attack has calculations for applying a condition, critical hit and applying an enchant. The skill itself could have any number of actions depending on if it was aoe etc, and the bash or block has its own actions.
combining all these things is the pinnacle of eso skilled gameplay. Our 3 actions per second might be closer to 30 or more server requests per second for one person depending on what is calculated serverside. Add in the sequential nature of how these requests are processed and you begin to understand the broken nature of eso combat. A whole bunch of requests might be processed before even one of yours is processed all within a second.
Sooner or later they will have to accept that their model is, was, and always will be too aggressive and will have to move to a more conservative model of 1 action (skill) per second +emergency DEFENSIVE actions ( dodge or block an attack). If they do not they will end up with all kinds of gutting like 3 second cooldowns on aoe etc.
Honestly, skill spamming is only part of the problem, and probably not even the biggest part. I’ve had plenty of lag, rubber banding, random load screens, etc., just running around the world in basically unpopulated areas. How do we explain that? I’m not casting any skills. Just rendering landscape is causing issues. ESO has way bigger problems than people using skills.1 -
Rungar✭✭✭✭✭
✭
Honestly, skill spamming is only part of the problem, and probably not even the biggest part. I’ve had plenty of lag, rubber banding, random load screens, etc., just running around the world in basically unpopulated areas. How do we explain that? I’m not casting any skills. Just rendering landscape is causing issues. ESO has way bigger problems than people using skills.
its actually pretty simple. You are on a megaserver. The point of a megaserver is shared resources. Even your location has to be calculated by the server. People dont always have to be around you for their cumulative actions to affect you.even the smallest flicker of light can overcome any amount of darkness
Rungar's Mystical Emporium of ESO REVAMP Ideas0 -
Pinja✭✭✭✭
its actually pretty simple. You are on a megaserver. The point of a megaserver is shared resources. Even your location has to be calculated by the server. People dont always have to be around you for their cumulative actions to affect you.
It depends on what CPU does what. There's still the partition Cyrodiil Idea. I was told ZOS was an indie company and couldn't afford it... It'd be a good band aid though. But I talked about this with someone while discussing cold storage. The only time you theoretically might get lag from the whole of everybody, the megaserver, is when loading and saving character data. Otherwise, the core of functions like combat are managed by individual CPUs down to a certain level. However, even this function is likely divided among storage servers.0 -
Pinja✭✭✭✭Another thing I was thinking about has been the change in what lag is and does. Nowadays there's lag from a deficit or circumvent in design. Back in the day you had regular lag, where things would get frammy and slow down. Now things don't necessarily go freeze-frame, but they just become unresponsive. Before you could play in the lag and still fire abilities, just the whole battle would be lagging. Like their would be a spike were things would slow down you could hit and fire abilities watch the battle progress frame by frame and continue to play.
I'm kinda going to basely point and say it may've potentially changed to reduce crashing. I don't think the difference of the two lags is documented enough for me to go and pinpoint a patch. Perhaps even at one point in time they overlapped.
I got some clips though:
https://gamerdvr.com/gamer/rasta-banana-xd/video/107395398
This first clip the game ran so smooth back the day you hardly notice. But there is a slight freeze that I can play through.
So I spent like 20mins looking through old clips did not find a substantial lag spike on xbox, so revisions should be taken seriously.
Here's a clip from a guild lead that I use to work with that has the same half second of lag.
https://gamerdvr.com/gamer/render-sts/video/860279601 -
SamanthaCarter✭✭✭Just nerf defile it’s the main desynch issue since years and snipe should be a spammable ability with low dmg0
-
Rungar✭✭✭✭✭
✭i dont blame zos for compensating for cheating by moving things serverside and using a service to prevent ddos attacks. Both have had a major impact on the performance of the game. Clearly the ddos service is a protection racket so i almost feel sorry for zos but this is gangsterworld afterall and the big gangsters still eat the small ones.
They could do something about this crappy combat system though and i dont mean aoe hard timers. They have to finally admit that its more than they can handle and bring it down a level so it will actually work. The only way to do that is to address a certain mechanic that gets everyones back up.
even the smallest flicker of light can overcome any amount of darkness
Rungar's Mystical Emporium of ESO REVAMP Ideas0 -
Tandor✭✭✭✭✭
✭✭✭✭✭Changing the way the game functions including the whole combat system in order to fix performance in Cyrodiil is fine, just so long as the changes are only introduced in Cyrodiil.1 -
Pinja✭✭✭✭I was looking around at new performance conflicts and speculation. I heard about this before I wrote my last post, but wanted to verify. Now I figure I'll just ask.
Does ESO add latency depending on location? I heard about this and thought that lag compensation may be too advanced a feature for ESO servers currently. The person(s) that built it might not be around any more, and it may have gone rouge and overlooked without its master.
Who knows what it could be stacking with all the servers other problems.
If it exists, what if it accounted for a Stadia player being two leagues away from a server and slows down everyone. A theory that may replace the reverse desync theory in timing, as they both would conveniently happened around U25. However, this lag compensation theory doesn't explain console lag.
@relentless_turnip I'm curious on your thoughts.Edited by Pinja on August 22, 2020 8:08PM0 -
Tandor✭✭✭✭✭
✭✭✭✭✭TineaCruris wrote: »
Why not make cyrodiil a seperate, pay for game? That seems to be what you are implying should happen.
I'm not remotely implying that, how on earth can you draw that conclusion from what I said? People have been arguing for a long time that skill balancing and other combat changes etc should be done separately for PvP and PvE, and that's the only point I'm making. If there are major changes to the way the game functions including the whole combat system that are needed in order to fix performance in Cyrodiil then there is no reason for those changes to be applied to the rest of the game where they're not needed. It's perfectly standard practice in MMORPGs for skills to work differently as between PvP and PvE.Edited by Tandor on August 22, 2020 10:32PM1 -
relentless_turnip✭✭✭✭✭
✭✭✭My thoughts on stadia and the timing of its release shortly after harrowstorm make me believe the new latency issues are too coincendental to be anything but a side effect of stadias implementation.
I believe they told us about the block changes being moved server side as they knew players would feel the difference in gameplay. In fact I think the change they made was much greater.
You'll notice in addition they have sold this year as the year of performance allowing any explanation to issues that arise to be put down to attempts to fix performance. They have since not ever addressed harrowstorm as a substantial factor in the decline in performance. Instead you will notice in any correspondence since they talk about it as if it is an ongoing issue and not an issue that got drastically worse upon the release of update 25.
I can't offer you any technical explanation as to what they did. I only see a pattern as to how it was communicated.2 -
idk✭✭✭✭✭
✭✭✭✭✭the "fixable" lag is eso is a simple equation of sequential inputs vs sequential outputs.
Other games have conservative limitations ( i.e cooldowns) on how often you can use a skill where eso has an extremely aggressive ( too aggressive if you ask me) model, no cooldowns, multiple global timers.
thus in other games you might be limited to one action per second, in eso its 3+ actions per second. light attack, skill, bash or block, and up to 3 proc sets, class group buffs.... Now the light attack has calculations for applying a condition, critical hit and applying an enchant. The skill itself could have any number of actions depending on if it was aoe etc, and the bash or block has its own actions.
combining all these things is the pinnacle of eso skilled gameplay. Our 3 actions per second might be closer to 30 or more server requests per second for one person depending on what is calculated serverside. Add in the sequential nature of how these requests are processed and you begin to understand the broken nature of eso combat. A whole bunch of requests might be processed before even one of yours is processed all within a second.
Sooner or later they will have to accept that their model is, was, and always will be too aggressive and will have to move to a more conservative model of 1 action (skill) per second +emergency DEFENSIVE actions ( dodge or block an attack). If they do not they will end up with all kinds of gutting like 3 second cooldowns on aoe etc.
Additionally, we have more calculations per action than most if not all MMORPGs out there. Think about CP, all the buffs and debuffs, and more. We have more of these types of additional calculations now than we had when the game launched. It has added tremendous strain to the servers.
Ofc, the statement in the OP is really just restating what someone else said. Yes, if ZOs moved server-side checks back to client as it was when the game launched it would reduce server-side lag. That is server 101 type stuff and we know that server lag increased after Zos moved from the trusted client to server-side checks.
However, all this is based on an assumption that a solid protection to block the use of CE could be implemented and that is where the idea falls apart. If it was truly that easy then that is what Zos would have done in the first place. It is really that simple.0 -
Pinja✭✭✭✭
Additionally, we have more calculations per action than most if not all MMORPGs out there. Think about CP, all the buffs and debuffs, and more. We have more of these types of additional calculations now than we had when the game launched. It has added tremendous strain to the servers.
Ofc, the statement in the OP is really just restating what someone else said. Yes, if ZOs moved server-side checks back to client as it was when the game launched it would reduce server-side lag. That is server 101 type stuff and we know that server lag increased after Zos moved from the trusted client to server-side checks.
However, all this is based on an assumption that a solid protection to block the use of CE could be implemented and that is where the idea falls apart. If it was truly that easy then that is what Zos would have done in the first place. It is really that simple.
I wrote the thread. Never said it would be easy. Though it may be the cheapest solution of three solutions I proposed. Following the full thread irons out vulnerabilities to the point they're as rare lightning strikes. Even Equifax got hacked, showing that if your a big enough a target even server side checks can fail. With a few GMs you wont have a problem. Any CE work around to the Anti-cheat proposed wouldn't be wildly dispersible or functional.
There's still probably more to this lag story outside of server anti cheats, but that is a big one.1