Maintenance for the week of March 30:
• [COMPLETE] ESO Store and Account System for maintenance – April 1, 1:00PM EDT (17:00 UTC) - 5:00PM EDT (21:00 UTC)

The reason why server performance is bad and a way to adress it

  • Delparis
    Delparis
    ✭✭✭✭✭
    The problem with games performance issues likes solely with their use of the TCP protocols. TCP has a ridiculous amount of overhead, and every single packet or group of packets had to be ACK. To make matters worse TCP has built in congestion controls that assume packets not ACK or revived with the TTL value are assumed to be loss due to network congestion, so it will dial
    Back throughput and attempt to resend.

    When you got two zergs in pvp all mashing skills and sending the server packets it must ACK and process with TCP what do you think is going to happen?

    There is reason no other triple A studio since the early 2000’s has used TCP for any real time or FPS combat game.

    ZOS is t going to re write their net ode to properly use a UDP implementation so no amount of game changes short of straight cooldowns is going to do much of anything to fix the issues, 99% of the problem lies with the idiotic choice to use TCP which scales poorly and has a huge amount of overhead. TCP wasn’t designed for this, there is a reason other real time games started using UDP instead with their own custom stack on top of it and client prediction.

    Good luck

    ok i'm not an IT former, i only studied maths.

    from what i understand this is related to single vs multi packages sent to server issue.

    Can this issue be fixed ?
  • RinaldoGandolphi
    RinaldoGandolphi
    ✭✭✭✭✭
    ✭✭✭✭
    Delparis wrote: »

    ok i'm not an IT former, i only studied maths.

    from what i understand this is related to single vs multi packages sent to server issue.

    Can this issue be fixed ?

    Not exactly. To put it very basic its like this

    https://www.vpnmentor.com/blog/tcp-vs-udp/

    Simply put UDP is faster because it has no overhead. TCP requires all packets delivered, UDP doesn't. a packet you missed 5 seconds ago is already useless so just discard and receive the next with all the updated information. It also means the server doesn't have to use CPU cycles ACK every packet, error correction, etc. Using TCP for this game was a huge design blunder on their part, and I don't think they will re-write it and move away from TCP to UDP although doing so would probably solve most of their lag issues.


    Rinaldo Gandolphi-Breton Sorcerer Daggerfall Covenant
    Juste Gandolphi Dark Elf Templar Daggerfall Covenant
    Richter Gandolphi - Dark Elf Dragonknight Daggerfall Covenant
    Mathias Gandolphi - Breton Nightblade Daggerfall Covenant
    RinaldoGandolphi - High Elf Sorcerer Aldmeri Dominion
    Officer Fire and Ice
    Co-GM - MVP



    Sorcerer's - The ONLY class in the game that is punished for using its class defining skill (Bolt Escape)

    "Here in his shrine, that they have forgotten. Here do we toil, that we might remember. By night we reclaim, what by day was stolen. Far from ourselves, he grows ever near to us. Our eyes once were blinded, now through him do we see. Our hands once were idle, now through them does he speak. And when the world shall listen, and when the world shall see, and when the world remembers, that world will cease to be. - Miraak

  • SweepsAllClowns
    SweepsAllClowns
    ✭✭✭✭
    Lag has never been any huge problem for Zenimax, players own the rights to enjoy this amazing ever improving online feature and it's just getting even better all the time!
  • Red_Feather
    Red_Feather
    ✭✭✭✭✭
    ✭✭
    hmmm I never really thought about all that. I hope it doesn't mean the game loses those things. I really like them.
  • Casowen
    Casowen
    ✭✭✭
    How about because their servers are from 10 years ago? All you have to do is look at bdo ff15 and wonder.
  • Gandrhulf_Harbard
    Gandrhulf_Harbard
    ✭✭✭✭✭
    ✭✭
    Trust me its addons, I never use them and I only ever experience bad server performance in Cyrodiil, and that is pretty rare for me.

    Be safe

    I've been having desync with looting corpses since I returned a couple of weeks ago.

    A lot of posts, including some from ZOS say "hey, it's the add ons".

    So I uninstalled all my add-ons, and played for a day.

    Desync still happened just as much, occasional "lag spikes" still occurred just as often.

    But the game-play experience, because the native UI is so appallingly bad, was excruciatingly bad I gave up and went back to my add-ons.

    Having played with and without add-ons on the same day, on the same server, with the same internet connection I feel fairly certain in saying the effect of add-ons on game performance is, effectively, negligible.

    The key factor is almost certainly on ZOS' side of the fence.


    All The Best
    Those memories come back to haunt me, they haunt me like a curse.
    Is a dream a lie if it don't come true, or is it something worse.
  • Merlight
    Merlight
    ✭✭✭✭✭
    Using TCP for this game was a huge design blunder on their part, and I don't think they will re-write it and move away from TCP to UDP although doing so would probably solve most of their lag issues.

    While I agree that UDP might've been a better choice, it's not a magic silver bullet. Point one: there is still the possibility that ZOS' implementation would be so poor that we'd end up with just a tad less laggy mess.

    Point two: the issue is not as simple as "any packet older than 5s is worthless, just roll with what arrives in time". Imagine you hit weapon swap button. When does the client UI show the back bar? When client sends the packet, or when it receives confirmation from server that you're allowed to weapon swap? If client shows back bar when sending packet, and that packet never arrives at server, the client thinks you're on back bar, while server thinks you're on front bar. On the other hand, if client waits for confirmation from server before swapping, what if that confirmation never arrives, even though the server received the request and sent confirmation? They would have to implement a consistency layer on top of UDP, and that'd be a lot of room for "killing it".

    Point three: you mentioned early 2000's... some games from that era serve as proof that large-scale real-time combat over TCP is definitely possible. I might be repeating myself as I'm pretty sure I've talked about Lineage 2 on these forums in the past. Perhaps it doesn't fit the "triple A" category (which in my opinion says nothing about quality, only budget), but that's irrelevant.

    tl;dr: you can choose TCP, if you know the implications, and if your combat design doesn't contradict those.

    Lineage 2 released in 2003/2004, and until around 2007 didn't have any instanced content. The whole game world was one persistent huge map with no loading screens; there were some dungeon-like areas, but these were actually just confined underground or underwater rooms still on that map, where you teleported instantly. Obviously they wouldn't be able to support unlimited number of players on one map, so they had many independent "shards". The shard server software consisted of several services running on dedicated machines (database, logging, etc); login server pointed the client to your selected shard's game server, that was your connection to the game world. This game server -- and I mean one dedicated machine -- routinely handled 3k concurrent players and tens of thousands of NPCs. Most of the time these were running around in small groups, killing mobs or other players, but there were a few towns and events where many players gathered. The events that attracted hundreds of players to duke it out in a small area were world bosses whose respawn timers counted in days, and castle sieges that occurred every other weekend (i.e. like Cyrodiil, but only 4 hours every 2 weeks).

    Granted, that game had a much simpler combat system than ESO has, but it also had ten times the population on a single server, which had no trouble running AI scripts for antelopes (cf. deer) and other creatures being slaughtered by players in various areas all over the map, while hundreds of players were spamming buffs, nukes and heals around castles. And it ran smooth over TCP in an era of much lower bandwidths. When some crafty Russians got hold of the server software, they figured out how to let 15k people in, and it didn't crumble. ESO under such stress would float out of Craglorn...
    EU ‣ Wabbajack nostalgic ‣ Blackwater Blade defender ‣ Kyne wanderer
    The offspring of the root of all evil in ESO by DeanTheCat
    Why ESO needs a monthly subscription
    When an MMO is designed around a revenue model rather than around fun, it doesn’t have a long-term future.Richard A. Bartle
    Their idea of transparent, at least when it comes to communication, bears a striking resemblance to a block of coal.lordrichter
    ... in the balance of power between the accountants and marketing types against the artists, developers and those who generally want to build and run a good game then that balance needs to always be in favour of the latter - because the former will drag the game into the ground for every last bean they can squeeze out of it.Santie Claws
  • Jhalin
    Jhalin
    ✭✭✭✭✭
    ✭✭✭✭
    Trust me its addons, I never use them and I only ever experience bad server performance in Cyrodiil, and that is pretty rare for me.

    Be safe

    I've been having desync with looting corpses since I returned a couple of weeks ago.

    A lot of posts, including some from ZOS say "hey, it's the add ons".

    So I uninstalled all my add-ons, and played for a day.

    Desync still happened just as much, occasional "lag spikes" still occurred just as often.

    But the game-play experience, because the native UI is so appallingly bad, was excruciatingly bad I gave up and went back to my add-ons.

    Having played with and without add-ons on the same day, on the same server, with the same internet connection I feel fairly certain in saying the effect of add-ons on game performance is, effectively, negligible.

    The key factor is almost certainly on ZOS' side of the fence.


    All The Best

    Of course it’s all on ZOS, they know they have no intentions of improving their game
  • Conduit0
    Conduit0
    ✭✭✭✭✭
    The reason server performance is so terrible is because they have utterly failed to keep up with the growth of the game's population. They are either unwilling or unable to spend the money necessary to upgrade the servers to what they actually need to be to accommodate the current player base and rate of growth. Any other reason given is just an excuse.
  • Delparis
    Delparis
    ✭✭✭✭✭
    Conduit0 wrote: »
    The reason server performance is so terrible is because they have utterly failed to keep up with the growth of the game's population. They are either unwilling or unable to spend the money necessary to upgrade the servers to what they actually need to be to accommodate the current player base and rate of growth. Any other reason given is just an excuse.

    that's the official answer, but the game kept being worst and worst since 1.6 so long ago before any newcomers join
  • aaisoaho
    aaisoaho
    ✭✭✭✭✭
    Delparis wrote: »
    imo the server got this bad performance because all the calculation responsable for the lag and frame rate drop are made server side. Those calculations are liked to:
    - many skills have multiple effects (dmg, buffs, debuffs) which make the calculation more complicated and time consuming
    These are not an issue when we are talking about Big O notation. The additional + or - in a damage calculation do not affect the run time in a meaningful fashion. F(x) = x and F(x) = 3*x+5 are both classified as O(n).
    Delparis wrote: »
    - skills don't have a long enough global cooldown to prevent too much ping a request to the server
    - animation cancelling with LA weaving is increasing the stress on servers
    Well, stripping these away would make the combat more clunky. Performing actions on your character should feel fluid and entertaining. What difference would it make if you had a huge lag where you can perform actions every 2 seconds versus a scenario where you had a minimal lag and you could perform actions every 2 seconds?
    Delparis wrote: »
    - there is too many buffs and debuffs available and they are splitted into major and minor buffs
    Yet again, these do not affect the big O notation.
    Delparis wrote: »
    - objects, containers and npc are dynamic so the server has to tell the client were they are (npc) and what they are and what they content (objects and container)

    to name a few..
    Yes, dynamic environment means the collision detection algorithm has a bigger Big O notation. With static environment you could use sweep and prune with a O(n), but with dynamic environment, the notation changes on the worst case scenario to a O(n^2). But! A static environment would be quite boring and would feel like it is not 'alive'.
    Delparis wrote: »
    i really think Zenimax devs should have a look on those points and improve them instead of putting more ressources in hardware.
    a way to adress those issues is as follow:
    1. Make skills have only 1 dmg, heal or buff effect
    2. Increase the global cooldown to skills to 1 sec and make LA share the same global cooldown (clunky combat maybe but at least you don't play on a slideshow)
    3. remove the animation cancelling on LA as they share the same gcd with skills
    4. standerdize buffs and debuffs, no more minor and major one (less factors for the calculation), and make some of those buffs and debuffs available through foods, drinks and potions
    5. npc position should be known from the client rather than the server so less useless informations transiting, the same thing for objects
    6. container should have an RNG tool when interacting with it, so the server won't have to comunicate to the client what they contain. also since rng players will be able to interact with the same container and get different loot depending on their rng. each container have 1 loot.

    hope this can help you Zenimax devs
    As said, making the skills have only 1 effect would not affect the calculation's big o notation in a meaningful way, rendering the 'fix' useless to combat lag and ping. The same thing with buff/debuff proposition. Switching the location data from server to client would still keep the collision detection algorithm's Big O notation as O(n^2). And worst of all, every action performed on client side is vulnerable to hacking and cheating - and cheat detection is usually quite demanding process when it comes to run times!

    I think the real reason why we have bad performance is because of the amount of players. It is a good thing for the game to have a growing playerbase, but it also means the servers need to be kept up to date and upgraded.

    We do not know if the programming is done in an optimized way or not, because we cannot access the source code to analyse it.
Sign In or Register to comment.