The true history of Cyrodiil's server problems and hope for the future.

Nomadic_Atmoran

ITT Small Scale PvPer trying to convince the forum goers that AoE caps ruined PvP.

Lowbei

Kungfu wrote: »

All due respect to your history of eso but I believe you're getting much negative feedback because you do kind of have a visible bias in it.
That said, a lot of the events listed definitely happened. I just cannot agree with this consensus that it's ONLY a matter of server resources. In IT, server resources are CHEAP when compared to developer paychecks. Had it been resources alone... ever, it wouldn't take but $2k - $10k to incrementally increase them.

However, I think you're missing two major points that *I believe* are the major contributors:
First and foremost is what others are trying to address as well: their attempts at anti-cheat.
Combine that with... routing traces through Akamai servers and you get what I believe (no real proof here) was a migration from on-prem to a cloud solution with "advanced security".

Current character desync issues at least anecdotally support this: cloud solutions and security checks en masse create delays in communication speed and high potential for packetloss. Either of which could easily cause location desync between client-server.

If they were stuck in a contract, it would explain why they cannot speak about these issues nor can they fix them. That's almost all theory but please know I've got the industry experience and have seen exactly the effects in other industries after a cloud migration.

There's the other, albeit more likely possibility that it's an issue with code that runs so deep through the game that a complete rewrite of major systems would be required in order to fix. And they probably no longer employ the genius(es) that wrote the original, nor do any of the current staff have the skill, education, or experience to take on that project.

I absolutely agree that a migration to cloud based security can and likely did contribute to a negative effect, but things have been getting increasingly worse and worse, to the point where pve groups are reporting major issues with even 12man content. At the current rate of performance degradation (ironically a month after the big “performance patch”) the game will be completely unplayable by the end of the year even for pvers, tho I hope not.

Lowbei

Nomadic_Atmoran wrote: »

ITT Small Scale PvPer trying to convince the forum goers that AoE caps ruined PvP.

The mass exodus of the tens of thousands of pvpers happened within a couple months of release, before the performance degradation began, and the reason they left were due to AoE caps resulting in ball zergs everywhere. That part really isnt up for debate.

Iskiab

Couple things you have wrong or didn’t mention in your MMO history:
- DAoC had small group size limits. This was what led to the 8 man groups fun pvp fights. After the population dropped that’s what kept things going
- Warhammer Online was meant to have a larger scale. The dominant groups were about the same size as ball groups are now in cyrodiil, and they were just as effective as ballgroups are now where 20 can take on 80 people. What killed that game was the endgame. After pushing and being the first group to down the emperor or whatever it was, people were furious it was a PvE event. They didn’t patch it to have pvp during the event until later. It also had player limits on keeps and too many people who bandwagon jumped on the winning faction so you’d be locked out. It got tiring to have to reroll on a losing server faction so you could pvp.
- PvP guilds jumped games a ton. Back then (after DAoC and around Warhammer) new MMOs were coming into the market all the time. People jumped like crazy. I was in one guild that always got beta early access to every new MMO and I ended up leaving them in RIFT solely because I couldn’t stand all the jumping, who wants to start a new game every 6 months? Point is the market’s different now so it would be worth the investment yea

Varana

See, and that is the thing: These problems are by no means exclusive to PvP. Unless you want to tell us that they have been siphoning resources away from PvE as well (to where?) because players whined about ... what, exactly?, there must be other things going on, as well.
It's just that these performance problems become visible first and most severely in PvP where you have larger groups doing lots of stuff very quickly. The only moments where you get similar load in PvE is when players camp some world boss during an event and take them down in a matter of seconds. But these occurrences are rare and short, and players kind of expect the game to buckle under the weight of 50 people all dropping their ultis at once. In Cyrodiil, that's what is supposed to happen all the time somewhere, for a prolonged time. Of course it's going to pass out sooner than PvE.
Cyrodiil never recovered from that "lighting patch" and some other attempts around that time, and has been playing catch-up ever since. Nothing in that patch had anything to do with zergs or AoEs in general. Also, that was months into the game's life, and not "a week before launch" or stuff like that.

Seriously, your timeline and argument in the OP is really weird. ZOS implemented an AoE cap, driving PvP players away, resulting in ... more lag? To which the solution would've been removing the AoE cap to bring all the PvP players back so more players would create less performance issues? Like, seriously - what?!?

Kungfu

Lowbei wrote: »

Kungfu wrote: »

All due respect to your history of eso but I believe you're getting much negative feedback because you do kind of have a visible bias in it.
That said, a lot of the events listed definitely happened. I just cannot agree with this consensus that it's ONLY a matter of server resources. In IT, server resources are CHEAP when compared to developer paychecks. Had it been resources alone... ever, it wouldn't take but $2k - $10k to incrementally increase them.

However, I think you're missing two major points that *I believe* are the major contributors:
First and foremost is what others are trying to address as well: their attempts at anti-cheat.
Combine that with... routing traces through Akamai servers and you get what I believe (no real proof here) was a migration from on-prem to a cloud solution with "advanced security".

Current character desync issues at least anecdotally support this: cloud solutions and security checks en masse create delays in communication speed and high potential for packetloss. Either of which could easily cause location desync between client-server.

If they were stuck in a contract, it would explain why they cannot speak about these issues nor can they fix them. That's almost all theory but please know I've got the industry experience and have seen exactly the effects in other industries after a cloud migration.

There's the other, albeit more likely possibility that it's an issue with code that runs so deep through the game that a complete rewrite of major systems would be required in order to fix. And they probably no longer employ the genius(es) that wrote the original, nor do any of the current staff have the skill, education, or experience to take on that project.

I absolutely agree that a migration to cloud based security can and likely did contribute to a negative effect, but things have been getting increasingly worse and worse, to the point where pve groups are reporting major issues with even 12man content. At the current rate of performance degradation (ironically a month after the big “performance patch”) the game will be completely unplayable by the end of the year even for pvers, tho I hope not.

So hard to believe they will let this continue. I am not a pve type but I will be happy to help in every way I can if they ask us for it.

My other fear is that they've completely mismanaged their income and rather than spend to fix what WE love, investors are saying just let it die. Though those people would have to be the dumbest, most short-sighted folks in the planet if they couldn't see how this game would explode if they could fix cyro alone and enhance pve to their creative hearts' desires.

Lowbei

Varana wrote: »

See, and that is the thing: These problems are by no means exclusive to PvP. Unless you want to tell us that they have been siphoning resources away from PvE as well (to where?) because players whined about ... what, exactly?, there must be other things going on, as well.
It's just that these performance problems become visible first and most severely in PvP where you have larger groups doing lots of stuff very quickly. The only moments where you get similar load in PvE is when players camp some world boss during an event and take them down in a matter of seconds. But these occurrences are rare and short, and players kind of expect the game to buckle under the weight of 50 people all dropping their ultis at once. In Cyrodiil, that's what is supposed to happen all the time somewhere, for a prolonged time. Of course it's going to pass out sooner than PvE.
Cyrodiil never recovered from that "lighting patch" and some other attempts around that time, and has been playing catch-up ever since. Nothing in that patch had anything to do with zergs or AoEs in general. Also, that was months into the game's life, and not "a week before launch" or stuff like that.

Seriously, your timeline and argument in the OP is really weird. ZOS implemented an AoE cap, driving PvP players away, resulting in ... more lag? To which the solution would've been removing the AoE cap to bring all the PvP players back so more players would create less performance issues? Like, seriously - what?!?

The servers at a datacenter cost a lot of money. I cannot tell you where the resources have been moved to, as they have likely simply been removed to save money. The result is that you feel it across the board in all areas. Certain ones of these servers that were dedicated to pvp (a gamemode they clearly dont care about anymore because it makes them less money) were shifted to the pve side years ago because the population dropped in Cyrodiil. The cause of that population drop, was the introduction of AoE caps, which had lead to ballzergs. The ballzergs themselves hurt the few remaining pvp focused servers even further, resulting in more players leaving, and the cycle continues until present day where 20 people cant fight in the same area.

At this point, anyone arguing for AoE caps clearly hasnt reviewed the history of the numerous previous mmos that failed because of them. The only people who benefit from AoE caps, are ballzergs, and everyone who has been to cyrodiil knows what happens to the server when ballzergs come into clip range.

The game would have had a massive pvp community for many years and likely still to this day due to no alternative mmo of ESOs caliber, if AoE caps hadnt been wedged into the launch notes, which resulted in a downward spiral that I detailed.

I will also point out that massive zergs still take place daily at Alik'r dolmens with very little lag, and all skills / bar swaps happening with no problem. It is a server resource allocation issue, not a coding issue. The remaining servers have been focused on the few locations left with large player numbers, which is mostly Alik'r, Craglorn, and a few other places, as if people in these zones start feeling it while they are leveling their characters, they will not stick around.

ykculnu

Lowbei wrote: »

I will also point out that massive zergs still take place daily at Alik'r dolmens with very little lag, and all skills / bar swaps happening with no problem. It is a server resource allocation issue, not a coding issue. The remaining servers have been focused on the few locations left with large player numbers, which is mostly Alik'r, Craglorn, and a few other places, as if people in these zones start feeling it while they are leveling their characters, they will not stick around.

Funny, I was actually thinking about this exact point while reading through this thread and some of the responses. May be anecdotal though because its usually 2-3 24 mans versus mobs. As apposed to 2-3 24 man raids fighting an additional 2-3 raids on the other side.

Lowbei

Kungfu wrote: »

Lowbei wrote: »

Kungfu wrote: »

All due respect to your history of eso but I believe you're getting much negative feedback because you do kind of have a visible bias in it.
That said, a lot of the events listed definitely happened. I just cannot agree with this consensus that it's ONLY a matter of server resources. In IT, server resources are CHEAP when compared to developer paychecks. Had it been resources alone... ever, it wouldn't take but $2k - $10k to incrementally increase them.

However, I think you're missing two major points that *I believe* are the major contributors:
First and foremost is what others are trying to address as well: their attempts at anti-cheat.
Combine that with... routing traces through Akamai servers and you get what I believe (no real proof here) was a migration from on-prem to a cloud solution with "advanced security".

Current character desync issues at least anecdotally support this: cloud solutions and security checks en masse create delays in communication speed and high potential for packetloss. Either of which could easily cause location desync between client-server.

If they were stuck in a contract, it would explain why they cannot speak about these issues nor can they fix them. That's almost all theory but please know I've got the industry experience and have seen exactly the effects in other industries after a cloud migration.

There's the other, albeit more likely possibility that it's an issue with code that runs so deep through the game that a complete rewrite of major systems would be required in order to fix. And they probably no longer employ the genius(es) that wrote the original, nor do any of the current staff have the skill, education, or experience to take on that project.

I absolutely agree that a migration to cloud based security can and likely did contribute to a negative effect, but things have been getting increasingly worse and worse, to the point where pve groups are reporting major issues with even 12man content. At the current rate of performance degradation (ironically a month after the big “performance patch”) the game will be completely unplayable by the end of the year even for pvers, tho I hope not.

So hard to believe they will let this continue. I am not a pve type but I will be happy to help in every way I can if they ask us for it.

My other fear is that they've completely mismanaged their income and rather than spend to fix what WE love, investors are saying just let it die. Though those people would have to be the dumbest, most short-sighted folks in the planet if they couldn't see how this game would explode if they could fix cyro alone and enhance pve to their creative hearts' desires.

You are absolutely correct. As others have stated on this forum within even within the last week, the game has clearly shifted focus from catering to endgame players, to only catering to newer players, as that is where the money is. This shift comes as a directive from the top, which can only mean that the game is making far less money now, and must focus on the players who bring in the most. This is not PvPers unfortunately, as we are content to fighting each other all day everyday and not buying from the crown store, whose prices are honestly some of the most ridiculous things I have seen on the market to this date. A shiny horse can cost as much as a DLC.

Lowbei

ykculnu wrote: »

Lowbei wrote: »

I will also point out that massive zergs still take place daily at Alik'r dolmens with very little lag, and all skills / bar swaps happening with no problem. It is a server resource allocation issue, not a coding issue. The remaining servers have been focused on the few locations left with large player numbers, which is mostly Alik'r, Craglorn, and a few other places, as if people in these zones start feeling it while they are leveling their characters, they will not stick around.

Funny, I was actually thinking about this exact point while reading through this thread and some of the responses. May be anecdotal though because its usually 2-3 24 mans versus mobs. As apposed to 2-3 24 man raids fighting an additional 2-3 raids on the other side.

True, but I will note that instead of 2-3 24mans fighting other 2-3 24mans, combat completely fails if only 20 total people fight now in Cyrodiil.

Wihuri

ryh wrote: »

No money from PvP = no point to care about PvP.

What MMO is now worth to play in similar fast paced action combat style with massive battles?

This is not true. To say that PvPers don't need resources for pots etc. which is made easier to gather with ESO+ or new interesting sets to test out in PvP which also requires you to purchase the DLC or get ESO+. The PvP players do bring in money. Any player has a chance to decide to use their real life money on this game, but if there is no player then there is no chance to do so. I refuse to believe that one ESO+ subscription would not cover the upkeep costs. Sure, maybe the average PvPer would spend less but it's still contributing to the fact that the MMO is alive and going on. If there's no players then there is no MMO.

Kungfu

I and many others I know pay for ESO+ mostly because we all agree that recurring monthly fees are the most sustainable business model for the industry.

Well... that and the craft bag.

Edit: In the spirit of full disclosure: just yesterday, I canceled my usual ANNUAL subscription renewal due to the cryo issues.

Nomadic_Atmoran

Lowbei wrote: »

Nomadic_Atmoran wrote: »

ITT Small Scale PvPer trying to convince the forum goers that AoE caps ruined PvP.

The mass exodus of the tens of thousands of pvpers happened within a couple months of release, before the performance degradation began, and the reason they left were due to AoE caps resulting in ball zergs everywhere. That part really isnt up for debate.

You have sources for how many PvPers left? I bet you dont. So yeah, its definitely debatable.

NinchiTV

I really really hope they read this.

Bryath

'Competitive' pvp'ers - "Hey, let's put together a small group consisting of elite players and use voice comms so we can destroy groups of 20-30 average players"

Average pvp-ers - "It's no fun losing to these guys every time, lets combine forces so we stand a chance"

'Competitive' pvp'ers - "Wtf, why are you guys zerging?"

I can't really blame them, its natural to want to put together a winning group, but it ruins the small-scale type game for average players. Railing against people who have to zerg to have a chance is just silly.

VaranisArano

Nomadic_Atmoran wrote: »

ITT Small Scale PvPer trying to convince the forum goers that AoE caps ruined PvP.

The funny thing about it is that - AOE caps or no caps- the organized raids and ball groups always dominated Cyrodiil. Organized small scale groups have always manhandled more than their number of PUGs.

idk

VaranisArano wrote: »

Nomadic_Atmoran wrote: »

ITT Small Scale PvPer trying to convince the forum goers that AoE caps ruined PvP.

The funny thing about it is that - AOE caps or no caps- the organized raids and ball groups always dominated Cyrodiil. Organized small scale groups have always manhandled more than their number of PUGs.

This is true on both accounts. Large zergs started before the game was launched. I remember seeing them during the stress test betas. They serve the group that is not in a guild that runs PvP but still wants to run with a group.

Early on I started running with a small group shortly after launch and we easily handled large zergs because we were well organized and had a good leader. It was fun traping the large zergs and clearing the field of them. So yes on the second account as well as a well organized small group can handle parties much larger than theirs though we do sometimes end up on the losing side of the battle.

Nomadic_Atmoran

idk wrote: »

VaranisArano wrote: »

Nomadic_Atmoran wrote: »

ITT Small Scale PvPer trying to convince the forum goers that AoE caps ruined PvP.

The funny thing about it is that - AOE caps or no caps- the organized raids and ball groups always dominated Cyrodiil. Organized small scale groups have always manhandled more than their number of PUGs.

This is true on both accounts. Large zergs started before the game was launched. I remember seeing them during the stress test betas. They serve the group that is not in a guild that runs PvP but still wants to run with a group.

Early on I started running with a small group shortly after launch and we easily handled large zergs because we were well organized and had a good leader. It was fun traping the large zergs and clearing the field of them. So yes on the second account as well as a well organized small group can handle parties much larger than theirs though we do sometimes end up on the losing side of the battle.

Which is how it should be. This whole thread is digging up a long buried horse and further beating whats left of its bones. All under the guise that somehow AoE caps was the true fall of PvP rather than the litany of performance issues and poorly thought out changes to combat over the last 6 years.

Vapirko

Im amazed people are still petitioning for Cyrodiil to be better. Or think there is some chance.

Lowbei

Vapirko wrote: »

Im amazed people are still petitioning for Cyrodiil to be better. Or think there is some chance.

Yes, it is unfortunate that the entire pvp community has given up on this game, but not as unfortunate as the company giving up on its own product due to its own bad decisions, and shifting almost all server resources away from it to the point where even pve raids of 12 people cant do the content without massive desyncs.

In 20+ years playing mmos I hadnt seen anything like that until now.

Grianasteri

Very interesting original post.

Last night I entered Cyrodiil, headed to one of the two battles taking place on the map, engaged the enemy, immediately found that basically none of my skills were firing at all, or only after several seconds... left Cyrodiil.

Personally I do get pleny of low laag time in Cyrodiil, but this is equalled by the poor laag and like last night, totally and utterly unplayable laag.

Its a crying shame.

Neophyte

Grianasteri wrote: »

Very interesting original post.

Last night I entered Cyrodiil, headed to one of the two battles taking place on the map, engaged the enemy, immediately found that basically none of my skills were firing at all, or only after several seconds... left Cyrodiil.

Personally I do get pleny of low laag time in Cyrodiil, but this is equalled by the poor laag and like last night, totally and utterly unplayable laag.

Its a crying shame.

Yeah I see where you are coming from sometimes it’s fine . Especially outside the peak times. I have no problem with large scale pvp as intended but just make it work . Sometimes I wonder if they if they didn’t expect people to stack up the way the ball groups do instead of a giant free for all

daemonios

yodased wrote: »

The lack of any sort of real consequence for exploitative behavior and the allowance of straight up cheating put the playerbase into a perception of what this game is.

Ask 100 people who pvp about cheating,exploits,overperforming sets or skills and see what those answers are.

The devs lackadaisical approach to curtailing cheating and exploits in the first 2 years of the pvp game set the tone and its put us where we are now.

People cheated hard, no one did crap about it so it continued and everyone either left, joined or dealt with the fact that there are players that cant be killed" player that had unlimited resources, players who increase lag and latency by being near them, players who move faster in combat than fully sprint mount, players who can fly etc etc.

This got to a boiling point where we had a social media push with flying players dropping unlimited meteors.

This publicity caused them to overcorrect and add an exponential increase of server side variables that were not in the initial server design specifications.

Over taxing the server with all these calculations is what started the problems and every patch afterword has been trying to balance anti cheat and performance.

Tl;Dr People cheated ZOS over corrected causing server hiccups.

[Snip]

The only instance where, to my knowledge, people were banned in any kind of meaningful numbers from cheating was after the meteor spam incident that was posted on social media. And even then, unless I'm mistaken, several players involved were later unbanned.

It's a shame, honestly. I don't enjoy PvP for the sake of just PvP, but the alliance war has something for me. I enjoyed fighting for the campaign score over the period of a month. I left in disgust due to one case of cheating and exploits after another: EU guilds stockpiling forward camps in the hours between the NA server coming up with the surprise removal of forward camps and EU going down for the same maintenance, giving them exclusive use for weeks because ZOS for some reason didn't deem it relevant to forbid their use, even knowing some guilds had exploited the advantage of knowing a few hours in advance that camps were being removed; one faction consistently abusing low population bonus to flip a campaign in the last 3 days, flooding it and taking over the entire map to get otherwise impossible faction ticks; and the obvious, though hard to prove, cheat engine users with unlimited resources and impossible movement, to name just a few. In the end, it's ZOS' game, and if they feel good with allowing these sorts of behaviours, then fine. But I have zero interest in playing in a competitive setting if there is no concept of fairness to the competition.

[Edited for bashing]

Varana

Lowbei wrote: »

I cannot tell you where the resources have been moved to, as they have likely simply been removed to save money. The result is that you feel it across the board in all areas. Certain ones of these servers that were dedicated to pvp (a gamemode they clearly dont care about anymore because it makes them less money) were shifted to the pve side years ago because the population dropped in Cyrodiil.

See, and now we're at the place where this thread should've been the whole time: Firmly in the realm of pure speculation and wild guessing.

stefan.gustavsonb16_ESO

Nomadic_Atmoran wrote: »

You have sources for how many PvPers left? I bet you dont. So yeah, its definitely debatable.

I would consider first hand experience to be a valid source, even though it's notoriously hard to verify. For what it's worth, I can say that I was here at launch, too, and I watched the Cyrodiil population decline strongly and rapidly in the manner described by the OP, with a timing that would be consistent with their line of reasoning. The game went from a large amount of well populated campaigns with a high population cap to just a few active campaigns with a population cap that is only a fraction of the original.

We can argue about the reasons for it, and we have no hard numbers on exactly how big it was, but the rapid and massive decline in PvP population was real. Compared to 2014-2015, only a handful of people are playing in Cyrodiil now.

daemonios

stefan.gustavsonb16_ESO wrote: »

Nomadic_Atmoran wrote: »

You have sources for how many PvPers left? I bet you dont. So yeah, its definitely debatable.

I would consider first hand experience to be a valid source, even though it's notoriously hard to verify. For what it's worth, I can say that I was here at launch, too, and I watched the Cyrodiil population decline strongly and rapidly in the manner described by the OP, with a timing that would be consistent with their line of reasoning. The game went from a large amount of well populated campaigns with a high population cap to just a few active campaigns with a population cap that is only a fraction of the original.

We can argue about the reasons for it, and we have no hard numbers on exactly how big it was, but the rapid and massive decline in PvP population was real. Compared to 2014-2015, only a handful of people are playing in Cyrodiil now.

I won't swear by this as I wasn't heavily into Cyrodiil in 2014, but to my recollection the big exodus was after the 1.2 update, dubbed the "lighting patch", in July 2014. This brought very large, very noticeable degradation to performance, notably in Cyrodiil (the only kind of PvP available then). I don't remember anyone singling out ability changes per se as reasons for leaving back then.

SpiderCultist

After reading your post I'm hesitating whether you are a white knight or an insider. I don't care anyway, the current state of the game is what it is since U25: UN-PLA-YA-BLE. And that's all that matters to me, I can't barely play the game and we are a month away since U25.

Nordic__Knights

daemonios wrote: »

stefan.gustavsonb16_ESO wrote: »

Nomadic_Atmoran wrote: »

You have sources for how many PvPers left? I bet you dont. So yeah, its definitely debatable.

I would consider first hand experience to be a valid source, even though it's notoriously hard to verify. For what it's worth, I can say that I was here at launch, too, and I watched the Cyrodiil population decline strongly and rapidly in the manner described by the OP, with a timing that would be consistent with their line of reasoning. The game went from a large amount of well populated campaigns with a high population cap to just a few active campaigns with a population cap that is only a fraction of the original.

We can argue about the reasons for it, and we have no hard numbers on exactly how big it was, but the rapid and massive decline in PvP population was real. Compared to 2014-2015, only a handful of people are playing in Cyrodiil now.

I won't swear by this as I wasn't heavily into Cyrodiil in 2014, but to my recollection the big exodus was after the 1.2 update, dubbed the "lighting patch", in July 2014. This brought very large, very noticeable degradation to performance, notably in Cyrodiil (the only kind of PvP available then). I don't remember anyone singling out ability changes per se as reasons for leaving back then.

Only ZOS can give us the numbers but to do that they couldn't use them numbers for marketing so have them tell it 13million play daily 😆 😆

ItsMeToo

Lowbei wrote: »

Vapirko wrote: »

Im amazed people are still petitioning for Cyrodiil to be better. Or think there is some chance.

Yes, it is unfortunate that the entire pvp community has given up on this game, but not as unfortunate as the company giving up on its own product due to its own bad decisions, and shifting almost all server resources away from it to the point where even pve raids of 12 people cant do the content without massive desyncs.

In 20+ years playing mmos I hadnt seen anything like that until now.

Yet another incorrect claim without facts or proof. I'm still here.

You need to speak only for yourself and not for others.

PrimusNephilim

Read the whole thread....my take away from all of this, the game play is not going to get any better and that's a sad thing because I really like this game.