ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

manukartofanu

twev wrote: »

manukartofanu wrote: »

@twev May I ask, who are you telling this to? Those who work in this industry know without any statements from anyone that information about what work is being done cannot help any wrongdoers. Yes, not everything can be said. But that's no explanation for complete silence or standard repetitive answers.

P.S. If what you're writing about actually worked that way, we wouldn't be getting any announcements or patch notes.

Simple.
The explanation for the stuff that is silenced is that that is what the mother ship feels is best for their own ends.

Who I'm talking 'to' is the people who have a passing acquaintance with tech, networking and general business without being inside the machinations of big business intrigue and subterfuge.

Lots of things that non-tech people would think are just normal info are guarded as proprietary information, and many things that are in the public domain of information would get a tech worker or dev fired for breaching a non-disclosure clause in their contract, by even just commenting on the record about info that is being discussed in public.

I was just commenting about the constant requests for 'updates', and 'what are you working on?, 'how many people are working on the problem?', What have you tried so far?' and other 'innocent' questions that keep being repeated, with increasing frustration.

Some companies answer questions, other companies ignore many such questions that they feel answering is not in their fiscal interest to address.
Many people here seem to think that the company can be badgered into responding, when, clearly they have little interest in posting such information.
If they wanted us to know, they'd have told us details long ago.
If they were able to fix the issues for what they consider to be a worthwhile expenditure, they would have by now.
They either can't, or won't, do certain things, and they don't feel obligated to inform us of which those things are.
Some people can't or just won't, accept that.

It's like asking your kid 'Who broke the vase??', and they start telling you about their day in school.

Posting patch notes is completely outside the scope of answering many/most of all the tech questions and requests that get posted here.

Players here who work in tech already know what most of those things that can be discussed under company non-disclosure agreements are. A lot of players not in tech don't know how many things aren't for public consumption.

The complete silence you referred to in the penultimate line of your post isn't company incompetence at communication, rather, it's what they feel benefits them the most.
We can ask Kevin 12 times/day for details, and most of the questions will go unanswered because he's not permitted to comment, or he's not given details pertaining to the questions asked.


I was just addressing some of the causes behind the actions that result in forum user's frustrations in a 61 page thread.
In no way am I implying the frustrations are without merit.

Oh yeah, because only tech companies have NDAs, no one outside the tech world even knows about them, and we need to explain to everyone how they work!

You're right that employees CAN be restricted in communication by internal policies and company limitations. But there are also many other POSSIBLE reasons for things happening the way they are. In addition to that, you do realize that policies and restrictions, including subcontract agreements, are written by other employees of the company, right? I don't recall seeing any personal accusations against any specific employee here. But what I do see is that you're speculating about things you don't know and can't possibly know. No one here, except the actual employees of ZOS, knows what's in their contracts. And you're no exception.

As for my previous reply, I wrote it because it seemed like you were justifying the silence or lack of details by implying that bad people would use those details. And that's simply not the case. A lot of details can be revealed without any benefit to bad people.

Yeah, I love how you present your personal opinion as something supposedly known by everyone in the tech industry. From the creators of 'scientists say' and 'studies show.'

kiwi_tea

I signed out of the game, properly, over 10 hours ago, long before maintenance started. I tried to sign in for the first time since just now. Error 307 instantly. How can I get Error 307 - Booted from Server - when I haven't been on the server?

This game - for its price and the size of its player base - clearly has such dysfunctional, stingy online infrastructure it would make Nintendo blush.

DinoZavr

Not discussing any policies, non-disclosure agreements and other administrative stuff:
Computing system has 4 core components: CPUs, RAM, disks, NICs (I won’t count GPU – on servers these are mainly AI devices)
These are well-known means to measure:
- utilization (and, hence, indirectly to estimate queues, also to find a bottleneck)
- queues
- for NICs there is also the very comprehensive statistics of packets sent/received/lost
Connections limits are also well known for the most of popular Operating Systems and services

From the technical standpoint: it is possible to monitor all the essential metrics (and trafic with Netflow) with SNMP or other means – in a real-time.
Nagios/Zabbix software can help (and Zabbix has good certified security), so in the absence of comprehensive monitoring it is possible to deploy a good monitoring (also there are vendor specific products like Cisco Works, HP OpenView, Dell OpenManage – which are a fraction of cost of the said vendors hardware).

I mean: technically it is possible to pinpoint bottlenecks like:
- fault network card cold solder joint sporadically causing packet loss spikes
- top or rack switches overheating and slowing down or rebooting
- number of connections exceeding OS / software capabilities
- 100% CPU utilization causing server to slow down
- too slow disk subsystem (again 100% utilization, device failures in logs)
even the faulty RAM is possible to detect, as servers memory has the extra parity bits

The issues very likely might be software, not hardware, as the game engine is hardly renovated quickly, or at all. When the game engine limitations reached (with introducing of house tours we have more disconnects with Error 334, this kind of issues can happen if servers do not have enough stacks programmed to keep all incoming “beam me” requests, and some are lost and disconnected as the engine array to store “move character A on server B to instance C on node D” could be insufficient.)
Spaghetti code might the cause of the issues, but not theirs solution.

We don’t know. I would prefer to hope Zenimax would find budget to:
- upgrade hardware to keep normal utilization OF EACH SYSTEM and COMPONENT below 50% and peak utilization below 75..80%
- monitor faults and performance in real-time
- make better SLA with theirs ISPs and CSPs with strict performance demands (akamai has issues)
- allocate man-hours to check the code, and debug discovered issues, maybe using a kind of load simulator, or luring players to PTS for massive testings.

It takes money and man-hours of skilled professionals, which I would like to think is not a problem for a company of such scale.

Though it would be nice if we receive at least confirmation of player-reported problems, and ETA,
as the most common situation is Zeni remaining silent when the forum is exploding with whines, and it is (in my humble opinion) not the communication the faithful customers do deserve.

baconaura

twev wrote: »

thinkaboutit wrote: »

twev wrote: »

manukartofanu wrote: »

@twev May I ask, who are you telling this to? Those who work in this industry know without any statements from anyone that information about what work is being done cannot help any wrongdoers. Yes, not everything can be said. But that's no explanation for complete silence or standard repetitive answers.

P.S. If what you're writing about actually worked that way, we wouldn't be getting any announcements or patch notes.

Simple.
The explanation for the stuff that is silenced is that that is what the mother ship feels is best for their own ends.

Who I'm talking 'to' is the people who have a passing acquaintance with tech, networking and general business without being inside the machinations of big business intrigue and subterfuge.

Lots of things that non-tech people would think are just normal info are guarded as proprietary information, and many things that are in the public domain of information would get a tech worker or dev fired for breaching a non-disclosure clause in their contract, by even just commenting on the record about info that is being discussed in public.

I was just commenting about the constant requests for 'updates', and 'what are you working on?, 'how many people are working on the problem?', What have you tried so far?' and other 'innocent' questions that keep being repeated, with increasing frustration.

Some companies answer questions, other companies ignore many such questions that they feel answering is not in their fiscal interest to address.
Many people here seem to think that the company can be badgered into responding, when, clearly they have little interest in posting such information.
If they wanted us to know, they'd have told us details long ago.
If they were able to fix the issues for what they consider to be a worthwhile expenditure, they would have by now.
They either can't, or won't, do certain things, and they don't feel obligated to inform us of which those things are.
Some people can't or just won't, accept that.

It's like asking your kid 'Who broke the vase??', and they start telling you about their day in school.

Posting patch notes is completely outside the scope of answering many/most of all the tech questions and requests that get posted here.

Players here who work in tech already know what most of those things that can be discussed under company non-disclosure agreements are. A lot of players not in tech don't know how many things aren't for public consumption.

The complete silence you referred to in the penultimate line of your post isn't company incompetence at communication, rather, it's what they feel benefits them the most.
We can ask Kevin 12 times/day for details, and most of the questions will go unanswered because he's not permitted to comment, or he's not given details pertaining to the questions asked.


I was just addressing some of the causes behind the actions that result in forum user's frustrations in a 61 page thread.
In no way am I implying the frustrations are without merit.

What is the point of your post? Nice job being condescending with your little "passing acquaintance" comment there.

Here's the bottom line, the issue has been ongoing for months. What would you have us do? Sit in silence?

By all means, no, don't sit in silence.
One of the best things about an open forum is players sharing the stuff in this thread, so we aren't feeling 'alone'.
Just look at how many players post here that they are relieved to know it's not just them, and the diversity of locations and connections/equipment/platforms lets us know that it's not our fault when half of a group all freeze or disconnect at the same time.

Some of the problems probably stem from jamming all the new stuff, locations, flashy effects into an old old engine that wasn't designed for it.
Some of it probably stems from the number of DLCs and locations the engine is trying to keep knitted together.
Some of it is the servers keeping track of the population density in game, especially in a map, or specific area.
Some of it is the increasing number of items, collectibles, in inventory, and banks as more players log in on the servers.
[Notice how logging into a guild bank at random times shows the 'I'm thinking about it' icon forEVAR, or until you close the attempt and try again, maybe a few times.]
Some of it is probably even due to a dodgy circuit board or two/many in the servers thats doing the countdown to eventual failure. (we had that one a few months before the server rebuild, where they found a few boards that were going bad.) Some IC boards just drift further out of spec over time.
Lets not discuss the issues with Akamai. That's it's own black hole of info.

But to the point, this thread alone has allowed us to compare notes and see just how pervasive the issues are for many of us, outside of our own sphere's, regardless of the number of blessed players who chime in with 'Working for me, never had a problem'...

And the forum population is small compared to the number of game licenses.
I'll bet the majority of players with issues don't even know about the forum, deal with issues for a while, and then just delete the realm from their drives and move on to other games without ever being heard.

We're lucky that as many of the players with issues even came here and voiced their concerns, hoping to keep the game not only alive, but make the game a better experience for everyone else.
About all we can do is point out the issues, and try to convince the Mother ship that we're here, we care, and we're worth being heard, so the game doesn't circle the drain before problems get fixed.

And I'm about as cynical as they come.


[edit to add:]
I'm sorry if you thought I was being condescending.
It was not my intent.
I was merely pointing out that of the roughly 20 million licensed copies of the game in circulation - you can count on a rather large population of players who have never worked in tech, for some of them this is even their first game on their first computer.
There are a lot of people here who have never assembled a computer, configured a network, or can diagnose which addons might be causing a problem with the game.
Many people here have looked at error msgs and tried to figure out what they were referring to, and can't understand why they didn't have a particular problem last week, or last month, but now it's a problem that happens several times a day.

I got out of tech years ago when I realized that there were too many times when management made it harder to fix a problem than it could/should have been.

And as an aside, I've been in on many guild conversations, and group conversations over the years in this game alone where it was revealed that half the guild or group was made up of retired people, people who had worked the trades for their careers, and half the tech they learned was from gaming forums like this one.

it's not condescending to recognize those players as people who haven't had the benefit of inside experience hands-on with tech companies while they're just trying to figure out why their game doesn't work right.

But I appreciate you pointing out the opportunity to clarify the wording in my original responses.

I think ZOS in general just needs to be more proactive and open with communication about what they are investigating, and provide some updates on the investigation. Just using a general example, but ffxiv posts status and updates on when things are causing issues, and when they get resolved on https://na.finalfantasyxiv.com/lodestone/news/category/4. They don't provide all the nitty gritty detail, but at least there is some visibility that yes, they are under DDOS attacks for that day, and when the DDOS has been mitigated it's communicated. Or if there is an internal server issue and certain things are affected, this is communicated when it is resolved.

Having an accurate and useful status page that acknowledges an issue goes a long way to respecting customers. Like in NYC, back around 2010, we didn't have the electronic signage that told us when the next subway would arrive. So you could be waiting 1 min, or like 20+ mins. And this could be really frustrating late night if you just want to get home, and have to decide to continue waiting longer after 20mins, because you waited so long already, or just walk or take a cab home. After they introduced the electronic signage that updates the customer on when the next train arrives, they can decide on alternative routes if there are issues and things are delayed. Like if there is a status page for ZOS, and they are under DDOS attack, and performance is degraded, raid leads can call off the night earlier, and give that time back to players to do something else, instead of having to suffer through multiple disconnects and freezing lag.

For more serious issues, they really should provide post mortem or communicate steps they are going to commit to, in order to prevent something from happening again in the future.

Even after that whole thing where PTS was pointed to live, and saved to the live database, there was a situation on xbox NA that logged you into PTS server for maybe 30-60mins after it came up discussed in this thread. Like how does that happen after the whole PTS fiasco a month ago, and why werent there more checks with server environments?

Regarding the recent freezing and rubberbanding, if they are finished some investigations, they should atleast give us some updates, and rule out things that are being speculated on. Like if they have investigated akamai latency, and concluded that its not an issue, they should just communicate that after they have done extensive testing and analysis of network log data, and have not found anything conclusive to point to akamai as the source of the lag spikes.

like we are customers, and its just frustrating not being able to enjoy a product we are paying for. We all provide alot of information when we can. I try to upload a video if im affected the night before. If a group of heavily affected individuals like Lady CP or myself for example, I feel ZOS should maybe just reach out to like the 5-10 of us, and work with us to gather the necessary information on when this is occuring, and we are more than happy to help. Like they could ask us to create a ticket, and share the ticket # with them, so it can be escalated directly to an engineer, and we can provide the time and information more quickly to them, and they can give us instructions on anything they want us to do.

SaintJohnHM

I agree with BaconAura above. The lack of communication and urgency from ZOS about a game-breaking issue like this makes us feel disrespected. How should we plan our time if we don't know if the game is going to work or not?

We continue to have huge lag spikes and simultaneous multiplayer disconnects in trials. We had a player disconnect at the last few seconds of a 36/36 KA trifecta last week.

LadyGP

baconaura wrote: »

twev wrote: »

thinkaboutit wrote: »

twev wrote: »

manukartofanu wrote: »

@twev May I ask, who are you telling this to? Those who work in this industry know without any statements from anyone that information about what work is being done cannot help any wrongdoers. Yes, not everything can be said. But that's no explanation for complete silence or standard repetitive answers.

P.S. If what you're writing about actually worked that way, we wouldn't be getting any announcements or patch notes.

Simple.
The explanation for the stuff that is silenced is that that is what the mother ship feels is best for their own ends.

Who I'm talking 'to' is the people who have a passing acquaintance with tech, networking and general business without being inside the machinations of big business intrigue and subterfuge.

Lots of things that non-tech people would think are just normal info are guarded as proprietary information, and many things that are in the public domain of information would get a tech worker or dev fired for breaching a non-disclosure clause in their contract, by even just commenting on the record about info that is being discussed in public.

I was just commenting about the constant requests for 'updates', and 'what are you working on?, 'how many people are working on the problem?', What have you tried so far?' and other 'innocent' questions that keep being repeated, with increasing frustration.

Some companies answer questions, other companies ignore many such questions that they feel answering is not in their fiscal interest to address.
Many people here seem to think that the company can be badgered into responding, when, clearly they have little interest in posting such information.
If they wanted us to know, they'd have told us details long ago.
If they were able to fix the issues for what they consider to be a worthwhile expenditure, they would have by now.
They either can't, or won't, do certain things, and they don't feel obligated to inform us of which those things are.
Some people can't or just won't, accept that.

It's like asking your kid 'Who broke the vase??', and they start telling you about their day in school.

Posting patch notes is completely outside the scope of answering many/most of all the tech questions and requests that get posted here.

Players here who work in tech already know what most of those things that can be discussed under company non-disclosure agreements are. A lot of players not in tech don't know how many things aren't for public consumption.

The complete silence you referred to in the penultimate line of your post isn't company incompetence at communication, rather, it's what they feel benefits them the most.
We can ask Kevin 12 times/day for details, and most of the questions will go unanswered because he's not permitted to comment, or he's not given details pertaining to the questions asked.


I was just addressing some of the causes behind the actions that result in forum user's frustrations in a 61 page thread.
In no way am I implying the frustrations are without merit.

What is the point of your post? Nice job being condescending with your little "passing acquaintance" comment there.

Here's the bottom line, the issue has been ongoing for months. What would you have us do? Sit in silence?

By all means, no, don't sit in silence.
One of the best things about an open forum is players sharing the stuff in this thread, so we aren't feeling 'alone'.
Just look at how many players post here that they are relieved to know it's not just them, and the diversity of locations and connections/equipment/platforms lets us know that it's not our fault when half of a group all freeze or disconnect at the same time.

Some of the problems probably stem from jamming all the new stuff, locations, flashy effects into an old old engine that wasn't designed for it.
Some of it probably stems from the number of DLCs and locations the engine is trying to keep knitted together.
Some of it is the servers keeping track of the population density in game, especially in a map, or specific area.
Some of it is the increasing number of items, collectibles, in inventory, and banks as more players log in on the servers.
[Notice how logging into a guild bank at random times shows the 'I'm thinking about it' icon forEVAR, or until you close the attempt and try again, maybe a few times.]
Some of it is probably even due to a dodgy circuit board or two/many in the servers thats doing the countdown to eventual failure. (we had that one a few months before the server rebuild, where they found a few boards that were going bad.) Some IC boards just drift further out of spec over time.
Lets not discuss the issues with Akamai. That's it's own black hole of info.

But to the point, this thread alone has allowed us to compare notes and see just how pervasive the issues are for many of us, outside of our own sphere's, regardless of the number of blessed players who chime in with 'Working for me, never had a problem'...

And the forum population is small compared to the number of game licenses.
I'll bet the majority of players with issues don't even know about the forum, deal with issues for a while, and then just delete the realm from their drives and move on to other games without ever being heard.

We're lucky that as many of the players with issues even came here and voiced their concerns, hoping to keep the game not only alive, but make the game a better experience for everyone else.
About all we can do is point out the issues, and try to convince the Mother ship that we're here, we care, and we're worth being heard, so the game doesn't circle the drain before problems get fixed.

And I'm about as cynical as they come.


[edit to add:]
I'm sorry if you thought I was being condescending.
It was not my intent.
I was merely pointing out that of the roughly 20 million licensed copies of the game in circulation - you can count on a rather large population of players who have never worked in tech, for some of them this is even their first game on their first computer.
There are a lot of people here who have never assembled a computer, configured a network, or can diagnose which addons might be causing a problem with the game.
Many people here have looked at error msgs and tried to figure out what they were referring to, and can't understand why they didn't have a particular problem last week, or last month, but now it's a problem that happens several times a day.

I got out of tech years ago when I realized that there were too many times when management made it harder to fix a problem than it could/should have been.

And as an aside, I've been in on many guild conversations, and group conversations over the years in this game alone where it was revealed that half the guild or group was made up of retired people, people who had worked the trades for their careers, and half the tech they learned was from gaming forums like this one.

it's not condescending to recognize those players as people who haven't had the benefit of inside experience hands-on with tech companies while they're just trying to figure out why their game doesn't work right.

But I appreciate you pointing out the opportunity to clarify the wording in my original responses.

Like they could ask us to create a ticket, and share the ticket # with them, so it can be escalated directly to an engineer, and we can provide the time and information more quickly to them, and they can give us instructions on anything they want us to do.

We did this around... the first week or two of this thread being made. I know OP has mentioned that they submitted many tickets and many YouTube videos (which last update OP said they didn't have a single view on them).

To be honest, without knowing the behind the scenes, it feels like to me Kevin brought this up a few times to them and no one really took it serious (or they were already aware and knew they can't fix it because it's an Engine issue). After 5 months of constant pining they are now "looking into it".

Hopefully they give something of value for @ZOS_Kevin to share with us because I know he is frustrated as well.

LadyGP

SaintJohnHM wrote: »

I agree with BaconAura above. The lack of communication and urgency from ZOS about a game-breaking issue like this makes us feel disrespected. How should we plan our time if we don't know if the game is going to work or not?

We continue to have huge lag spikes and simultaneous multiplayer disconnects in trials. We had a player disconnect at the last few seconds of a 36/36 KA trifecta last week.

The communication has been like this going back to 2017. To be honest, ZoS just has a "ignore it till the crying goes away" policy. They always have. Most gaming studios had this kind of mindset until recent years - just ZoS hasn't adapted yet.

thinkaboutit

Anyone have details on today's maintenance?

LadyGP

thinkaboutit wrote: »

Anyone have details on today's maintenance?

Nope, but I'll try logging in after sprint review and see.

Side note @ZOS_Kevin performance was VERY bad this weekend based on the last 3-4 pages here.

Jaraal

baconaura wrote: »

Regarding the recent freezing and rubberbanding, if they are finished some investigations, they should atleast give us some updates, and rule out things that are being speculated on. Like if they have investigated akamai latency, and concluded that its not an issue, they should just communicate that after they have done extensive testing and analysis of network log data, and have not found anything conclusive to point to akamai as the source of the lag spikes.

Except that there is a plethora of tracerts and other evidence posted by players that prove otherwise. So it would be quite strange if they made such an announcement.

However, you can piece together multiple statements from them over the years that give us clues as to the possible pain points. The sharding rework that had to be rolled back due to unforeseen negative effects, the network port hardware failure that they discovered, the "code rewrite" that evolved into "QOL improvements" instead, the announcement that alt achievements were hogging resources and had to be deleted to make room for new content, systems, and achievements (How many more of those things are clogging up server space in the years since?), the fact that they touted they change of one glyph as being a performance improvement, when they have released hundreds of new sets since then that have to make complex checks every tick..... and so on and so forth.

So I wouldn't expect them to be making many statements about what they've ruled out. In fact, it's a collection of things working together that is affecting performance. There's no single magic fix. However, the more they tinker with the code, the more opportunity for dysfunction is introduced. They have also stated on multiple occasions that one change they made broke something else that would be (attempted to be) fixed "in a future incremental patch." But we aren't always notified when that patch dropped, or even if it worked.

haleysarahw

Forgot to turn on my VPN yesterday and experienced significant ping spikes standing in the Galen bank trying to remove gear. Turned on my VPN and everything seemed fine.

LadyGP

haleysarahw wrote: »

Forgot to turn on my VPN yesterday and experienced significant ping spikes standing in the Galen bank trying to remove gear. Turned on my VPN and everything seemed fine.

Also had this happen. When I have VPN on things are "better"-ish.

@ZOS_Kevin any updates from the team?

katorga

Current experience today:

1. Random 300+ ping, just standing around with no players near
2. Lag playing overland ToT, lol
3. Extreme lag starting 3rd ARC in AI with two sorc pets, companion, dual pet vision, and maw. Unplayable.
4. Companions despawning when zoning

thinkaboutit

Called this one out a bunch of pages back lol..

Jaraal

thinkaboutit wrote: »

Called this one out a bunch of pages back lol..

And this is just ONE of 650+ and growing sets that have to be calculated by the server. And as time goes on, the calculations are becoming more and more complex, with so many different conditions that have to be kept track of. Add to that the ever increasing particle density of the flashy new mounts, armor packs, polymorphs, and whatnot, and the scribing variations, expanding pool of player achievements.... all on data hard capped servers.... and it's easy to see how performance can suffer.

Xandreia_

LadyGP wrote: »

With Thrones and Liberty coming out tomorrow and little to no update of substance thus far... and this issue going on for 5 months...

Consider this one of my last posts regarding this issue. My days with ESO are numbered at this point. As someone who has been with this game since beta I really hate to say it but ZoS has lost their way and IMO got complacent.

yeah same here, already pre downloaded throne so its probably bye bye eso for me, this has gone on far too long and weve had absolutely nothing from zos for coming up to 2 weeks, im done...

baconaura

Jaraal wrote: »

thinkaboutit wrote: »

Called this one out a bunch of pages back lol..

And this is just ONE of 650+ and growing sets that have to be calculated by the server. And as time goes on, the calculations are becoming more and more complex, with so many different conditions that have to be kept track of. Add to that the ever increasing particle density of the flashy new mounts, armor packs, polymorphs, and whatnot, and the scribing variations, expanding pool of player achievements.... all on data hard capped servers.... and it's easy to see how performance can suffer.

UI stuff like mount effects, particles, armor/weapons and even the azure blightseed explosion particle effects isnt done by the server. its done by the client with the assets stored and processed locally.

The issues we are having is network/serverside. Calculations maybe. but like i said a few pages back, they need to reinvest some of that $15m/month into upgrading servers over time. And lets be honest, 650+ sets, but i would wager less than 50 are probably seeing any use. half of them are flat stats without proc conditions. Even doing Infinite Archive solo or 4 man rnd dungeons, we are seeing this performance degradation. so its not limited to cyrodiil or 12man trials with everyone popping azure in vLC.

And computer hardware is many times faster than what eso started out with(2012 era hardware), and even from the upgraded stuff in 2022, its a few generations after now.

To be honest, if i were them, i would use some of that $15m/month and ask for help from their parent company microsoft to migrate more of their infrastructure into Azure. So they don't have to deal with the underlying infrastructure and can scale up easier/faster as new hardware becomes available, and focus on writing code, and let the Azure devops team manage the network infrastructure and service reliability.

Like at my work, we use AWS, which allows us to scale up in instance sizes, and horizontally via autoscaling based on load patterns. This allows us to focus more on the code/service itself, and worry less about the infrastructure. Sure we could host it ourselves, in a datacenter that isnt cloud like the old way(~2010 era), but its just so much more difficult to scale efficiently, and it would use up so much more time to manage and maintain.

KaironBlackbard

Jaraal wrote: »

thinkaboutit wrote: »

Called this one out a bunch of pages back lol..

And this is just ONE of 650+ and growing sets that have to be calculated by the server. And as time goes on, the calculations are becoming more and more complex, with so many different conditions that have to be kept track of. Add to that the ever increasing particle density of the flashy new mounts, armor packs, polymorphs, and whatnot, and the scribing variations, expanding pool of player achievements.... all on data hard capped servers.... and it's easy to see how performance can suffer.

I did have a suggestion to merge sets. Although technically that would just add more sets, or cause more sets to be firing... perhaps it wasn't as good an idea... If only we could test the theory.

KaironBlackbard

baconaura wrote: »

Jaraal wrote: »

thinkaboutit wrote: »

Called this one out a bunch of pages back lol..

And this is just ONE of 650+ and growing sets that have to be calculated by the server. And as time goes on, the calculations are becoming more and more complex, with so many different conditions that have to be kept track of. Add to that the ever increasing particle density of the flashy new mounts, armor packs, polymorphs, and whatnot, and the scribing variations, expanding pool of player achievements.... all on data hard capped servers.... and it's easy to see how performance can suffer.

UI stuff like mount effects, particles, armor/weapons and even the azure blightseed explosion particle effects isnt done by the server. its done by the client with the assets stored and processed locally.

The issues we are having is network/serverside. Calculations maybe. but like i said a few pages back, they need to reinvest some of that $15m/month into upgrading servers over time. And lets be honest, 650+ sets, but i would wager less than 50 are probably seeing any use. half of them are flat stats without proc conditions. Even doing Infinite Archive solo or 4 man rnd dungeons, we are seeing this performance degradation. so its not limited to cyrodiil or 12man trials with everyone popping azure in vLC.

And computer hardware is many times faster than what eso started out with(2012 era hardware), and even from the upgraded stuff in 2022, its a few generations after now.

To be honest, if i were them, i would use some of that $15m/month and ask for help from their parent company microsoft to migrate more of their infrastructure into Azure. So they don't have to deal with the underlying infrastructure and can scale up easier/faster as new hardware becomes available, and focus on writing code, and let the Azure devops team manage the network infrastructure and service reliability.

Like at my work, we use AWS, which allows us to scale up in instance sizes, and horizontally via autoscaling based on load patterns. This allows us to focus more on the code/service itself, and worry less about the infrastructure. Sure we could host it ourselves, in a datacenter that isnt cloud like the old way(~2010 era), but its just so much more difficult to scale efficiently, and it would use up so much more time to manage and maintain.

good points

I don't think merging with azure would be a good idea though.
This company is rich enough to harbor its own servers. It doesn't need to spend 1000 times more being harbored on cloud servers that they can't maintain for peak effectiveness.
They could swap ISPs though, if they're using Akamai as their ISP.
If they swap, we should see better performance.

LadyGP

Mentioned this in the Azure blight thread (that got closed) that the calculations were causing massive stress on the servers and you could literally feel the server chug along as the blight chain was going off and basically got told to sit down I'm dumb.

Glad to see I was right here.

baconaura

KaironBlackbard wrote: »

baconaura wrote: »

Jaraal wrote: »

thinkaboutit wrote: »

Called this one out a bunch of pages back lol..

And this is just ONE of 650+ and growing sets that have to be calculated by the server. And as time goes on, the calculations are becoming more and more complex, with so many different conditions that have to be kept track of. Add to that the ever increasing particle density of the flashy new mounts, armor packs, polymorphs, and whatnot, and the scribing variations, expanding pool of player achievements.... all on data hard capped servers.... and it's easy to see how performance can suffer.

UI stuff like mount effects, particles, armor/weapons and even the azure blightseed explosion particle effects isnt done by the server. its done by the client with the assets stored and processed locally.

The issues we are having is network/serverside. Calculations maybe. but like i said a few pages back, they need to reinvest some of that $15m/month into upgrading servers over time. And lets be honest, 650+ sets, but i would wager less than 50 are probably seeing any use. half of them are flat stats without proc conditions. Even doing Infinite Archive solo or 4 man rnd dungeons, we are seeing this performance degradation. so its not limited to cyrodiil or 12man trials with everyone popping azure in vLC.

And computer hardware is many times faster than what eso started out with(2012 era hardware), and even from the upgraded stuff in 2022, its a few generations after now.

To be honest, if i were them, i would use some of that $15m/month and ask for help from their parent company microsoft to migrate more of their infrastructure into Azure. So they don't have to deal with the underlying infrastructure and can scale up easier/faster as new hardware becomes available, and focus on writing code, and let the Azure devops team manage the network infrastructure and service reliability.

Like at my work, we use AWS, which allows us to scale up in instance sizes, and horizontally via autoscaling based on load patterns. This allows us to focus more on the code/service itself, and worry less about the infrastructure. Sure we could host it ourselves, in a datacenter that isnt cloud like the old way(~2010 era), but its just so much more difficult to scale efficiently, and it would use up so much more time to manage and maintain.

good points

I don't think merging with azure would be a good idea though.
This company is rich enough to harbor its own servers. It doesn't need to spend 1000 times more being harbored on cloud servers that they can't maintain for peak effectiveness.
They could swap ISPs though, if they're using Akamai as their ISP.
If they swap, we should see better performance.

yeah, but it feels like they cant even maintain their own infrastructure right now. scaling and upgrading hardware is way more complex and expensive in a non cloud environment. Atleast if its managed by Azure, they have their dedicated devops taking care of it. Like it's all owned by Microsoft at the end of the day, which owns Bethesda/ZOS. They could probably gain some efficiency there, and focus more on the service/code.

LadyGP

baconaura wrote: »

KaironBlackbard wrote: »

baconaura wrote: »

Jaraal wrote: »

thinkaboutit wrote: »

Called this one out a bunch of pages back lol..

And this is just ONE of 650+ and growing sets that have to be calculated by the server. And as time goes on, the calculations are becoming more and more complex, with so many different conditions that have to be kept track of. Add to that the ever increasing particle density of the flashy new mounts, armor packs, polymorphs, and whatnot, and the scribing variations, expanding pool of player achievements.... all on data hard capped servers.... and it's easy to see how performance can suffer.

UI stuff like mount effects, particles, armor/weapons and even the azure blightseed explosion particle effects isnt done by the server. its done by the client with the assets stored and processed locally.

The issues we are having is network/serverside. Calculations maybe. but like i said a few pages back, they need to reinvest some of that $15m/month into upgrading servers over time. And lets be honest, 650+ sets, but i would wager less than 50 are probably seeing any use. half of them are flat stats without proc conditions. Even doing Infinite Archive solo or 4 man rnd dungeons, we are seeing this performance degradation. so its not limited to cyrodiil or 12man trials with everyone popping azure in vLC.

And computer hardware is many times faster than what eso started out with(2012 era hardware), and even from the upgraded stuff in 2022, its a few generations after now.

To be honest, if i were them, i would use some of that $15m/month and ask for help from their parent company microsoft to migrate more of their infrastructure into Azure. So they don't have to deal with the underlying infrastructure and can scale up easier/faster as new hardware becomes available, and focus on writing code, and let the Azure devops team manage the network infrastructure and service reliability.

Like at my work, we use AWS, which allows us to scale up in instance sizes, and horizontally via autoscaling based on load patterns. This allows us to focus more on the code/service itself, and worry less about the infrastructure. Sure we could host it ourselves, in a datacenter that isnt cloud like the old way(~2010 era), but its just so much more difficult to scale efficiently, and it would use up so much more time to manage and maintain.

good points

I don't think merging with azure would be a good idea though.
This company is rich enough to harbor its own servers. It doesn't need to spend 1000 times more being harbored on cloud servers that they can't maintain for peak effectiveness.
They could swap ISPs though, if they're using Akamai as their ISP.
If they swap, we should see better performance.

yeah, but it feels like they cant even maintain their own infrastructure right now. scaling and upgrading hardware is way more complex and expensive in a non cloud environment. Atleast if its managed by Azure, they have their dedicated devops taking care of it. Like it's all owned by Microsoft at the end of the day, which owns Bethesda/ZOS. They could probably gain some efficiency there, and focus more on the service/code.

Boom.

Team I work with manages hundreds of these Azure instances. would be a boss move if they moved everything to here.

thinkaboutit

Wh

LadyGP wrote: »

Mentioned this in the Azure blight thread (that got closed) that the calculations were causing massive stress on the servers and you could literally feel the server chug along as the blight chain was going off and basically got told to sit down I'm dumb.

Glad to see I was right here.

where's that azure thread? I can't find it

baconaura

thinkaboutit wrote: »

Wh

LadyGP wrote: »

Mentioned this in the Azure blight thread (that got closed) that the calculations were causing massive stress on the servers and you could literally feel the server chug along as the blight chain was going off and basically got told to sit down I'm dumb.

Glad to see I was right here.

where's that azure thread? I can't find it

its in PTS section.

thinkaboutit

baconaura wrote: »

thinkaboutit wrote: »

Wh

LadyGP wrote: »

Mentioned this in the Azure blight thread (that got closed) that the calculations were causing massive stress on the servers and you could literally feel the server chug along as the blight chain was going off and basically got told to sit down I'm dumb.

Glad to see I was right here.

where's that azure thread? I can't find it

its in PTS section.

Looking for the closed one havn't found it yet

Sluggy

LadyGP wrote: »

Mentioned this in the Azure blight thread (that got closed) that the calculations were causing massive stress on the servers and you could literally feel the server chug along as the blight chain was going off and basically got told to sit down I'm dumb.

Glad to see I was right here.

Which, honestly, is insane. Like, I get that this engine is old and written in a time before multi-core CPUs but I can't imagine how inefficient it must be. There can't be that many players using this so frequently and on so many targets that it actually would cause a performance issue on even a desktop machine much less industrial-scale hardware.

EDIT: On second thought, I suspect performance could mean any number of things here. While I don't think you'd ever stress a CPU even in a large scale PvP fight with only a few dozen people procing this effect I *could* see how it would stress the network if a ton of events that all have high priority (like deaths, etc) all trigger at once and need to be sent out asap. I've long thought that bandwidth was a large part of the issues and this sort of thing would check out with that line of logic.

lpoki

It sounds funny that Azure caused a serious drop in server performance.

CatoUnchained

lpoki wrote: »

It sounds funny that Azure caused a serious drop in server performance.

I don't know anyone who's laughing though. Everyone I know is done with anything ZOS if they don't fix this disconnecting issue.

Logos

Today is the worst it's been for me. I tested with 3 others games running at the same time as ESO, and while people are skipping around on my screen and I can't interact with merchants, the other games have 0 connection issues, no lag spikes or packet loss.

silky_soft

Error 307 Booted from server again. Person playing pc next to me didn't. They still playing