Professional Opinions on ZOS Infrastructure

Greevir

Downtime doesn't bother me too much. The lack of communication during and after these downtimes are what frustrates me as a customer. Game was down a long time last night and we haven't heard anything about it from ZoS.I just want more communication. Is there anyway we can get @ZOS_GinaBruno to dismiss her real life and focus on ESO 24/7 for us?

capper123

dont know how any of this may play into it but couple years ago there was a terrost attack somewhere and through the
investigations they found they were organizing in online games. so i heard that the govt monitors all game chat. how that system runs i dont know but its an extra step in the system.

Agalloch

I understand ESO is "Not a Standard MMO" as devs said.

Actually is a very nice and complex MMO.

Also the lore is fantastic.

It's missing some standard features which every mmo have ( customizable UI , real gear comparison and mechanics tips , real progression ( horizontal and vertical) and many other things that I could enumerate but this is not the proper thread to remind about them)

ESO is fantastic but the real problem is since launch and until now we see almost every time ..almost the same problems : random lag spikes, long down times, unfinished patch notes , unannounced maintenances , server crashes ( remember the pvp event?)

Performance is better anyway than it was 2 or 3 years ago.


I'm also working in IT ..and take care of servers ..and know what means to bring them offlline...to interrupt the services .

And speaking about MMOs ..ESO is probably the one with the most downtime from all of them. And I played a lot of them since 1997

I hope until the beginning of the next year the devs to upgrade the infrastucture and the bandwidth .

Also they need to hire better coders and if it is possible to bring back some of the people who left.

Good luck ZOS!


English is not my native language.

rootimus

capper123 wrote: »

so i heard that the govt monitors all game chat. how that system runs i dont know but its an extra step in the system.

I heard this one guy was born in Kenya, not Hawaii. Neither have anything to do with the topic at hand, even if what you heard was accurate.

Eve Online doesn't have the infrastructure issues that ESO has. As previous posts have noted, it's likely all down to money.

Elsonso

Opticon wrote: »

Periodic unplanned downtime to fix code-related bugs, like yesterdays for example, are to be expected and should even praised when a game-breaking flaw has been found. This thread has zero to do with instances like that. Could tonight's downtime be another example of this? Unfortunately we will most likely never know, but this thread is absolutely NOT just about tonight....

Any Internet presence worth anything will eventually get some sort of DDOS. Why DDOS someone when you won't get any response from it? That said, there are plenty of services out there to help protect against it, and plenty of things in-house that can be done to mitigate (NOT 100% prevent) the problem. I hate to say this part but... some people on here will be like "Oh well Gmail/Facebook/Yahoo doesn't go down for hours" etc etc. While their basis for the argument is generally flawed, their point is valid.

I can't fathom how management in the server/network department(s) allow for such horrendously frequent downtimes. Sure ESO (and basically all MMO's) make it clear in their TOS that they do not guarantee availability, but does anyone expect downtime to be often and for so many minutes at a time? Technically they could be up for an hour a month and still fall under that clause, so let's be realistic here.... the bottom line demands a certain level of service availability to retain customers, especially for an exclusively online service. We all know about five 9's etc etc, but I'm not sure if this could even be a one 9.

Systems people, Network people, Devs, etc.... what do you think about all of this? I just can't wrap my head around it frankly. Anyone with first-hand gaming industry experience please do comment from your perspective.

A note on DDoS, before I start. We know that ZOS is investing in DDoS mitigation for NA. It is plainly evident, like the nose on your face. I have no idea what they are doing for EU, but I am confident that if they need to do something, they will be doing something there, too.

It is extraordinary, in the history of ESO, to have three down times in one week. It does not happen very often. Say what you will about ZOS, but they are able to routinely limit down time to once a week.

Weekly routine maintenance seems to be what the megaserver was designed for. They can run longer, but "weekly" maintenance is required. There is nothing wrong with that. Can they design a system that does not need a near weekly down time? Definitely. Is it worth the expense to do that? ZOS obviously feels that it is not. This is a game. Weekly maintenance is fine.

Routine maintenance seems to take 2-4 hours, with patch installation and other tasks taking additional time. Can it be sorter? Maybe. Sometimes things just take time and optimizing the process ends up being a lot of money spent for a relatively small return. Again, the issue comes down to one of how much do they want to spend to achieve what % of efficiency. As this is a game, I think that maintenance that takes a few hours is fine, if that is the level of spending that they are comfortable with.

In any case, without knowing more about their HA environment and the extent to which they have hardware redundancy and fail over, and how hardened the facilities (power, network, HVAC, etc) are, nothing more can really be said about the infrastructure. My feeling, from just observing, is that the infrastructure is more than adequate for the game that it supports, and that the team that runs the infrastructure (part of Live Services, apparently) is doing fine.

If anything, the game server software is not efficiently using the infrastructure that is running it. My impression is that a lot of the software that runs this game has evolved from a period where they were not as experienced. This all needs to be updated or replaced, when it causes problems, and this is a Big Thing. That is often the sort of thing that gets addressed in The Next Game, because it is just too much work to try and fix it in place.

I know "this thread is absolutely NOT just about tonight", but it really is. We don't know why they had to take the server down in the middle of the night, but whatever it was, it was an exceptional reason. They don't like to take the server down in the middle of the night. I actually cannot remember the last time they did. The overnight maintenance is an anomaly, a one off, until we find out otherwise.

The maintenance earlier this week for the (rumored) gold exploit was a software problem in the game. They obviously felt it was important to take the game down, fix the problem, and clean up the mess. I tend to agree.

It is nice that they took the server down to fix the exploit. Too many times, they let exploits run for weeks while they pretend they don't care. I would like to give some credit the people on the forum for them fixing it so fast, but I also gather that it was a big issue that they probably would have expedited a response on, anyway. We want ZOS to address these things right away, and they did. Conditional victory for the players.

However, the exploit should not have been there in the first place. The hot fix they did earlier this week should not have been necessary. The treasure hunter passive should be working. (etc) This goes back to the core problem that I see at the studio. From a development quality perspective, they are still have the appearance of being wildly out of control, three years into the game. I have no idea what it is they are doing behind the curtain. I am certain it is really cool, but it certainly seems inadequate for what they need to do.

That, more than network and server infrastructure, is what is driving ESO downtime.

Network and server infrastructures appear the be under control and well managed. The bulk of the problem seems to be coming from the development side of the studio in the form of process quality problems and less than optimal design, probably due to a number of cost-benefit decisions, and simple learning curve issues that have not been, or cannot be, addressed.

EttinTheBreton

Just wanted to post this here for everyone to see:

ZOS_GinaBruno wrote: »

Just to let everyone know what happened, we had an issue with our backend infrastructure, but was able to resolve it and also determined why it became an issue in the first place. Thanks so much for your patience overnight!

That quote is comment #160 in this thread: https://forums.elderscrollsonline.com/en/discussion/366953/pc-na-server-down/p6 It did end up being an infrastructure problem.

ManDraKE

What always annoyed me about this game servers is the awful routing when coming from outside the US (i'm in south america). I usually get like 180ping to US servers in the East Coast, and in general all games have their servers there (or at least a few if it is a multi-realm game) because it provides way better ping for everyone outside the US. With ESO i have over 280 ping, is insane, and i'm not is not just my case because is the same problem for Australian players for example. I asked a couple times to add something like a proxy location on East Coast with a low-latency to the ESO datacenters to mitigate the awful routing for everyone playing from outside the US. I been playing all kind of games with servers in the USA and this is the only one with this absurd ping.

idk

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

Doctordarkspawn

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

Then back up the asserstion with some facts. What precisely is he geting wrong? Why? Give specifics or shut up.

ccfeeling

They don't know how to run game servers correctly

idk

Doctordarkspawn wrote: »

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

Then back up the asserstion with some facts. What precisely is he geting wrong? Why? Give specifics or shut up.

lol @Doctordarkspawn

Read the post before replying. One downtime recently due to a system failure outside of DDoS or planned/emergency maintenance and he called it horrendous and frequent. Lmao.

I require him, or you, to provide proof to the contrary To back up his words. Otherwise, move on.

rootimus

Giles.floydub17_ESO wrote: »

most of what we have dealt with has been DDoS attack.

Yet there's little, if any, evidence to support this when looking at external tools that monitor DDoS attacks. The correlation between ESO events and massive lag issues on the ESO server is astounding (and, yes, I know correlation doesn't equal causation).

idk

rootimus wrote: »

Giles.floydub17_ESO wrote: »

most of what we have dealt with has been DDoS attack.

Yet there's little, if any, evidence to support this when looking at external tools that monitor DDoS attacks. The correlation between ESO events and massive lag issues on the ESO server is astounding (and, yes, I know correlation doesn't equal causation).

Even OP admits DDoS attacks and they are a constant these days.

Aurielle

Opticon wrote: »

Aurielle wrote: »

In the many MMOs I have played over the years, downtime is a fact and a reality. I even remember LoTRO going down for 5 whole days! You can imagine how lit the forums were.

Yes you are absolutely correct, however it still begs to question... why, exactly. From a server standpoint, which I am taking, there is little to no acceptable excuse besides underfunding of their infrastructure team(s).

I don't mean to argue with you here, but the "this is how other games are" doesn't really excuse why it's currently an issue in ESO.

I don't think @Nermy and I were necessarily giving a free pass to ZOS for their performance issues. Yes, it would be ideal if we had little to no downtime, and yes, it would be ideal if we had a consistently lag-free, low-latency experience. However, the fact that these issues are so ubiquitous among MMORPGs suggests to me that keeping MMOs stable is a very difficult undertaking. Comparing it to other networking applications might be unfair.

rootimus

Giles.floydub17_ESO wrote: »

rootimus wrote: »

Giles.floydub17_ESO wrote: »

most of what we have dealt with has been DDoS attack.

Yet there's little, if any, evidence to support this when looking at external tools that monitor DDoS attacks. The correlation between ESO events and massive lag issues on the ESO server is astounding (and, yes, I know correlation doesn't equal causation).

Even OP admits DDoS attacks and they are a constant these days.

The OP has also been very scrupulous about stating they don't know anything that goes on within Zenimax. DDoS attacks are common, yes, which is why people seize on them to handwave away organisational failings. It's become the new "it must be a virus".

idk

rootimus wrote: »

Giles.floydub17_ESO wrote: »

rootimus wrote: »

Giles.floydub17_ESO wrote: »

most of what we have dealt with has been DDoS attack.

Yet there's little, if any, evidence to support this when looking at external tools that monitor DDoS attacks. The correlation between ESO events and massive lag issues on the ESO server is astounding (and, yes, I know correlation doesn't equal causation).

Even OP admits DDoS attacks and they are a constant these days.

The OP has also been very scrupulous about stating they don't know anything that goes on within Zenimax. DDoS attacks are common, yes, which is why people seize on them to handwave away organisational failings. It's become the new "it must be a virus".

Easily said empty words.

Drakkdjinn

My professional opinion on ZoS infrastructure: it's made of popsicle sticks and used hypodermic needles.

rootimus

Drakkdjinn wrote: »

My professional opinion on ZoS infrastructure: it's made of popsicle sticks and used hypodermic needles.

TazESO

Welcome to DevOps. Nuff said

Elsonso

Drakkdjinn wrote: »

My professional opinion on ZoS infrastructure: it's made of popsicle sticks and used hypodermic needles.

The needles confuse me, but you are a professional ice cream truck driver?

Egonieser

Opticon wrote: »

aToken wrote: »

I just feels like manpower or something is low. Like there is a rush or struggle somewhere. Causing the careless mistakes. Maybe overwhelming employees almost.

I can wholeheartedly agree with that! I've seen the outcome from understaffed groups, and this kinda smells just like it.

Do remember, ZoS original developers and large portion of the staff left after launch, and some more after console launch. It was felt tremendously in terms of bugs, issues, netcode problems because when someone is working on a game for almost 7 years, who knows the best about it's build and architecture? Certainly not those who were replacing many of the senior staff and many lead programmers who left. The newcomers had to learn years worth of progress in tiny amount of time resulting huge problems both in game and server infrastructure spheres.

If you look at the original interviews with many ex-lead programmers and staff, they had entirely different visions of the game in mind. Although details being undisclosed, I can only imagine there being huge disagreements between high ups and development team resulting in many lost jobs and cutbacks. The result was obvious and still haunts the game and it's servers to this day.

di_rty

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

What is the point in even calling out OP's credentials? Judging by your spelling and sentence structure, you may be foreign, but here in America we have freedom of speech and the right to amicably question the procedure of a company. Furthermore, there is no demand of response from ZoS, they're just asking a question to fellow forum readers. You say you're "someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with" and it just seems like a personal attack. Who cares if you've dealt with more "critical systems"? It doesn't negate that IT businesses employ standard practices like server redundancy, version control and internal testing. All that seem to be missing in the case of ESO. It seems that OP is trying to provide a place for other IT professionals to voice their opinion on the practices of ZoS and in my humble opinion, seems to be a success, considering there are 3 pages of posts that vary greatly in assessment.

Rohamad_Ali

The head of the IT department is a very qualified and expert in the field of cable .

rootimus

di_rty wrote: »

here in America we have freedom of speech and the right to amicably question the procedure of a company

I don't want to derail the thread, but your first amendment only protects you against your goverment restricting your speech, except when your congress or supreme court ignore it because they can. The only rights to speech on a Zenimax forum are those given by Zenimax.

Tomg999

Longtime IT Solution Architect here.

I find that there is a "free market" type of situation around user expectations of availability.
Consumers of IT services have different expectations & tolerance levels for uptime and proper functionality depending upon the service, and their level of tolerance is a dynamic "tug of war" between the provider and the consumer of the service.

Users vote with their feet, so to speak, and SW implementers see what they are required to provide to keep their consumers.

For example, a new version of Windows OS - users have a certain expectation & tolerance for bugs, reboots, etc. A new stand alone game, same thing. On the other hand, a financial app that a customer runs their business on has much stricter requirements for uptime, data loss, etc. Their customers will not tolerate the same level gamers do when a new patch comes out.
This is true outside IT as well; anyone here fly commercially 20 years ago? When you bought a ticket, you actually expected to get there around when it was scheduled, and without an added 2 hrs on each end of the trip.

I believe that in the case of an MMO like ESO this tension between consumer & provider, and the lack of clarity around the feedback loop, contributes to all of the frustration posts and threats to quit, and frustration with those posts as well. it is not like candy bar sales, where they make it a little smaller, and than run some numbers to see if sales go up or down. It is more like pay scales for skilled workers, if either side miscalculates, it can be too late to fix.

As a side note, this is an interesting dilemma as IT moves more and more to subscription & consumption based services.

Doctordarkspawn

Giles.floydub17_ESO wrote: »

rootimus wrote: »

Giles.floydub17_ESO wrote: »

rootimus wrote: »

Giles.floydub17_ESO wrote: »

most of what we have dealt with has been DDoS attack.

Yet there's little, if any, evidence to support this when looking at external tools that monitor DDoS attacks. The correlation between ESO events and massive lag issues on the ESO server is astounding (and, yes, I know correlation doesn't equal causation).

Even OP admits DDoS attacks and they are a constant these days.

The OP has also been very scrupulous about stating they don't know anything that goes on within Zenimax. DDoS attacks are common, yes, which is why people seize on them to handwave away organisational failings. It's become the new "it must be a virus".

Easily said empty words.

As are yours.

idk

di_rty wrote: »

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

What is the point in even calling out OP's credentials? Judging by your spelling and sentence structure, you may be foreign, but here in America we have freedom of speech and the right to amicably question the procedure of a company. Furthermore, there is no demand of response from ZoS, they're just asking a question to fellow forum readers. You say you're "someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with" and it just seems like a personal attack. Who cares if you've dealt with more "critical systems"? It doesn't negate that IT businesses employ standard practices like server redundancy, version control and internal testing. All that seem to be missing in the case of ESO. It seems that OP is trying to provide a place for other IT professionals to voice their opinion on the practices of ZoS and in my humble opinion, seems to be a success, considering there are 3 pages of posts that vary greatly in assessment.

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

Doctordarkspawn

Giles.floydub17_ESO wrote: »

di_rty wrote: »

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

What is the point in even calling out OP's credentials? Judging by your spelling and sentence structure, you may be foreign, but here in America we have freedom of speech and the right to amicably question the procedure of a company. Furthermore, there is no demand of response from ZoS, they're just asking a question to fellow forum readers. You say you're "someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with" and it just seems like a personal attack. Who cares if you've dealt with more "critical systems"? It doesn't negate that IT businesses employ standard practices like server redundancy, version control and internal testing. All that seem to be missing in the case of ESO. It seems that OP is trying to provide a place for other IT professionals to voice their opinion on the practices of ZoS and in my humble opinion, seems to be a success, considering there are 3 pages of posts that vary greatly in assessment.

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

See, was that so hard? Giving those specifics?

It wasn't, was it?

idk

Doctordarkspawn wrote: »

Giles.floydub17_ESO wrote: »

di_rty wrote: »

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

What is the point in even calling out OP's credentials? Judging by your spelling and sentence structure, you may be foreign, but here in America we have freedom of speech and the right to amicably question the procedure of a company. Furthermore, there is no demand of response from ZoS, they're just asking a question to fellow forum readers. You say you're "someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with" and it just seems like a personal attack. Who cares if you've dealt with more "critical systems"? It doesn't negate that IT businesses employ standard practices like server redundancy, version control and internal testing. All that seem to be missing in the case of ESO. It seems that OP is trying to provide a place for other IT professionals to voice their opinion on the practices of ZoS and in my humble opinion, seems to be a success, considering there are 3 pages of posts that vary greatly in assessment.

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

See, was that so hard? Giving those specifics?

It wasn't, was it?

Glad you took the time to read this time. Wasn't to hard I hope.

di_rty

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

Please grow up. All you've done is attack people in this thread and provided nothing to contribute to the actual cause of the thread. Someone who worked on federal (USA) government infrastructures sure as hell wouldn't act like you are. So I call BS.