Professional Opinions on ZOS Infrastructure

idk

di_rty wrote: »

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

Please grow up. All you've done is attack people in this thread and provided nothing to contribute to the actual cause of the thread. Someone who worked on federal (USA) government infrastructures sure as hell wouldn't act like you are. So I call BS.

@di_rty

I'd suggest reading your previous post to me where you started off with a clear intent to belittle me and my grammar. Extremely clear.

Second, where did I say I worked on federal infrastructure? Federal requirements do not mean federal assets (not in the USA at least). Please get off your high horse and stop accusing me of exactly what your doing.

Doctordarkspawn

Giles.floydub17_ESO wrote: »

di_rty wrote: »

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

Please grow up. All you've done is attack people in this thread and provided nothing to contribute to the actual cause of the thread. Someone who worked on federal (USA) government infrastructures sure as hell wouldn't act like you are. So I call BS.

@di_rty

I'd suggest reading your previous post to me where you started off with a clear intent to belittle me and my grammar. Extremely clear.

Second, where did I say I worked on federal infrastructure? Federal requirements do not mean federal assets (not in the USA at least). Please get off your high horse and stop accusing me of exactly what your doing.

No, what you've done is come in here and needlessly bash a bunch of people trying to have reasonable discussion because they dared to criticize a game you like.

In otherwords exactly what he's accused you of. After you called the people who started this threads claims bull after refusing to state why.

Please, just leave. You bring nothing of substance to this thread.

idk

Doctordarkspawn wrote: »

Giles.floydub17_ESO wrote: »

di_rty wrote: »

@di_rty

Lol. Yes. Foreigner. You clearly don't understand freedom of speech.

Love the single paragraph and criticism of my grammar.

Zos probably has a minimal amount of server redundancy. However, the game and its servers are not at the level of critical infrastructure that requires a high level of redundancy. Certainly not a dual system.

Last network I was part of building had a redundancy at every point. Dual data paths, dual power supply at most location and battery backups at every location. Some virtual networks on the system had 4 levels of redundancy due to federal (USA) government requirements. Critical infrastructure if you will. ESO isn't to that level.

I am not arguing Zos should not do more. They should as this is the least stable game I've played. But to your point, I'm exercising my free speech to suggest OP doesn't really know what's going on, outside of what Zos has told us.

Please grow up. All you've done is attack people in this thread and provided nothing to contribute to the actual cause of the thread. Someone who worked on federal (USA) government infrastructures sure as hell wouldn't act like you are. So I call BS.

@di_rty

I'd suggest reading your previous post to me where you started off with a clear intent to belittle me and my grammar. Extremely clear.

Second, where did I say I worked on federal infrastructure? Federal requirements do not mean federal assets (not in the USA at least). Please get off your high horse and stop accusing me of exactly what your doing.

No, what you've done is come in here and needlessly bash a bunch of people trying to have reasonable discussion because they dared to criticize a game you like.

In otherwords exactly what he's accused you of. After you called the people who started this threads claims bull after refusing to state why.

Please, just leave. You bring nothing of substance to this thread.

@Doctordarkspawn I merely pointed out OP made assumptions. It really boiled down to that. Your the first one that flamed on my response and did so in a manner that made clear you had not bother reading my post.

Then when I simplified it your response was clearly intended to be belittling. Seriously. I look at this last post of yours and lol.

You and dirty are merely attempting to derail this thread by bashing people like me who pin things out differently.

In fact each post you made in response to posts I made were specifically intended to bash me rather than add to the conversation or provide an oppinion on the topic. Who has the sole intent of bashing here?

Have a great day.

Mettaricana

Game is ok at best the huge lack of fun events and basic quality of life things that every mmo of this nature has restyling better quests less boring fetch and run quests. But a huge lack of other things as well the lromosed horse racing etc a team pvp that can be done anywhere not just battlegrounds etc just a lot of stuff always on the back burner and i grown tored waiting for atuff that may never come or gets halfassed on arival

Reivax

They are a big enough company to invest money in infrastructure and to have sufficient hardware for failovers.
They're also a sufficiently large enough endeavor to manage some protection against DDOS attacks
I do not feel sorry for the company in any way, nor do I feel like "oh, they're doing a great job." as if they're some group of high school students on a CIS lab project.
If ZOS cannot afford to run an MMO correctly, they probably ought to not try running one.

The simple fact that 100% of the time I can access an EU server when the NA servers are choking, should tell you something. Without fail, I can always reach EU servers, every time. So it's not "routing" or "open your ports!"
Like I'm frequently told in Cyrodiil, "Get good"

SirAndy

Opticon wrote: »

Anyone with first-hand gaming industry experience please do comment from your perspective.

I have worked in the gaming industry for many years (developer, manager, executive) and i was part of the super secret Psijic Order beta group.
We had 24/7 access to the PTS (Then called Private Test Server) for over 6 months before ESO launched on the PC.

Even back then things were already so bad that i actually send in my resume and applied for a job at ZOS while ESO was still in beta.
I genuinely believe that my experience in both building games and building high performance, enterprise level, client/server applications could have been a great benefit for ESO.

DMuehlhausen

aToken wrote: »

It's not the downtime imo that's bad it's the excessive amount of bugs every time there is a patch. Almost like patch is executed porely or to quickly

If you went through all the code and saw how many systems interact with eachother you would rethink your statement.

When you truly realize how much is going on in a game like this it's amazing they even get finished sometimes.

D0PAMINE

I don't believe ZOS are as open as they used to be with potential issues, coupled with the forums experience with the countless overreacting threads and comments targeting ZOS as being incompetant or greedy etc, have really hurt what could have been an effective improvement tool that that could have allowed more involvment with a very dedicated communiry of fans.

anitajoneb17_ESO

SirAndy wrote: »

Even back then things were already so bad that i actually send in my resume and applied for a job at ZOS while ESO was still in beta.

They were soooo bad that you... wanted to work for them ? With them ?
Is that what you thought ? Is that what you told them ?
If so, it's not really surprising that they rejected your application...

Or is there a misunderstanding here ?

Opticon

Giles.floydub17_ESO wrote: »

Doctordarkspawn wrote: »

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

Then back up the asserstion with some facts. What precisely is he geting wrong? Why? Give specifics or shut up.

lol @Doctordarkspawn

Read the post before replying. One downtime recently due to a system failure outside of DDoS or planned/emergency maintenance and he called it horrendous and frequent. Lmao.

I require him, or you, to provide proof to the contrary To back up his words. Otherwise, move on.

This is hilarious. You "require proof"? You chose to focus on one line in my entire post and blew it out of proportion and context. Take your own advice and move on, you clearly have nothing of value to add to this thread.

rootimus

anitajoneb17_ESO wrote: »

SirAndy wrote: »

Even back then things were already so bad that i actually send in my resume and applied for a job at ZOS while ESO was still in beta.

They were soooo bad that you... wanted to work for them ? With them ?

That's what I call being part of the solution, rather than part of the problem.

SirAndy

anitajoneb17_ESO wrote: »

SirAndy wrote: »

Even back then things were already so bad that i actually send in my resume and applied for a job at ZOS while ESO was still in beta.

They were soooo bad that you... wanted to work for them ? With them ?
Is that what you thought ? Is that what you told them ?
If so, it's not really surprising that they rejected your application...
Or is there a misunderstanding here ?

Yes, that was the general idea.
And no, i'm not really surprised they didn't invite me for an interview. Nobody likes to be told that you think they need help.

My Silicon Valley salary requirements probably didn't help either.

idk

Opticon wrote: »

Giles.floydub17_ESO wrote: »

Doctordarkspawn wrote: »

Giles.floydub17_ESO wrote: »

I truly doubt OP has worked as a systems engineer to any degree. Based on tye original statement he certainly doesn't have a clue.

We have dealt with much, especially in the past few months. Outside of the obvious planned server maintenance that will happen regardless and a couple unplanned maintenance such as two days ago most of what we have dealt with has been DDoS attack.

That leaves last night as a system failure on Athens part of Zos and OP is absurd enough to call a single hardware failure affecting us as an example of Sos permitting horrendous downfield.

OP lacks a clue for sure.

As someone who has worked with networks dealing with systems much more critical that ESO ans cleared more critical than what OP has dealt with hardware does fail from time to time

Maybe OP can provide more specific examples outside of last night that aren't DDoS attacks or needed system maintenance. Otherwise my statement holds pretty strongly.

Thx for the laugh though. Always get a kick out if the "expert" assessment.

PS. In no way am I suggested by Zos doesn't need to up their tame. Just pointing out OP's assessment is rather lacking.

Then back up the asserstion with some facts. What precisely is he geting wrong? Why? Give specifics or shut up.

lol @Doctordarkspawn

Read the post before replying. One downtime recently due to a system failure outside of DDoS or planned/emergency maintenance and he called it horrendous and frequent. Lmao.

I require him, or you, to provide proof to the contrary To back up his words. Otherwise, move on.

This is hilarious. You "require proof"? You chose to focus on one line in my entire post and blew it out of proportion and context. Take your own advice and move on, you clearly have nothing of value to add to this thread.

Not really. Just saw it as an assumption being made earlier. I agree, I think this is basically what you were getting to, that ESo has been the least stable platform I have played on and Zos do more.

Egonieser

DMuehlhausen wrote: »

aToken wrote: »

It's not the downtime imo that's bad it's the excessive amount of bugs every time there is a patch. Almost like patch is executed porely or to quickly

If you went through all the code and saw how many systems interact with eachother you would rethink your statement.

When you truly realize how much is going on in a game like this it's amazing they even get finished sometimes.

Well yeah, it's not like people are PAID to get it done, no... It is us who have to cut them slack for not doing what they are paid to do after already parting from our money and expecting some kind of functioning service up to a certain standard.

If you go into a job you know you are underqualified - you are not fit for the job and don't deserve the money it pays, pure and simple.
There are a LOT more complex things in the world, and i mean A LOT, and it gets done and nobody complains and expects any less of it, it gets done because people get paid to get it done, but in this industry some people somehow think it's ok not to have that standard. Comments from folks like you certainly don't set the right precedent and devs think they can get away with it because they know white knights like you will back them up and downplay those who actually do expect some kind of consistency and service for their money.

No, I personally will keep pestering and pestering until things improve, because I know one thing for sure, complacancy doesn't improve the product, only complaints and consumer dissatisfaction and criticism does.
And that doesn't mean I have to work for them, No. I pay them to do it. And if they can't do it, means they are not qualified for the job they are currently sitting at.

Just because it's a game doesn't mean it's right for them not to keep it as polished and top shape as possible. It's a business first and foremost, the only time I would cut them some slack if it was a nonprofit organization where all the proceeds went to charity or something, that would be the ONLY exception. Otherwise, it's just another product that me, as a consumer has purchased (or leased the right to use) and I will demand, as is my right, to have better quality and service. "It's just a game" "It's far too complex" is not a valid argument in any sense. They built a product they thought they could handle and manage well and sold it as such, and in return I will consume it and deem it as such, and if I deem the quality of it being sub-standard, I will complain and no white-knighting will persuade me to do otherwise.
It's not like they marketed it as: "Well, this is our great game, it has X, Y and Z and these cool features and A, B, C and numerous amounts of bugs we are not likely to be able to fix. Buy it at your discretion." Maybe with that level of openness and honesty I would be able to forgive and forget as it was disclosed and brought open beforehand, but that never happens does it, because being honest doesn't sell things.

Malic

Let me give a different perspective, this will of course be another post on the internet so believe whatever you want.

I used to own a business, one of four over my life. This particular business I was a partial owner but was given control of operations to get the product viable to be bought out by a larger company. It was a software product that ran on servers and allowed the server owner to mine data and watch where people went etc. This was in the 90's during the dot.com bubble to give a perspective.

Im a finance guy, I can get around on a PC no problem even trouble shoot basic stuff, at the end of the day my forte was reading finanicals, forecasting and predictive analytics. So im running this company to keep this product up to date required a lot of engineers but we also had network and server people who trouble shooted as we did host servers for some clients (wasnt the big money maker just an add on service).

We paid those clowns a lot of money to ensure the servers were up and running 24/7. We invested in back up servers and ran rotations so that maintenance would not disrupt service. For 3 years never had an issue, I sold my part of the business and got out.

My point?

There were people on my payroll who were paid to keep things running, i never questioned their expertise it would be like me going to an auto mechanic, he tells me I need an alternator and I argue with him that I actually need spark plugs. I dont know the pressures of their job and IT but I do no in the 20+ years since the general principal is the same. If you have a customer base that relies on your network and you derive income from it, you invest in assurances that the servers dont come down.

Human capital, backups, capital assets whatever it takes within a good cost basis. The problem we have here is this simple logic doesnt apply to MMO's because WE PLAY ANYWAY.

The game will run like crap this weekend, you'll have load screens, crashes, log in issues and it doesnt make a god dam difference. You'll still log in, pay, and keep playing. Whats the worse you are going to do QQ on the forums? In my business I couldnt do that, the consumer relied on my servers to host their data so they could derive income.

This is all leisure activity and ZOS knows it. You arent actually harmed by the servers running like crap your out disposable income and leisure time. That isnt to say that it should be like this but the business model is different, no matter what happens youre still logging in and still paying to play. Even more so now then ever before.

I would have killed for customers like this, the only accountability is a made up name on a message board squaking about how the devs must be idiots that they arent going to read.

Who are you shittin? You arent changing a dam thing.

Saturn

Opticon wrote: »

Nermy wrote: »

I love the game and my only gripe would be there is not enough communication from ZoS. Sure @Gina_Bruno and @Jessica_Folsom do their best but they can only communicate what they are given. I also get fed-up with the entitlement threads... f-off with those.

Communication is crucial, but as you know this forum is about 95% hate to one thing or the other, so I can understand the limited responses. HOWEVER.... a BIT more detail would go a LONG way to making customers happy.

I honestly don't know if I fully believe the DDOS story they are putting out, but then I'm also not an expect, but I just feel like they said nothing about it for 5 months until all the people on the forum forced an answer out of them. If it is DDOSing then f**k whoever is doing it, but honestly, I just wish they would be way more transparent about stuff like that. I mean they didn't even pin the thread where they talk about it, which to me seems kind of odd, since that's something I'd put right at the top of the forum, so people know what's up, when they come complaining about server downtime every week.

I've played a few other MMOs in the past as well, and feel like they weren't as consistently poorly optimised nor had as much downtime, but with those you had to deal with log-in queues for your server during primetime, which is something they've forgone with ESO.

To be honest, the servers crashing don't bother me nearly as much as the poor optimisation does. I swear they broke something with the game (for me at least) when they released One Tamriel, which put quite the strain on their servers as well, or at least the client-side, as now you tend to have to wait a minute or two for npcs / players to render in crowded cities, as well as general performance in cities being absolute turd.

Opticon

Malic wrote: »

Let me give a different perspective, this will of course be another post on the internet so believe whatever you want.

<quote snipped>

Excellent points all around, thanks for contributing.

thamightyboro

Opticon wrote: »

First off -- This is not an entitlement thread, this is not a ZOS is horrible thread or an I want a refund thread... this is (hopefully) a constructive dialogue thread. Even saying that, I'm know sure how constructive even I will be, but it's gotten to that point. I do not wish this to be a pure hate thread though, rather a professional opionins thread, but of course anyone is more than welcome to chime in.

Fingers crossed though that this doesn't go too far south before at least 10 replies.

I want to make clear that, 9/10 times, I side on ZOS on server/network/dev issues in this forum, with a good educated reason.

I have been a Systems Engineer, professionally, for 17 years. I have managed tens - hundreds of thousands of servers in many industries... financial, commercial, semiconductor, social, etc.... but not online gaming. Uptime is critical in any industry that I've worked in, be it for in-house services or externally-facing services. Being in this profession for so long has also made me privy to the goings-on of related fields such as development, networking, and security.

Periodic unplanned downtime to fix code-related bugs, like yesterdays for example, are to be expected and should even praised when a game-breaking flaw has been found. This thread has zero to do with instances like that. Could tonight's downtime be another example of this? Unfortunately we will most likely never know, but this thread is absolutely NOT just about tonight.... it's just the proverbial straw. On that note.... NO they should not release the technical details of the huge bug yesterday or any bug.

Any Internet presence worth anything will eventually get some sort of DDOS. Why DDOS someone when you won't get any response from it? That said, there are plenty of services out there to help protect against it, and plenty of things in-house that can be done to mitigate (NOT 100% prevent) the problem. I hate to say this part but... some people on here will be like "Oh well Gmail/Facebook/Yahoo doesn't go down for hours" etc etc. While their basis for the argument is generally flawed, their point is valid.

OK SO

I can't fathom how management in the server/network department(s) allow for such horrendously frequent downtimes. Sure ESO (and basically all MMO's) make it clear in their TOS that they do not guarantee availability, but does anyone expect downtime to be often and for so many minutes at a time? Technically they could be up for an hour a month and still fall under that clause, so let's be realistic here.... the bottom line demands a certain level of service availability to retain customers, especially for an exclusively online service. We all know about five 9's etc etc, but I'm not sure if this could even be a one 9.

Systems people, Network people, Devs, etc.... what do you think about all of this? I just can't wrap my head around it frankly. Anyone with first-hand gaming industry experience please do comment from your perspective.

I would have chimed in at the time but wasn't signed up to the forums so better late than never.

13 Years as a Community Manager on Star Wars Galaxies, Aion, Football Superstars and Star Wars: The Old Republic prior to leaving work to be a full time carer for my severely autistic daughter.

There is no typical setup so to speak some can get away with a very small server infastructure and others require individual servers for certain features it really depends on the playerbase size, complexity and optimisation.

FS was a relatively small game, 5k concurrent users 100k created accounts. We had a login server to communicate login which would signal to the front end and DB servers to pull up character selection, then there was the world server and from the world server you would go to the match server. The only server with any redundancy was the DB server as this is where all player info was stored.

We had a number of issues some due to a small team and poor optimisation eg memory leaks within the open world server because players when entered match were leaving behind a ghost remnant of themselves on the open world server, eventually after so many players ghosting to the server the open world server would crash which required a clear and reboot by one of our techs. Solution was to perform a daily 1 hour maintenance at 6am until code was fixed to stop the ghosting issue.


Another time with no warning all our servers went down and where down for 18 hrs, this was due to a digger going through the fiber lines coming out of the data centre whilst digging for a tube tunnel expansion in london.


SWG we had similar issues and had multiple DB servers, every player, item, shop, furniture was an individual item with its own unique serial number and as everything existed within a single world (no instanced housing etc) the DBs were a monstrosity, chunking was a major issue as planets despite open world where seperated into zones (chunks) some planets had 4 servers handling them, the busier planets that you could build houses and towns on would be up to 20 and everytime you crossed a server boundry (chunking) it would instigate a massive hit to the DB server to pull all info for everything within that zone.

Daily mainenance was the only solution for this and went on for the entire duration of my 4 year with Sony.

We once lost the servers for 13 hrs due to unpredictability by the playerbase, the night we launched the death watch bunker brought people back in their masses suddenly a chunk designed to take 5-10000 people was being flooded with 20-30k people the only solution was to bring more servers online and for the first time not have everyone within the same chunk with 2/3 running for this zone temporarily.

But we also had a number of out of our control issues, power cuts, pigeon infestations

SWTOR we didnt really have any issues and when you consider around 30 servers made up a game server it was a massive success a testament to the team and the dublin data centre that was initially set up for swtor but now handles alongisde a mirror site in holland all EA official servers, FIFA, Battelfields, Battlefront, SWTOR etc

It of course has the occassional blip but as these are highly instanced game worlds with a mass of redundancies in place they usually rectify themselves.


Of course being the community manager for these games was a barrel of laughs during unplanned server downtimes, the amount of abuse you would get via forums, twitter, facebook etc was comical. Even a great number to your personal facebook etc, I kept the best to one day publish some got just crazy though especially on swtor, we had in fact EA still do a "suicide" policy and your not allowed to use common sense so every threat of it had to be reported to local police.

Little Johnny - WAAAH servers down my afternoons ruined im going to kill myself.

Of course Johnny is just being a dramatic *** but as per policy we had to follow it up

Look up little johnnys data and location, use tool we had compiled on police, call police and notify of name address etc. Little Johnnys parents get a knock on the door.