More EU maintenance this morning?

Maphusail · September 2016

Does the price for ESO Plus include 4 days/month maintenance and unplanned maintenances?

tat2mano · September 2016

HtFde wrote: »

Legedric wrote: »

They don't dio maintenance on the server but i the data center, replacing switches, firewalls whatever central thing to permanently fix yesterday's issue (I assume). The probably do not have to work on the 3 servers individually.

Well, we started renting IT services outside of our company about 12 years ago - never, ever have we had an outage in a server center because they were changing switches, firewalls, routers and whatnot, Our lowest availability we offer to our customers is 99.5% and we simply couldn't fulfill this if simple maintenance jobs like those you mentioned would take the infrastructure down.

You either plan for continuous service and build your software accordingly or you have no clue what that is and start coding away. ZOS obviously has no clue.

Many people claim that maintenance in MMOs is a regular thing - I'd like to point out that Guild Wars 2, for example, even managed to roll their first expansion pack into production with 0 downtime. You get a message that a new build is available, you log off and back on - you download the new content as fast as your connection allows and - you're back into the game.

So the downtime there is individual (since you can decide when to log out to load the new build) and - except for a few cases when they patched a critical exploit - the windows in which you could remain online was between 45 minutes and 2 hours.

I've never experienced something like ESO offers with 4+ hours in which the service just isn't available. Poor software design, poor technical infrastructure. Having a dedicated server that needed downtime when you replaced components is a concept that was modern 10 years ago. Nowadays you have virtual servers and if you need more servers you just start more instances and you stop them when you no longer need them.

Of course, the datacenter hosts running those virtual servers can run into problems - all of our external providers have spare servers, though, and the longest downtime we had for single server (or instance) applications was less than an hour when the image was moved to a new ESX host. If you have an architecture that is using multiple servers already you may get a load peak and somewhat sluggish response until the new instance is online but you do not have a complete outage.

Thanks for elaborating about this,
there are way to many IT people out here on the forum that defend ZoS when servers are down trying to explain us simple people that dot understand IT stuff and make it sound like it is very normal for a MMO company to take the server offline so many times and for so long.

Chickenstein · September 2016

josefcifkaeb17_ESO wrote: »

The community here is amazing, no other mmorpg would get away with this much downtime and ZOS does not even appreciate that. Wonder how long it can last though... Thanks for ruining a potentionally amazing game!

Thing is, they probably wouldn't get away with this if people would would know how the internet, especially data centre/server infrastructures, actually work.

It's not as if there is some tiny guy swiping server intestines. I wouldn't know of a single professionally run data centre in Germany who would require a company to have a downtime of several hours, and I think I know quite a few. So this is not about data centre infrastructure, but about server maintenance IN a data centre (notice the difference) done by the team renting the space there (Zenimax or a local contractor). Still you would think that it's not necessary to take the whole thing offline when maintaining, but I guess thats an error deeply nested in the development of the whole software; some kind of mismanagement that can't be corrected at this point.

There is only one more or less valid reason for a downtime of several hours, and that would be server relocalization to some other locale (city/country), but besides the fact that this could be prepared in other ways (it's 2016, not 2001) they would have told us. Probably.

Edit: HtFde said more or less the same while I was typing, good going

Bonzodog01 · September 2016

The Datacentre that ZoS uses is not owned by or connected to Zenimax Media in any way shape or form. They are contracted by ZoS to provide space and bandwidth for the 3 EU Megaservers. That datacentre also provides services to dozens of other companies and games.

So Zenimax can only stand by and watch helplessly as the company that owns said datacentre decides to take it offline to do essential work on various areas of the centre. They haven't even told Zenimax what the maintenance actually is, but its likely connected to power problems or possibly re-organising the centre due to an expansion.

This is the 3rd or 4th Datacentre maintenance in the last 6 weeks, so something is definitely going on there.

Oh, and if you are on Xbox, Star Trek Online has just released, but you will need to switch your console to US language and Region to download it, then you can switch it back afterwards.

Drakoleon · September 2016

" ESO , a masterpiece in the worst hands"! I have said that already couple of times but...its true!
And fanboys take your tinfoil hats off, there is no conspiracy here. You don't do any good by trying to excuse zeni's worst behavior and incompetence towards customers and this great game.
I'm so very sorry cause i really love this game.

Chickenstein · September 2016

Bonzodog01 wrote: »

They haven't even told Zenimax what the maintenance actually is, but its likely connected to power problems or possibly re-organising the centre due to an expansion.

We're all just speculating of course (because of a no-info-policy by Zenimax, apparently), but how long do you think a data centre with this kind of problems would exist? I highly doubt that, especially since usually not all infrastructure is affected simultaneously (i.E. console servers != pc servers, etc., people staying logged in but login not working, etc.).

Chickenstein · September 2016

Oh look, the NA data centre is also being maintained!

Yeah, definitely data centre issues.

OneKhajiitCrimeWave · September 2016

Meanwhile at the EU Datacentre...

tat2mano · September 2016

Drakoleon wrote: »

" ESO , a masterpiece in the worst hands"! I have said that already couple of times but...its true!
And fanboys take your tinfoil hats off, there is no conspiracy here. You don't do any good by trying to excuse zeni's worst behavior and incompetence towards customers and this great game.
I'm so very sorry cause i really love this game.

Agreed,
just send an email to Bethesda if they please could buy the game back from ZoS and sell it to some company that actually cares about theyr customers.

Ofcourse i have no hope that it would have any succes....

Arioco396 · September 2016

maintenance for what?

two years playing with combat bug...

two years testing sets traits and champion points...less damage , less maximun magica or sta, les dmg, no dodge, less dmg absortion than other players with same race, traits, sets and champion points...

and now maintenance, maintenance and maintenance during day......

this is for all players:

CANCEL SUSCRIPTION !!!

sorry my bad english

tat2mano · September 2016

Chickenstein wrote: »

Bonzodog01 wrote: »

They haven't even told Zenimax what the maintenance actually is, but its likely connected to power problems or possibly re-organising the centre due to an expansion.

We're all just speculating of course (because of a no-info-policy by Zenimax, apparently), but how long do you think a data centre with this kind of problems would exist? I highly doubt that, especially since usually not all infrastructure is affected simultaneously (i.E. console servers != pc servers, etc., people staying logged in but login not working, etc.).

Maybe Gina Bruno will have some awesome stories to tell on the next eso live about how they got some injuries with the great company sporting matches they are doing atm,
she likes sharing stuff like that on eso live and spending a lot of time giggling with Jessica instead of telling us important things that we want and need to know in the only live show they have.

HtFde · September 2016

Chickenstein wrote: »

There is only one more or less valid reason for a downtime of several hours, and that would be server relocalization to some other locale (city/country), but besides the fact that this could be prepared in other ways (it's 2016, not 2001) they would have told us. Probably.

Nods, that's the most iffy thing that can happen as you *need* a frozen zone in which the "system to be moved" must not be altered and it depends mainly on the bandwidth between the old and the new location.

We have switched providers in the past and - with the exception of datawarehouses that have a daily update volume of 1 TB+ - you usually need about an hour downtime for this when you have to synch the new instance with the old instance and copy over the changes that were made since you created the backup/images with which the new servers were set up. Add in some DNS cache fun and you may be close to 2h where the service isn't available.

Another "valid" reason, of course, is when - for some reason - you cannot use virtual servers and have to change physical components but, as I said previously, (a) with some effort you can go virtual and become mainly independant of mainboard upgrades, etc. and (b) if you plan (!) on continuous service you have more than one server so even in a non-virtual environment you don't suffer from service outage if you take one server down.

Eweroun · September 2016

xheidelberg_castle_ruins_2.jpg.pagespeed.ic.Sn9vOj-S44.jpg

eso datacenter

raglau · September 2016

Legedric wrote: »

Thevampirenight wrote: »

Hopefully that eight hours will not be 12 hours but I would not be surprised if its a 12 hour maintenance.
For one they have three servers, or four plus the accounting store they have to do. So I suspect doing each one would take time. So yeah don't expect to be in game this morning.

They don't dio maintenance on the server but i the data center, replacing switches, firewalls whatever central thing to permanently fix yesterday's issue (I assume).

That's not correct for the following reasons:

The datacentre capacity will be leased from a professional datacentre services supplier. ESO is too small a service to warrant its own datacentre. A company specialising in leasing such a service will have highly resilient, redundant infrastructure such as switches, routers, firewalls etc. No prospective customer would ever use their services otherwise. Such redundancy allows for failure and failover of any defective parts, and replacement of those defective parts will result in zero downtime due to this redundancy.

The application itself, ESO, very likely has limited resilience from what we're seeing. Either some or all components are not capable of failover, and this is always the result of a cost vs benefit architectural decision. It is possible to make all services resilient, but the capital outlay and total cost of ownership rises. ZOS obviously feel that the increased cost is not offset by increased revenue, or the loss in revenue from offering poor service is not that high.

Basically, they don't feel their customer base is worth investing in ESO's availability. Make of that what you will.

Bonzodog01 · September 2016

So, shortly after my last post, I discovered a thread where a picture of the datacentre and the name of the CDN were provided. Its Level 3 Communications Inc, the 3rd largest CDN in the world, who provide services to Netflix, Yahoo, Facebook, Google, and numerous other companies. Just an interesting tidbit of info there.

HtFde · September 2016

Chickenstein wrote: »

Bonzodog01 wrote: »

They haven't even told Zenimax what the maintenance actually is, but its likely connected to power problems or possibly re-organising the centre due to an expansion.

We're all just speculating of course (because of a no-info-policy by Zenimax, apparently), but how long do you think a data centre with this kind of problems would exist? I highly doubt that, especially since usually not all infrastructure is affected simultaneously (i.E. console servers != pc servers, etc., people staying logged in but login not working, etc.).

Coughs ... long

Nowadays accountants are running most companies and for them a penny saved is worth a lot of inconvenience their customers might experience.

If you take a look at some of the replies here you see that there *is* a crowd of people who are actually buying it.

So maybe this datacenter - which apparently seems to have one electricity plug that's needed twice a week for the cleaning lady and her vacuum - will have a bright and secure future.

josefcifkaeb17_ESO · September 2016

@Chickenstein I know, just look at my previous comment #65 ^-^

Also by observing this topic further, I think maybe this time ZOS won't get away with it as easily, seems like people are finally getting fed up with it.

Chickenstein · September 2016

HtFde wrote: »

Another "valid" reason, of course, is when - for some reason - you cannot use virtual servers and have to change physical components but, as I said previously, (a) with some effort you can go virtual and become mainly independant of mainboard upgrades, etc. and (b) if you plan (!) on continuous service you have more than one server so even in a non-virtual environment you don't suffer from service outage if you take one server down.

Well, it's the EU PC megaserverS they're sometimes talking about, so I guess it is not one single machine (which would be, all bad management aside, total madness anyway), so I guess they could pull out single blades/instances/whatever. Even so, what components could you think of that would require a downtime of even one hour?

The infrastructures I've seen (some by bigger fishes than Zenimax) all rely on some sort of proprietary distributed system, by the way, all of which had fallbacks and backups so you could isolate single components. You'd expect the same from a company with as much experience in the market as Zenimax.

Arthur_Spoonfondle · September 2016

Trion patch Rift pretty much every week. Mostly the downtime is only around an hour but, if there is a problem which causes much longer downtime, especially as we see this week in ESO, there is some compensation for those who sub.

LaiTash · September 2016

Oh, i've just read the announcement.

EIGHT HOURS AND MAYBE MORE? REALLY?! HOW IN THE WORLD IS THIS JUSTIFIED?

And i was thinking of eso+ next week. No thanks. Just buying the mmo with worst service EVER (trust someone who's been playing mmo since the MUD era) was more then enough for me, i'll wait until i finish the vanilla content first. Which, considering the uptime percentage here, isn't going to happen anytime soon. At least maybe bethesda will buy the game back from zenimax by then.

tat2mano · September 2016

josefcifkaeb17_ESO wrote: »

@Chickenstein I know, just look at my previous comment #65 ^-^

Also by observing this topic further, I think maybe this time ZOS won't get away with it as easily, seems like people are finally getting fed up with it.

I can asure you i am fed up with it,my eso+ is ending in 3 days and i am not going to get another untill ZoS shows me they can keep the server alive with no more then 16 hours of total downtime in 30 days,
if they exceed it its not worth my money anymore,
i rather buy an extra account to act as my bank instead of paying every 30 days just for my materials to go in another bag.

Suter1972 · September 2016

I wonder how many of the total EU populace are at home playing during a week day and how many are out at work/college/school etc. From my limited experience the hubs are rammed in the eveings but dead in the day times.

Maintenance while im at work, all ready for me to jump into on an evening suits me very (selfishly) well

Rosveen · September 2016

In case anyone ever wondered, which they didn't, this is one of the two reasons why I'm not subscribed (I'm not going to go off-topic with the other reason).

Three unplayable days in the row. No, I'm not going to say mornings, because the maintenance takes half a day and not all of us work 9-5, especially not in the summer. This isn't the first time it's happening. We had several unplayable evenings in August, I also distinctly rememeber an identical 3-days-in-a-row situation. How long are these "datacenter issues" going to last, Zenimax?

Chickenstein wrote: »

Oh look, the NA data centre is also being maintained!

Yeah, definitely data centre issues.

That's ordinary planned PS4 maintenance. But am I going crazy, or are they taking down the account system for the second time this week?

tat2mano · September 2016

LaiTash wrote: »

Oh, i've just read the announcement.

EIGHT HOURS AND MAYBE MORE? REALLY?! HOW IN THE WORLD IS THIS JUSTIFIED?

And i was thinking of eso+ next week. No thanks. Just buying the mmo with worst service EVER (trust someone who's been playing mmo since the MUD era) was more then enough for me, i'll wait until i finish the vanilla content first. Which, considering the uptime percentage here, isn't going to happen anytime soon. At least maybe bethesda will buy the game back from zenimax by then.

In fact i send an email to Bethesda asking exact this,
dont think it will have any succes tho.

But Bethesda should really act up imo,
ZoS is giving the much beloved Elder Scrolls series a bad name.

Am sure that there are companies that would take much better care of the game and the customers.

Chickenstein · September 2016

Rosveen wrote: »

That's ordinary planned PS4 maintenance.

Oh, my bad. I was not using the twitter machine correctly.

HtFde · September 2016

Bonzodog01 wrote: »

The Datacentre that ZoS uses is not owned by or connected to Zenimax Media in any way shape or form. They are contracted by ZoS to provide space and bandwidth for the 3 EU Megaservers. That datacentre also provides services to dozens of other companies and games.

So Zenimax can only stand by and watch helplessly as the company that owns said datacentre decides to take it offline to do essential work on various areas of the centre. They haven't even told Zenimax what the maintenance actually is, but its likely connected to power problems or possibly re-organising the centre due to an expansion.

This is the 3rd or 4th Datacentre maintenance in the last 6 weeks, so something is definitely going on there.

That's BS!

If this is a professional datacenter there's a contract and the contract usually specifies:

(a) overall availability of the rented sevices - either annual or monthly
(b) independent of (a) a maximum continuous downtime
(c) a minimum of days between outages

Should the datacenter be in violation of the contract there's usually a hefty fine associated with this breach - trust me, I handle those contracts on a daily basis - with our providers and with our customers.

Most datacenters that I know of in Frankfurt offer the same type of service for roughly the same fees. You do have exceptions but then ... if I decide to buy a 20 year old Honda Civic with 3 million miles to it I cannot really complain if the thing breaks down on me. can I?

"watch helplessly" ... seriously ... they have a CONTRACT. The only people watching helplessly are the players because the terms of service state that the game might be available or not, may have a downtime of minutes, hours even days - a contract between a datacenter and its customers is not that vague and one-sided, believe me.

ESO became subscription based because they claimed this was the only way they were able "to offer the best service" to the playerbase. If this is "the best" they can think of ... I shudder to think about what they would call "bad".

Liptrot · September 2016

Suter1972 wrote: »

I wonder how many of the total EU populace are at home playing during a week day and how many are out at work/college/school etc. From my limited experience the hubs are rammed in the eveings but dead in the day times.

Maintenance while im at work, all ready for me to jump into on an evening suits me very (selfishly) well

I play in the daytime on ps4 eu and find it's quite busy. I get what your saying most people play during the evenings but alot of people can't like me who work night shift.

Arthur_Spoonfondle · September 2016

tat2mano wrote: »

But Bethesda should really act up imo,
ZoS is giving the much beloved Elder Scrolls series a bad name.

Am sure that there are companies that would take much better care of the game and the customers.

But, look at all the new stuff in the Crown Store...

As long as there enough new players starting the game, replacing some of the disgruntled ones and spending money in the Crown Store, Zenimax won't give a monkey's.

Tandor · September 2016

I'm normally very quick to defend ZOS on most issues, because I actually agree with 99% of the way they are developing and running the game. I'm no fanboy, however, as there certainly are things about ESO that I regularly criticise most especially the wretched apology for a trading system!

In spite of my usual support for the company, however, I do have to agree that the situation with regard to the EU servers has now got out of hand, partly because of the frequency of the downtime but especially because of its duration, and while the problems may or may not be within the control of ZOS the issue of communication with their playerbase most definitely is within their control, and it's high time a top-level official announcement was made here on the official forums and in a pop-up window on the launcher explaining exactly what the problems have been these past few weeks and what the company is doing about them so as to ensure a more reliable service from hereon.

For me it isn't a question of not subscribing, or even not playing, it's simply a question of the company acknowledging that the present service standards are not acceptable and informing us what they're doing about it.

@ZOS_KaiSchober , @ZOS_GinaBruno This is the message I would ask you both to pass up the line please

!

Delpi · September 2016

Suter1972 wrote: »

I wonder how many of the total EU populace are at home playing during a week day and how many are out at work/college/school etc. From my limited experience the hubs are rammed in the eveings but dead in the day times.

Maintenance while im at work, all ready for me to jump into on an evening suits me very (selfishly) well

I work on afternoons so not everybody are at office on mornings :P