Update on PC NA & EU Issues [Updated 10/27 @ 8:40pm]

lRaiden

Besides extending the witches festival, as to which you shouldn't but you MUST, you MUST also extend ESO+ subscribers for the period the servers started to behave like this. It started on Saturday around 1am GMT+1

Marto

PaintsOfDust wrote: »

Also, please stop with the never-ending "some players experienced" stuff... its ALL PLAYERS during ALL peak-times.

How do you know it was all players? Maybe you're right, maybe you are not.
This is why ZOS keeps saying "we're investigating" each time an issue shows up. Because assuming things and making wild, overarching guesses just leads to more issues.

PaintsOfDust wrote: »

its like you purposely try to hide, that the servers were designed for lower player-numbers. which is okay... every server needs to be updated or re-stocked at some point, especially for a growing game like ESO. there is no shame in admitting, that you need to add more server capacity, and that you are not up to the job at the moment.

Except that's not how servers work.
Modern servers are designed to adapt, shrink, expand, and change their behavior and properties, depending the load, timeframes, etc.

Saying "The EU server has a lower capacity" is a gross oversimplification. The issue might not even be related to capacity in the way you expect. It's not that the servers are full. It's that they fill too quickly, leaving it not enough time to expand and adapt.

Some aspects of the game also behave differently based on the amount of players.
For example, when the server is near full, matchmaking will try to find you some Battleground opponents near your level.
But at 3 AM? The matchmaking system will probably stop caring about skill levels, and just match whomever it can get.

The issue is complicated. This is why ZOS can't immediately communicate and say what's happening. Because ZOS are human, and often times, they do not know what's happening.

C0RTEX4

PaintsOfDust wrote: »

I appreciate the communication, but i dislike that no official member of the Tech-Staff replied and no specific plan is given for improvements, etc. This is beyond a community-communication-issue now...
for the last week people have tried to meet up with their friends, do in-game stuff, be social, etc. for an event you hosted. To play this game with friends is part of the game. It's how you advertise this game, it's how you wish to be represented officially.
Despite that, i have tried literally every day for the past 3-5 days to get something going with friends or to play this game together with them... all attempts failed, cause your server has no uptime that could be recorded the last 3 days for the 6pm-9pm CEST timeslot as far as i know. And that is simply not acceptable. The Statement now is too late and too little with almost no direct consequences, yet you still take money from people happily every day. Maybe or maybe not people should pay for their eso+ subscriptions, lets "consider these options" for a bit... how about you re-gift a month worth of eso+ subs ? Also, please stop with the never-ending "some players experienced" stuff... its ALL PLAYERS during ALL peak-times. if you want to play you were restricted to the 1-3am-timeslot or the 11am-3pm timeslot. this is not normal.
its like you purposely try to hide, that the servers were designed for lower player-numbers. which is okay... every server needs to be updated or re-stocked at some point, especially for a growing game like ESO. there is no shame in admitting, that you need to add more server capacity, and that you are not up to the job at the moment. but you need to say so ! .... i know GINA is not responsible for the tech-departments short-comings, and maybe even the tech-people are horribly understaffed, but change needs to happen. please dont pretend, as if only "some players" experience an "issue". its ALL the players at ALL peak-times for several MONTHS now. you havent been able to host a single event without issues in the last year and you still pretend as if it was an issue on the optimization-side. it is simply not true. and its unfair to do so to your biggest player base.
i have given up to convince my friends to play this game, it is pointless and frankly, i dont want them to stare at another loading screen for several hours while i try to explain to them, why it is, that neither your social media nor your official website isnt able to post an official statement for atleast 40 minutes before the eventual "some players experience...." comes up. you will loose your "new" playerbase that way, and to be honest, i think you already did... over a few puny investment into some server tech... well played, good game!

I agree with you buddy, for the 5+ years of this game we have all suffered and this festival is the icing on the cake, they need to redeem themselves before us old and new players leave, its not fun coming home from work to stare at load screens, I agree that all of us from the EU server would appreciate something like a 1 month re gift of subs or 1 month's subs worth of crowns or crates, with at least 1 legendary mount or something, non of that usual rubbish you get in the crates, or a 1 time, 1 item in the game of our choice, because In my mind and I think all would agree, We deserve an apology...

Azrhyel

...we deserve a Raven pet as apology, yes we do!

kyle.wilson

the issue is moving to PC/NA right now.

jazsper77

Omg really , I can’t believe a generic all words and no action statement from Gina (ZOS) the same statement that’s used every time and your all HAPPY now. Lmfao

It’s Called ACTION not words , how long has this been going on . I was taking PITY with the EU. Guys but now forget it. When the next event happens just please don’t post the EMD OF THE WORLD threads when a general statement of the same stuff makes you all happy.

Jeth

kyle.wilson wrote: »

the issue is moving to PC/NA right now.

They're just making it "fair". Can't have NA working if EU is broken. Share the misery.

jazsper77

Nemesis7884 wrote: »

stop praising them for doing the absolute bar minimum

You nailed it.

kyle.wilson

Jeth wrote: »

kyle.wilson wrote: »

the issue is moving to PC/NA right now.

They're just making it "fair". Can't have NA working if EU is broken. Share the misery.

Maybe its time zos fires their entire programming team and replaces them with actual engineers from India, instead of D students willing to live in Maryland.

RefLiberty

kyle.wilson wrote: »

the issue is moving to PC/NA right now.

Kyle bro, we are with you. Less greediness, more shards and V8's.

Pyr0xyrecuprotite

kyle.wilson wrote: »

the issue is moving to PC/NA right now.

Yup, confirmed, getting much longer load screens on PC/NA if rezoning or logging in to a character at the moment, especially busy zones. (I have played since beta, know what's normal and what is problematic, this ain't normal). I was in nMA for a while so can't say exactly when it started, but definitely a problem since 11pm EDT (Saturday, Oct 26). People in Southern Elsweyr are saying it's faster to run to get skyshards than to use wayshrines.
I'm having a long loadscreen at the moment just trying to port into a guild hall (primary house of a guild member, Grand Topal Hideaway), so it's not just the zone population affecting the servers.

ForzaRammer

LFG tool was fine b4 1 tam, maybe you guys should revert all the way back to pre 1 tam

Gamer1986PAN

i was able to run a normal fungal. yesterday i had about 15 - 20 minutes loadingscreen to enter the dungeon. a few minutes ago the whole dungeon took about 30% of that loading screen time - from entering to leaving. i hope its not only because there a not much players online at the moment ^^.

Ramber

No test system? just a pts that no one uses?

Chicharron

kyle.wilson wrote: »

Jeth wrote: »

kyle.wilson wrote: »

the issue is moving to PC/NA right now.

They're just making it "fair". Can't have NA working if EU is broken. Share the misery.

Maybe its time zos fires their entire programming team and replaces them with actual engineers from India, instead of D students willing to live in Maryland.

But students are paid with Crowns, a real engineer is paid with real money.

I don't think ZOS likes that.

PaintsOfDust

Marto wrote: »

How do you know it was all players? Maybe you're right, maybe you are not.

let me rephrase... i "assume" its all players, because >95% of the sample size (aka me and my friends) can't login. This is how every empirical study works, this is how the industry works. Generally, i was able to re-produce the non-functionality of the servers amongst different platforms at different times. so yes. maybe im right, "maybe" im not. the probability im not is about alpha <= 5%. thats usually scientific standard and its enough for me. if you could stop trolling serious comments with serious implications now, id appreciate it

Moonsorrow

Thanks for the update.

I wish the upcoming event days won`t have such problems anymore. But can understand the reason, there has been so much players grinding their event Skull boxes everywhere, some dailies were like Cyrodiil primetime zergfests. Server hamsters need stronger stimulants.

And yes, couple days of extra event would be nice to get, have missed plenty of playtime due to these issues. Not gonna do entitled baby tantrum over it, but.. baby needs the new golden event recipes *kitten eyes* :

Hymzir

Well, whatever... Of course it was some issue with the way the system has been implemented, obviously it was due to whatever got changed with the last update, and obviously ZOS will try to fix it. I don't see much reason in celebrating a company doing what it obviously has to do to fix an issue with the product they are peddling.

The real damage is done anyway, people are pissed, people are miffed, another day wasted, another day of subscription down the drain, and it's not as if this is the first time something like this has happened either. The reliability of the service ZOS is providing has a dubious reputation and things like this will not make it any better, or nurture customer confidence.

The lesson for ZOS to learn here is, that they really need to amp up their game and make sure that disasters like this won't happen again. I mean once or twice would be perfectly understandable, but when every single time there is an event, the game just breaks down, signals something about the ability of ZOS to provide the things they claim to offer.

You guys at ZOS really need to make sure things like this won't happen again, and that maybe you need to allocate a bit more resources into testing things before you push the changes through. Or at the very frigging least, do not push major updates out couple of days before a big event you know will see a large increase in daily traffic. That's just common sense.

AS for what else should be done... Well, it really wouldn't hurt to offer some sweets to the angry crowd congregating at your gates brandishing pitchforks and torches...

Extending the Witches festival is obviously like the bare minimum you need to do, but that will not placate the restless natives.

Distributing some extra glass beads or something is strongly advised.

Look at it from the end user perspective - Satruday is prime gaming time. People set out time to do a bunch of stuffs during the weekends they don't have time for during weekdays. I for example can't really see myself having the time or opportunity to do trials except during weekends when there are a lot more people online,

That was not possible today. I mean I lost an opportunity to get a crucial Dremora coffer containing the only motif pages I need. I got two full sets for my primary and secondary crafters last year, and was planning on getting the missing pages for my tertiary and quaternary crafters this year. Now I an not so sure I can do that except by buying the missing pages from other players. Or by going though the extra hassle needed to get a trial group going during a weekday.

Now all of that is, of course, meaningless in the bigger scope of things, and am sure I will still manage to meet my goals for the event even with the lost evenings taken into account, but it will induce a tad more stress and urgency in my mind, and make the whole event feel that much less entertaining and rewarding. Which is already soured by all these technical issues we are having.

And am pretty sure I am not the only one feeling the burn, feeling the disappointment, and who has this nagging growing resentment building inside then towards the whole darn game.

So... whatever you do, remember that doing the bare minimum (i.e. fixing the servers and adding a few more days to the event) will not really make all the unhappiness and anger and resentment go away.. At the end of the day, the truth of the matter is, that you went and *** it up, and people are pissed about the state of things. If you want to retain customer goodwill, then I strongly urge you to do something extra.

Like, I dunno, throw us some Crown Crates, send as a day's worth of dremora boxes in the mail, or something - but don't just expect that people will shrug and be okay with things constantly being borked in this game.

Anyway... that's all I gotta say, time to go to bed I think.

solasub

Marto wrote: »

How do you know it was all players? Maybe you're right, maybe you are not.

Well absolutely 100% of the ppl I met in game those days met those issues ; all my friend list (100/ 100 ppl) ; all my guilds ; all the friends of my friends ; all the random people I met by doing MMO thingies -or should I tell, trying to do them (random normal, pledges, hunting dragons, sharing dailies, trading motifs... ) ; all the people in zone chat ; all the raiding groups I know had to cancel / delay their events ; etc, etc ; all the Discords I am in were dealing about the issues ; etc, etc ; so yes, that's quite A BIG amount of people at this point.

coercer02neb18_ESO

Server up and running. First BG game - 15 min loading screen. This just isnt fun.

JamuThatsWho

While the communication is appreciated, something drastic has to be done any this situation.

No other online game I've ever played has had problems of this magnitude or frequency. Maybe I'm just lucky, but I expect a certain level of technical competence, especially from such a big company.

daedalusAI

ZOS_GinaBruno wrote: »

Hi everyone,

We'd like to give you an update about the instability some of you are seeing, particularly on the PC EU megaserver, which includes issues such as long loading times, inability to log in, and other "servers are overloaded" experiences. As you likely have surmised, this is related to the Update 24 launch, and has been exacerbated by the Witches Festival event currently live which is resulting in a lot of players logging on to participate.

We thought we had fixed the problems that have plagued the last few big in-game events by introducing an updated LFG system, some server optimizations, etc., but obviously it wasn't enough. The updated LFG system appears to be working as it should, but it has exposed another underlying issue that is causing certain parts of our servers - especially our database clusters - to become overloaded and eventually unresponsive. This is why we put a cap on the number of concurrent players that can log in last Friday, and then published the patch this morning trying to alleviate that issue.

At this time, we have another hotfix that we plan to deploy tonight (around 10:00pm EDT) that we believe will allow everything to run as normal, but frankly, we'll wait to see what it does before making any more promises. This hotfix will require a small amount of downtime for PC EU.

We totally understand it's not fair that PC EU is hit by these types of problems far more often than any of the other realms. This is because there are more concurrent users there during peak times than on any other realm. The problem you all have been experiencing only shows up at levels of concurrency seen on PC EU.

We are already discussing additional options - besides directing all of our resources to identify and fix this issue, of course - which include extending the Witches Festival for a couple days. We'll keep you posted on that as soon as we get a handle on the underlying cause of the instability.

As always, thank you for your patience. We don't like this any more than you do, and are doing everything we can to get everything back to normal as quickly as possible.

Pathetic and laughable.

Those issues are occurring frequently, so I heavily doubt it's just due to U24.

"We thought we had fixed the problems..." reads to someone doing QA professionally that you didn't test a thing and just pushed U24 and the hotfix yesterday to live server anyway.

I mean how utterly incompetent do you have to be that on live servers it's the first time you "...exposed underlying issues..."?

2 hotfixes in the span of 2 days to fix login/connection issues you repeatedly told "they're really fixed now" just screams clueless and incompetent.

"PC EU has more player during peak times, so instead of upgrading the server cluster/structure to guarantee a pleasant playtime during peak hours we do nothing year round and just deploy hotfix after hotfix".

Did anyone notice that connotation?
It's not "Of course we extend the event, that's the normal thing do to" but "we discuss extending the event".

And to end it nothing but pure mockery: "Thank you for your patience as yet again PC EU is struggling/unplayable during peak times and we really, really are doing "everything we can" this time to get everything back to normal"

What normal: PC EU normal in terms struggling/unplayable servers during peak hours - or normal as in the best server structure a company with integrity and the drive to publish a quality product will opt for?

b92303008rwb17_ESO

Like I said in another complaint post a couple of weeks ago. It has been for like 4 or 5 years and we are still struggling with basic connection issues. ZOS is simply pathetic.

halucin0g3n

ZOS_GinaBruno wrote: »

This is why we put a cap on the number of concurrent players that can log in last Friday, and then published the patch this morning trying to alleviate that issue.

At this time, we have another hotfix that we plan to deploy tonight (around 10:00pm EDT) that we believe will allow everything to run as normal, but frankly, we'll wait to see what it does before making any more promises.

We are already discussing additional options - besides directing all of our resources to identify and fix this issue, of course - which include extending the Witches Festival for a couple days.

This made my dog puke. So your fixes basically mean you put a cap on the number of concurrent players, restricting players to use the service they pay for.

You still need to wait to see what your hotfixes are doing? You don't know? Who makes them? How come you need to wait to see what it does?

Extending this witches event is not debatable, extend it for sure, it's not a favor you're making, it's an obligation of common sense.

Get your [snip] together.

[Edit for minor profanity]

Faulgor

ZOS_GinaBruno wrote: »

We totally understand it's not fair that PC EU is hit by these types of problems far more often than any of the other realms. This is because there are more concurrent users there during peak times than on any other realm. The problem you all have been experiencing only shows up at levels of concurrency seen on PC EU

That's really interesting but not suprising, considering most EU players are concentrated in 1-2 neighboring time zones compared to 4+ across NA.

Itzmichi

Lots of generic words, making it plan obvious that apperatly some people don't even know what they are coding and how to handle their infrastructures and coding.

How is that acceptable?

KyraCROgnon

looks like they broke PC NA server, so now we can no longer claim PC EU is working differently ?

Gythral

lRaiden wrote: »

Besides extending the witches festival, as to which you shouldn't but you MUST, you MUST also extend ESO+ subscribers for the period the servers started to behave like this. It started on Saturday around 1am GMT+1

September or October 2018 more like the start date, and PC/EU should have a free ESO+ until the hardware/software issues are fixed (and verified by users or a not related to Zenimax third party)

ZOS_GinaBruno

Update: To ensure that the game can run as smoothly as possible, we are going to disable the Activity Finder until we determine the root cause of the issues that are impacting game performance. As stated before, we’ll look into extending the Witches Festival, but first we need to identify and fix the problems that are impacting player experience.

For this change to take place, we'll need to reboot both PC NA and EU to stabilize everything, then turn off the Activity Finder; maintenance for this will begin around 3:30am EDT and should take an hour or two. Note that you'll still be able to form a group manually, but you will not be able to queue for Battlegrounds while the system is disabled.

We'll have another update for everyone on the situation later today (Sunday). Thanks again for your patience as we get to the bottom of this.

Darkmage1337

ZOS_GinaBruno wrote: »

Update: To ensure that the game can run as smoothly as possible, we are going to disable the Activity Finder until we determine the root cause of the issues that are impacting game performance. As stated before, we’ll look into extending the Witches Festival, but first we need to identify and fix the problems that are impacting player experience.

For this change to take place, we'll need to reboot both PC NA and EU to stabilize everything, then turn off the Activity Finder; maintenance for this will begin around 3:30am EDT and should take an hour or two. Note that you'll still be able to form a group manually, but you will not be able to queue for Battlegrounds while the system is disabled.

We'll have another update for everyone on the situation later today (Sunday). Thanks again for your patience as we get to the bottom of this.

Hallelujah! Thanks for the update.
Any news is better than no news.