Update on PC NA & EU Issues [Updated 10/27 @ 8:40pm]

ZOS_GinaBruno

Hi everyone,

We'd like to give you an update about the instability some of you are seeing, particularly on the PC EU megaserver, which includes issues such as long loading times, inability to log in, and other "servers are overloaded" experiences. As you likely have surmised, this is related to the Update 24 launch, and has been exacerbated by the Witches Festival event currently live which is resulting in a lot of players logging on to participate.

We thought we had fixed the problems that have plagued the last few big in-game events by introducing an updated LFG system, some server optimizations, etc., but obviously it wasn't enough. The updated LFG system appears to be working as it should, but it has exposed another underlying issue that is causing certain parts of our servers - especially our database clusters - to become overloaded and eventually unresponsive. This is why we put a cap on the number of concurrent players that can log in last Friday, and then published the patch this morning trying to alleviate that issue.

At this time, we have another hotfix that we plan to deploy tonight (around 10:00pm EDT) that we believe will allow everything to run as normal, but frankly, we'll wait to see what it does before making any more promises. This hotfix will require a small amount of downtime for PC EU.

We totally understand it's not fair that PC EU is hit by these types of problems far more often than any of the other realms. This is because there are more concurrent users there during peak times than on any other realm. The problem you all have been experiencing only shows up at levels of concurrency seen on PC EU.

We are already discussing additional options - besides directing all of our resources to identify and fix this issue, of course - which include extending the Witches Festival for a couple days. We'll keep you posted on that as soon as we get a handle on the underlying cause of the instability.

As always, thank you for your patience. We don't like this any more than you do, and are doing everything we can to get everything back to normal as quickly as possible.

MaleAmazon

I hope you get it running. While it is frustrating that this happens so frequently during events, I believe you do your best.

MartiniDaniels

Thanks for the update! Event prolongation will be nice and also will be a good test of servers for next weekend.

scorpius2k1

THANK YOU for the update Gina, excellent post and communication - even this late in the weekend and evening. I truly hope things will turn around, and all these issues that are starting to surface will continue to be fixed and make this game what it can and *should* be. Your community, even though we rage like crazy on here...just know we truly appreciate you and everyone addressing the issues.

All the best. Moving forward.

JamieAubrey

Thanks for the update

Bexy

Thank you. Now I can sleep in peace.

Jack-0

That's more like it! Most of us* appreciate additional info, we like it when you acknowledge performance isn't what it should be. What some of us* would like to know is why all this keeps happening - as you have now known for months that EU PC prime time causes critical server load. Why can't you just add more servers to the farm or more resources to the existing boxes? What do you spend per month on maintaining this server? How much are you making from ESO+ subs in this region each month?

Kind regards,

A lot of your dedicated fans*

*Because there's always some tool that says 'hey, not me, I love spending hours just trying to log in, everything is fine.'

logarifmik

Thank you for keeping us posted, comrade Bruno.

StarMightyMaster

I guess you know this but the activity finder is not working atm sometime you can que sometime you can't but when you do que you can't ready up

BlueViolet

Thanks for keeping us updated. Appreciate the additional info!

Chicharron

Thank You For Your Patience Online

xenowarrior92eb17_ESO

Now I know this isn't nice and tbf it sux...but can people think before coming here to cry? eso its a good game and yes for our money this shouldn't happen...we shouldn't be punished for stuff that should had been fixed years ago...but here we are and I assume 99% of the playerbase have no clue how "easy" is to run an mmo especially one like eso...by all means if you complain so much that the game sux...good news...we dont apply Coldharbour soul binding to the players...so go wherever you want or be a man and deal with it like grown ups...this isn't WoW...nor zenimax its a small indie studio...so yeah thx for info and I hope you get things done the way you are expecting them to...meanwhile we shall resume playing Loadingscreen Simulator: Online.

Cirantille

Thank you, now this sounds like some nice explanation.

Also we will appreciate the extension.

Dusk_Coven

Intermittent communication (which is understandable if they don't actually have new info to report) is better than no communication. Which is what I used to get from NWO when I couldn't log in for the better part of a day and there wasn't maintenance scheduled.

That said, this round is pretty scary. I hope we don't get a total crash next year when they ask us to re-download everything. Push back the upgrade schedule if you need to but please test it thoroughly before going live.

ZOS_GinaBruno wrote: »

We totally understand it's not fair that PC EU is hit by these types of problems far more often than any of the other realms. This is because there are more concurrent users there during peak times than on any other realm.

Thanks for this tidbit <.< Now EU's gonna fly it in our face every time the NA minority asks for something <.< :P

GaianSpirit

Nice with some honest and extensive communication, it shows you guys do care. Please keep these kinds of posts coming, it goes a long way to deal with the frustration that most of the players are feeling.

People don't like downtime, but all the downtime in the world is better than playing a game that is broken, better to do what is required to fix it so we can eventually play the game with a smooth experience

Firstclaw

Thanks a lot!

VerboseQuips

Thank you for communicating about it.

I hope you can find ways to alleviate the issues.

Ganjia

wow , an honest post , i actually cant be mad !

ehaque

we want free stuff :P

baratron

Any idea how long the hotfix will take? In other words, should I go to bed because it's 2 am, or stay up for a bit to get my Event Tickets on EU?

Kalixte

I didn't forget and still want my stamina whip though.

C0RTEX4

ZOS_GinaBruno wrote: »

Hi everyone,

We'd like to give you an update about the instability some of you are seeing, particularly on the PC EU megaserver, which includes issues such as long loading times, inability to log in, and other "servers are overloaded" experiences. As you likely have surmised, this is related to the Update 24 launch, and has been exacerbated by the Witches Festival event currently live which is resulting in a lot of players logging on to participate.

We thought we had fixed the problems that have plagued the last few big in-game events by introducing an updated LFG system, some server optimizations, etc., but obviously it wasn't enough. The updated LFG system appears to be working as it should, but it has exposed another underlying issue that is causing certain parts of our servers - especially our database clusters - to become overloaded and eventually unresponsive. This is why we put a cap on the number of concurrent players that can log in last Friday, and then published the patch this morning trying to alleviate that issue.

At this time, we have another hotfix that we plan to deploy tonight (around 10:00pm EDT) that we believe will allow everything to run as normal, but frankly, we'll wait to see what it does before making any more promises. This hotfix will require a small amount of downtime for PC EU.

We totally understand it's not fair that PC EU is hit by these types of problems far more often than any of the other realms. This is because there are more concurrent users there during peak times than on any other realm. The problem you all have been experiencing only shows up at levels of concurrency seen on PC EU.

We are already discussing additional options - besides directing all of our resources to identify and fix this issue, of course - which include extending the Witches Festival for a couple days. We'll keep you posted on that as soon as we get a handle on the underlying cause of the instability.

As always, thank you for your patience. We don't like this any more than you do, and are doing everything we can to get everything back to normal as quickly as possible.

And that's all we wanted, a little friggin insight as to what you are doing... about time...

stefan.gustavsonb16_ESO

Timely, open and honest communication, acknowledging the problem, telling us what you think the problem is, in general but technical enough terms to instill confidence that you have a plan, and giving us reasonable hope for some improvement soon. Thank you! I am a patient person, but even I get into a bad mood from being kept in the dark. Good night, and good luck!

Chouettelfique

Thank you for your explanation and your honesty ~

PaintsOfDust

I appreciate the communication, but i dislike that no official member of the Tech-Staff replied and no specific plan is given for improvements, etc. This is beyond a community-communication-issue now...
for the last week people have tried to meet up with their friends, do in-game stuff, be social, etc. for an event you hosted. To play this game with friends is part of the game. It's how you advertise this game, it's how you wish to be represented officially.
Despite that, i have tried literally every day for the past 3-5 days to get something going with friends or to play this game together with them... all attempts failed, cause your server has no uptime that could be recorded the last 3 days for the 6pm-9pm CEST timeslot as far as i know. And that is simply not acceptable. The Statement now is too late and too little with almost no direct consequences, yet you still take money from people happily every day. Maybe or maybe not people should pay for their eso+ subscriptions, lets "consider these options" for a bit... how about you re-gift a month worth of eso+ subs ? Also, please stop with the never-ending "some players experienced" stuff... its ALL PLAYERS during ALL peak-times. if you want to play you were restricted to the 1-3am-timeslot or the 11am-3pm timeslot. this is not normal.
its like you purposely try to hide, that the servers were designed for lower player-numbers. which is okay... every server needs to be updated or re-stocked at some point, especially for a growing game like ESO. there is no shame in admitting, that you need to add more server capacity, and that you are not up to the job at the moment. but you need to say so ! .... i know GINA is not responsible for the tech-departments short-comings, and maybe even the tech-people are horribly understaffed, but change needs to happen. please dont pretend, as if only "some players" experience an "issue". its ALL the players at ALL peak-times for several MONTHS now. you havent been able to host a single event without issues in the last year and you still pretend as if it was an issue on the optimization-side. it is simply not true. and its unfair to do so to your biggest player base.
i have given up to convince my friends to play this game, it is pointless and frankly, i dont want them to stare at another loading screen for several hours while i try to explain to them, why it is, that neither your social media nor your official website isnt able to post an official statement for atleast 40 minutes before the eventual "some players experience...." comes up. you will loose your "new" playerbase that way, and to be honest, i think you already did... over a few puny investment into some server tech... well played, good game!

Michae

I'm all for extending the Witches Festival.

Night_Watch

It is good to have communication like this and we all, I'm sure, can have at least some empathy for the frontline forum staff that seem to be taking the hits as it were.

I cannot help but wonder though.

Are we seriously to believe that the issue is only due to the patch and event - been seeing complaints on EU side for months.

So you guys know that 'The problem you all have been experiencing only shows up at levels of concurrency seen on PC EU' but you are not prepared for such. Are we seriously to believe that there is no forward planning to combat these expected issues and that this does not show a degree of incompetence?

To me this is a hollow apology. It is in fact an admission of failure with a less than hopeful indication that things may improve. After all the post reads 'we'll wait to see what it does before making any more promises'.

@ZOS_GinaBruno; Thankyou. We know it's not your fault. I see that your post recognizes the frustration of players. Sadly, I can't help think it's not enough.

Just one thing else to add; As always, thank you for your patience.

ClericPro

@ZOS_GinaBruno
You are considering extending the festival, but there are other problems associated with the inability to login for the last 2 days.
I am talking about items in a crown store with a limited duration. I can not contact people selling crown gifts for gold.
If problems with login are not resolved in the coming days, it is highly likely that I will not be able to get some items into my collection (for example, a Chokethorn Arms Pack).
What do you think of these issues?

RefLiberty

'We thought we had fixed the problems that have plagued the last few big in-game events by introducing an updated LFG system, some server optimizations, etc., but obviously it wasn't enoug"


So simple... Your infrastructure can't take it. Are you running a business or what. If something is not paying off, which I doubt, shut it down, if not then improve Datacenter capacity, contrary to majority of other MMOs this game like mature audience who has job and can spend money. Decide what to heck you want and do it finally, at this point your reputation is worst than $5 bucks Chinese Alibaba where you order a real size silicone vagena and you receive a pencil holder. Sry guys didn't manage to get better comparison. But you get the point. If you have the the interest to keep the game running, and let's be honest, by the quality, you are,strange as it sounds, top tier in MMPRPG, get your *** straight or FO. In short, deliver or die ***. Thank you. Honest as always.

oddbasket

Move part of the population out with an oceanic server?

Nemesis7884

stop praising them for doing the absolute bar minimum