PC-EU Server Update - April 11

ZOS_MattFiror

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

So, here is what we are doing to fix the problem:

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

Matt

Schemering

i noticed this yesterday; it said 16 minutes waiting and it was actually smt like 8 minutes. If it stays in the few minutes range its no problem for me, its nice to have a growing population.

Nifty2g

log in to do a raid on eso and be put on a queue, hopefully that doesn't become a problem for the raid groups.

CMDR_Un1k0rn

Good information.

I would only ask that this all gets sorted in that two-week time-frame you have now posted, otherwise, there's likely going to be a possibly irreparable damage in customer faith, in my opinion.

Thank you for the update.

Suddwrath

Unless there really hasn't been an influx in players and this is a tactic to convince people that ESO has a growing community in order to convince them to buy the game...



On a more serious note: It's nice seeing a post from Matt huzzah for communication!

SantieClaws

Greetings traveller Matt

Is there any possibility of an extension to the Anniversary Festival, perhaps by a week?

So no one can say they have missed out event tickets and delicious shiny lootables.

Khajiit also worries what will happen to the login system at the weekend when longer guild events and things take place.

If the server is full and no one logs off then is it possible this one, or even someone else not this one, might not be able to log in at all?!

Yours with paws
Santie Claws

Lloydmp

@ZOS_MattFiror
Really appreciate the communication on what’s going on and what the plan is.

ZeroXFF

New hamsters, yay!

Miswar

We all do respect but the PS4 console server performances have also been crap for ages now so by saying that you are "looking closely on the performance"... does that mean in ZoS opinion they are functioning "as intended" at the moment?

If so than that is just absurd statement itself since we all do know the performance on these servers have been pure crap for longtime now!!!

SwampDrainerGoat

So kinda like closing the gate after the horse has bolted. You let it get this bad and enable queueing for EU, yet you are going to upgrade NA server before the influx happens. Nice to see who your priority customers are.

gepe87

Wow. If Matt replied to this then this problem is really serious!
I hope both devs and players will overcome this

#ESOfam

Minno

Thanks for the ETA

YerBud

I waited in a 2-minute queue in the NA server this morning around 5 am on the East Coast USA. It was the first time I logged in without a subscription in several months and figured it was just because of that reason.

lardvader

Thanks Matt for taking some time to elaborate the issues at hand. Hope it will be solved in a reasonable timeframe.

Minno

YerBud wrote: »

I waited in a 2-minute queue in the NA server this morning around 5 am on the East Coast USA. It was the first time I logged in without a subscription in several months and figured it was just because of that reason.

NA server doesnt have the que, as far as I know.

Turelus

Thanks for the update Matt, good to know what's been happening and what the plan is going forward.

I hate to add a negative comment to my post but if the cause of some of the EU performance issues have been known since January why couldn't we get a statement as to why earlier?
There have been A LOT of threads from angry PC EU players over the last few months which were met with silence, and only now when the situation has become critical do we know what the cause was.

I think a lot of people might have been appeased if they had known the reasons sooner.

Any way I don't want to focus too much on the negative but I wanted to add a bit of critic, especially with ZOS talking about wanting to improve communication this year.

Onwards to upgraded servers and many new friends to play the game with!

menschikov2

@ZOS_GinaBruno
I cant confirm that we have a 20 minutes time to log back in. Yesterday i crashed in vMoL HM(what leaded to a wipe because i was the maintank) and then i logged back in and had a Queue (i instantly logged in after the disconect so there were no 20 minutes. The whole raid lost roundabout 15-20 minutes because of this bug Another Player had the same Thing t the start of the Raid so Overall we lost 30 minutes auf our 2 Hours raidtime

Iluvrien

All hail extra communication, and thank you for explaining the situation.

The timing differential between NA and EU fixes really isn't helping with the sense of EU being the red-headed stepchild though.

YerBud

I'm saying, in fact, I waited for 2 minutes to connect to the character login screen from my Steam Account after it said that it placed me in a queue, are you calling me a liar? Do I need to have some silly proof of this like a screenshot that I would never have dreamed of taking because as I said I figured it was due to the fact I had no sub so I brushed it off as a new policy for non-patrons?

Parrot1986

Thanks for the update @ZOS_MattFiror good to see something be done just concerning it took it to kick in before communicating it. Other MMOs have log in queues as well so it’s not totally foreign just not ideal when ESO has never had this.

One thing I noticed yesterday is this quite isn’t happening I practise.

ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.

Yesterday 2 from our group got kicked and couldn’t get back in when trying and ended up in a cycle of log in attempts. I’m sure this will get flagged tonight if this kicks in but it doesn’t seem to work like this in reality. It really needs to though based on how often disconnects are happening. Even swapping between characters I can be sent back to log in screen so if that led to going back into the queue would be really poor.

Alex_Lex

Do you really had to wait till issues would drop the server to start actually doing some long-term plans?
Can't you make it earlier?

ManwithBeard9

YerBud wrote: »

I'm saying, in fact, I waited for 2 minutes to connect to the character login screen from my Steam Account after it said that it placed me in a queue, are you calling me a liar? Do I need to have some silly proof of this like a screenshot that I would never have dreamed of taking because as I said I figured it was due to the fact I had no sub so I brushed it off as a new policy for non-patrons?

It has nothing to do with you not being subbed. It has everything to do with the amount of players in game. And possible you using Steam.

Suddwrath

Turelus wrote: »

I hate to add a negative comment to my post but if the cause of some of the EU performance issues have been known since January why couldn't we get a statement as to why earlier?
There have been A LOT of threads from angry PC EU players over the last few months which were met with silence, and only now when the situation has become critical do we know what the cause was.

I have to agree with this 100%. I don't even play on PC/EU, but it only took a minute to glance at the forums to see the sheer number of threads being made by PC/EU players who were frustrated with the performance. So if the reason was known back in January, why was this not shared with the community?

I appreciate the communication now and especially the two week ETA, but why couldn't this information have been shared back in January when EU players were voicing their frustrations on the forums?

OwnerOfSuccuby

It is good, that you find and understand you mistake, good that 2 weaks for fix. But bad that you see that problem is starting from January and do not make some thing to prevent it. But i hope we will normally play good again. Like it was half year ago.

Sanguinor2

ZOS_MattFiror wrote: »

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

Nice to see EU is still treated as second class Citizen.... why you can only get ahead of the Problem for NA but couldnt have done the same for EU shall forever remain a mystery.

Morgul667

Nice communication but we have been saying there is something wrong with eu server for years

Performance is going lower every patch and this is not the customers connexion issue

Please invest in something real this time and start with a clean slate with your player base

ManwithBeard9

Sanguinor2 wrote: »

ZOS_MattFiror wrote: »

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

Nice to see EU is still treated as second class Citizen.... why you can only get ahead of the Problem for NA but couldnt have done the same for EU shall forever remain a mystery.

Because they were hoping it wouldnt get to a critical point. So now that its at critical on one server, they might as well add for both instead of repeating the process later.

daemonios

Thank you @ZOS_MattFiror for the update. I for one can't say I'm fully satisfied by your answer, though.

I feel the PC-EU players' complaints have been constantly downplayed or outright ignored until the issues couldn't be swept under the rug any more. It's been 3 months since the moment things got really bad, yet only now are you reacting, and the first reaction of turning on the queue, while I can understand the purpose, is further punishing the players. Why wasn't the problem acknowledged and addressed sooner?

Secondly, either I'm very much mistaken, or you already have the ability to increase your server capacity. I haven't played PvP regularly in years, but during the Midyear Mayhem event I spent many hours in Cyrodiil and was pleasantly surprised at the performance even with every faction pop-locked, as were others who talked about it in-game or in the forums. Did you not expand your server capacity then, and if you did, why didn't you do the same now?

We'll have to see how things go from now on, but in my book ZOS has barely any credit left. I hope you can turn this around.

Mitrenga

I am one of the many players that have been complaining and joking about the PC EU server performance (or underperformance), but this is a good sign. Problem identified, temporary measures are on and a permanent plan is in motion. Thank you for the information and please keep up the efforts to improve the game in any way you can.

olsborg

I hope the hardware updates are in place quickly and that they work. For myself personally, I wont bother queuing to play, I will simply play a different game, and if that is ongoing beyond "these 2 weeks" that you mentioned, I will probably try to log in less frequently etc..

essi2

Thanks for the info, but in the future you might not want to use the EU server as a guinea pig to test if action is needed to alleviate server and game issues, especially not when they actively prevent people from effectively playing the game.

We're all starting to get fairly fatigued by the constant performance issues, gameplay issues and unknowable amount of 'known shippables'.