Maintenance for the week of March 18:
• [IN PROGRESS] ESO Store and Account System for maintenance – March 19, 9:00AM EDT (13:00 UTC) - 1:00PM EDT (17:00 UTC)
• Xbox: NA and EU megaservers for maintenance – March 20, 6:00AM EDT (10:00 UTC) - 10:00AM EDT (14:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – March 20, 6:00AM EDT (10:00 UTC) - 10:00AM EDT (14:00 UTC)

PC-EU Server Update - April 11

ZOS_MattFiror
Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

So, here is what we are doing to fix the problem:
  • Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
  • Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.
I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

Matt
Edited by Psiion on March 3, 2022 10:21PM
Matt Firor
Studio Director, ZeniMax Online Studios
Staff Post
  • Schemering
    Schemering
    ✭✭✭
    i noticed this yesterday; it said 16 minutes waiting and it was actually smt like 8 minutes. If it stays in the few minutes range its no problem for me, its nice to have a growing population.
    PC/EU AD 1500+ PC/NA 300+

    Schemering - Breton magicka Nightblade
    Ambergloed - Argonian Templar Healer
    Fonkeling - Argonian Dragonknight Tank
    Twinkeling - Dunmer magicka Nightblade
    Sprankeling- Altmer magicka Nightblade
    Schittering - Redguard stamina Nightblade
    Glinstering - Khajiit stamina Sorcerer
    Spiegeling - Altmer magicka Necromancer
    Flonkering - Orc stamina Necromancer
    Glimmering - Argonian Necromancer
    Duisternis - Dunmer magicka Dragonknight
    Maanlicht - Altmer magicka Templar
    Weerlicht - Altmer magicka Sorcerer
    Zonnestraal - Redguard stamina Warden EP char
    Slagschaduw - Dunmer magicka Warden - Healer or Damage Dealer
    Ochtendgloren - Imperial stamina Templar
    Avondval - Redguard stamina Dragonknight
    Aurora Noorderlicht - Breton magicka Nightblade DC char
    Dageraad - Breton magicka Sorcerer

    Wisseling - Breton magicka Nightblade NA
    Zonsverduistering - Breton Templar NA Healer
    Tinteling - Argonian Dragonknight NA Tank
  • Nifty2g
    Nifty2g
    ✭✭✭✭✭
    ✭✭✭✭✭
    log in to do a raid on eso and be put on a queue, hopefully that doesn't become a problem for the raid groups.
    Edited by Nifty2g on April 11, 2019 1:59PM
    #MOREORBS
  • CMDR_Un1k0rn
    CMDR_Un1k0rn
    ✭✭✭✭✭
    ✭✭
    Good information.

    I would only ask that this all gets sorted in that two-week time-frame you have now posted, otherwise, there's likely going to be a possibly irreparable damage in customer faith, in my opinion.

    Thank you for the update.
    In-game username: Un1korn | Happy member of the PCNA UESP guild (Resident Daggerfall Covenant enjoyer) | Main & basically only character: Crucian Vulpin, Imperial Dragonknight of the Daggerfall Covenant, and Undaunted Bulwark (I tank) | If you know me from PCEU: No
  • Suddwrath
    Suddwrath
    ✭✭✭✭✭
    ✭✭✭
    Unless there really hasn't been an influx in players and this is a tactic to convince people that ESO has a growing community in order to convince them to buy the game...

    ?u=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl0IylOPCNkiqOgMyA%2Fgiphy.gif&f=1

    On a more serious note: It's nice seeing a post from Matt :) huzzah for communication!
  • SantieClaws
    SantieClaws
    ✭✭✭✭✭
    ✭✭✭✭✭
    Greetings traveller Matt

    Is there any possibility of an extension to the Anniversary Festival, perhaps by a week?

    So no one can say they have missed out event tickets and delicious shiny lootables.

    Khajiit also worries what will happen to the login system at the weekend when longer guild events and things take place.

    If the server is full and no one logs off then is it possible this one, or even someone else not this one, might not be able to log in at all?!

    Yours with paws
    Santie Claws
    Shunrr's Skooma Oasis - The Movie. A housing video like no other ...
    Find it here - https://youtube.com/user/wenxue2222

    Clan Claws - now recruiting khajiit and like minded others for parties, fishing and other khajiit stuff. Contact this one for an invite.

    PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!

    https://www.imperialtradingcompany.eu/
  • Lloydmp
    Lloydmp
    ✭✭✭
    @ZOS_MattFiror
    Really appreciate the communication on what’s going on and what the plan is.
  • ZeroXFF
    ZeroXFF
    ✭✭✭✭✭
    New hamsters, yay!
  • Miswar
    Miswar
    ✭✭✭✭✭
    We all do respect but the PS4 console server performances have also been crap for ages now so by saying that you are "looking closely on the performance"... does that mean in ZoS opinion they are functioning "as intended" at the moment?

    If so than that is just absurd statement itself since we all do know the performance on these servers have been pure crap for longtime now!!!
  • SwampDrainerGoat
    SwampDrainerGoat
    ✭✭✭
    So kinda like closing the gate after the horse has bolted. You let it get this bad and enable queueing for EU, yet you are going to upgrade NA server before the influx happens. Nice to see who your priority customers are.
  • gepe87
    gepe87
    ✭✭✭✭✭
    Wow. If Matt replied to this then this problem is really serious!
    I hope both devs and players will overcome this :)

    #ESOfam
    Gepe, Dunmer MagSorc Pact Grand Overlord | Gaepe, Bosmer MagSorc Dominion General

    If you see edits on my replies: typos. English isn't my main language
  • Minno
    Minno
    ✭✭✭✭✭
    ✭✭✭✭✭
    Thanks for the ETA ;)
    Minno - DC - Forum-plar Extraordinaire
    - Guild-lead for MV
    - Filthy Casual
  • YerBud
    YerBud
    Soul Shriven
    I waited in a 2-minute queue in the NA server this morning around 5 am on the East Coast USA. It was the first time I logged in without a subscription in several months and figured it was just because of that reason.
  • lardvader
    lardvader
    ✭✭✭✭✭
    Thanks Matt for taking some time to elaborate the issues at hand. Hope it will be solved in a reasonable timeframe.
    CP 1200+ PC EU EP
  • Minno
    Minno
    ✭✭✭✭✭
    ✭✭✭✭✭
    YerBud wrote: »
    I waited in a 2-minute queue in the NA server this morning around 5 am on the East Coast USA. It was the first time I logged in without a subscription in several months and figured it was just because of that reason.

    NA server doesnt have the que, as far as I know.
    Minno - DC - Forum-plar Extraordinaire
    - Guild-lead for MV
    - Filthy Casual
  • Turelus
    Turelus
    ✭✭✭✭✭
    ✭✭✭✭✭
    Thanks for the update Matt, good to know what's been happening and what the plan is going forward.

    I hate to add a negative comment to my post but if the cause of some of the EU performance issues have been known since January why couldn't we get a statement as to why earlier?
    There have been A LOT of threads from angry PC EU players over the last few months which were met with silence, and only now when the situation has become critical do we know what the cause was.

    I think a lot of people might have been appeased if they had known the reasons sooner.

    Any way I don't want to focus too much on the negative but I wanted to add a bit of critic, especially with ZOS talking about wanting to improve communication this year.

    Onwards to upgraded servers and many new friends to play the game with!
    Edited by Turelus on April 11, 2019 2:15PM
    @Turelus - EU PC Megaserver
    "Don't count on others for help. In the end each of us is in this alone. The survivors are those who know how to look out for themselves."
  • menschikov2
    menschikov2
    ✭✭✭
    @ZOS_GinaBruno
    I cant confirm that we have a 20 minutes time to log back in. Yesterday i crashed in vMoL HM(what leaded to a wipe because i was the maintank) and then i logged back in and had a Queue (i instantly logged in after the disconect so there were no 20 minutes. The whole raid lost roundabout 15-20 minutes because of this bug :/ Another Player had the same Thing t the start of the Raid so Overall we lost 30 minutes auf our 2 Hours raidtime :(
  • Iluvrien
    Iluvrien
    ✭✭✭✭✭
    ✭✭
    All hail extra communication, and thank you for explaining the situation.

    The timing differential between NA and EU fixes really isn't helping with the sense of EU being the red-headed stepchild though.
  • YerBud
    YerBud
    Soul Shriven
    I'm saying, in fact, I waited for 2 minutes to connect to the character login screen from my Steam Account after it said that it placed me in a queue, are you calling me a liar? Do I need to have some silly proof of this like a screenshot that I would never have dreamed of taking because as I said I figured it was due to the fact I had no sub so I brushed it off as a new policy for non-patrons?
  • Parrot1986
    Parrot1986
    ✭✭✭✭✭
    Thanks for the update @ZOS_MattFiror good to see something be done just concerning it took it to kick in before communicating it. Other MMOs have log in queues as well so it’s not totally foreign just not ideal when ESO has never had this.

    One thing I noticed yesterday is this quite isn’t happening I practise.

    ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.

    Yesterday 2 from our group got kicked and couldn’t get back in when trying and ended up in a cycle of log in attempts. I’m sure this will get flagged tonight if this kicks in but it doesn’t seem to work like this in reality. It really needs to though based on how often disconnects are happening. Even swapping between characters I can be sent back to log in screen so if that led to going back into the queue would be really poor.
  • Alex_Lex
    Alex_Lex
    ✭✭✭✭✭
    Do you really had to wait till issues would drop the server to start actually doing some long-term plans?
    Can't you make it earlier?
    EU: Beatrice gra-Brog, Rebellie, Yshterie, Jearra , Lirielline
    Christess, Fleuretta, Jubilee Cake, Braviliana, Onyxette
    M'adjirra, Dances-around-Nirn, Nerieth, Estrice, Ann der Wood
    Éternelle, Burns-all-the-babies, Jyggurag
    NA: Dice Twice, Nikka from the barrel
  • ManwithBeard9
    ManwithBeard9
    ✭✭✭✭✭
    ✭✭
    YerBud wrote: »
    I'm saying, in fact, I waited for 2 minutes to connect to the character login screen from my Steam Account after it said that it placed me in a queue, are you calling me a liar? Do I need to have some silly proof of this like a screenshot that I would never have dreamed of taking because as I said I figured it was due to the fact I had no sub so I brushed it off as a new policy for non-patrons?

    It has nothing to do with you not being subbed. It has everything to do with the amount of players in game. And possible you using Steam.
  • Suddwrath
    Suddwrath
    ✭✭✭✭✭
    ✭✭✭
    Turelus wrote: »
    I hate to add a negative comment to my post but if the cause of some of the EU performance issues have been known since January why couldn't we get a statement as to why earlier?
    There have been A LOT of threads from angry PC EU players over the last few months which were met with silence, and only now when the situation has become critical do we know what the cause was.

    I have to agree with this 100%. I don't even play on PC/EU, but it only took a minute to glance at the forums to see the sheer number of threads being made by PC/EU players who were frustrated with the performance. So if the reason was known back in January, why was this not shared with the community?

    I appreciate the communication now and especially the two week ETA, but why couldn't this information have been shared back in January when EU players were voicing their frustrations on the forums?
  • OwnerOfSuccuby
    OwnerOfSuccuby
    ✭✭✭
    It is good, that you find and understand you mistake, good that 2 weaks for fix. But bad that you see that problem is starting from January and do not make some thing to prevent it. But i hope we will normally play good again. Like it was half year ago.
  • Sanguinor2
    Sanguinor2
    ✭✭✭✭✭
    ✭✭
    Everyone,

    After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

    • Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
    • Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.


    Nice to see EU is still treated as second class Citizen.... why you can only get ahead of the Problem for NA but couldnt have done the same for EU shall forever remain a mystery.



    Politeness is respecting others.
    Courage is doing what is fair.
    Modesty is speaking of oneself without vanity.
    Self control is keeping calm even when anger rises.
    Sincerity is expressing oneself without concealing ones thoughts.
    Honor is keeping ones word.
  • Morgul667
    Morgul667
    ✭✭✭✭✭
    ✭✭✭✭✭
    Nice communication but we have been saying there is something wrong with eu server for years

    Performance is going lower every patch and this is not the customers connexion issue

    Please invest in something real this time and start with a clean slate with your player base
  • ManwithBeard9
    ManwithBeard9
    ✭✭✭✭✭
    ✭✭
    Sanguinor2 wrote: »
    Everyone,

    After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

    • Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
    • Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.


    Nice to see EU is still treated as second class Citizen.... why you can only get ahead of the Problem for NA but couldnt have done the same for EU shall forever remain a mystery.



    Because they were hoping it wouldnt get to a critical point. So now that its at critical on one server, they might as well add for both instead of repeating the process later.
  • daemonios
    daemonios
    ✭✭✭✭✭
    ✭✭✭✭✭
    Thank you @ZOS_MattFiror for the update. I for one can't say I'm fully satisfied by your answer, though.

    I feel the PC-EU players' complaints have been constantly downplayed or outright ignored until the issues couldn't be swept under the rug any more. It's been 3 months since the moment things got really bad, yet only now are you reacting, and the first reaction of turning on the queue, while I can understand the purpose, is further punishing the players. Why wasn't the problem acknowledged and addressed sooner?

    Secondly, either I'm very much mistaken, or you already have the ability to increase your server capacity. I haven't played PvP regularly in years, but during the Midyear Mayhem event I spent many hours in Cyrodiil and was pleasantly surprised at the performance even with every faction pop-locked, as were others who talked about it in-game or in the forums. Did you not expand your server capacity then, and if you did, why didn't you do the same now?

    We'll have to see how things go from now on, but in my book ZOS has barely any credit left. I hope you can turn this around.
  • Mitrenga
    Mitrenga
    ✭✭✭✭✭
    I am one of the many players that have been complaining and joking about the PC EU server performance (or underperformance), but this is a good sign. Problem identified, temporary measures are on and a permanent plan is in motion. Thank you for the information and please keep up the efforts to improve the game in any way you can.
  • olsborg
    olsborg
    ✭✭✭✭✭
    ✭✭✭✭✭
    I hope the hardware updates are in place quickly and that they work. For myself personally, I wont bother queuing to play, I will simply play a different game, and if that is ongoing beyond "these 2 weeks" that you mentioned, I will probably try to log in less frequently etc..

    PC EU
    PvP only
  • essi2
    essi2
    ✭✭✭✭✭
    Thanks for the info, but in the future you might not want to use the EU server as a guinea pig to test if action is needed to alleviate server and game issues, especially not when they actively prevent people from effectively playing the game.

    We're all starting to get fairly fatigued by the constant performance issues, gameplay issues and unknowable amount of 'known shippables'.
    "The Heritance are racists yes? Idiots. But dangerous, destabilizing racist idiots." - Razum-dar

    "Wood Elves aren't made of wood, Sea Elves aren't made of water. M'aiq still wonders about High Elves" - M'aiq the Liar

    ** Leyawiin Layabouts (PC-EU) - Leyawiin Layabouts (PC-NA) **

    *** https://www.youtube.com/@essi2 - https://www.twitch.tv/essi2 ***
This discussion has been closed.