Maintenance for the week of March 18:
• PC/Mac: No maintenance – March 18
• ESO Store and Account System for maintenance – March 19, 9:00AM EDT (13:00 UTC) - 1:00PM EDT (17:00 UTC)
• Xbox: NA and EU megaservers for maintenance – March 20, 6:00AM EDT (10:00 UTC) - 10:00AM EDT (14:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – March 20, 6:00AM EDT (10:00 UTC) - 10:00AM EDT (14:00 UTC)

PC-EU Server Update - April 11

  • Elsonso
    Elsonso
    ✭✭✭✭✭
    ✭✭✭✭✭
    Nagastani wrote: »
    So... ummm... why do subscribers have to queue?

    The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.
    PC NA/EU: @Elsonso
    XBox EU/NA: @ElsonsoJannus
    X/Twitter: ElsonsoJannus
  • zergbase_ESO
    zergbase_ESO
    ✭✭✭✭
    Everyone,

    After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

    Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

    Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

    However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

    A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

    So, here is what we are doing to fix the problem:
    • Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
    • Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.
    I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

    Matt

    So you knew since January... Now is when you order hardware... [snip]. You saw the increase but held off on upgrades... If you ordered this hardware earlier like when you saw it your user base would not have this issue at all.

    [edited to remove bashing]
    Edited by ZOS_Ragnar on May 27, 2020 5:49PM
  • Nagastani
    Nagastani
    ✭✭✭✭
    Elsonso wrote: »
    Nagastani wrote: »
    So... ummm... why do subscribers have to queue?

    The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

    Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I :)

    It's about priorities. And on that note, the game has been lagging forever so yeah... I understand you wanting to defend them but this is not a new problem.
    Edited by Nagastani on May 27, 2020 2:15AM
  • lurkin777
    lurkin777
    ✭✭✭
    If the queue was for less than 5 minutes I might be able to stand it but with it being 41 minutes, and the timer does not change at all so is not working at all, so that is unacceptable. Seems that queue is not allowing players in at all if the population is at a certain number. This should not have happened at all. If you know this was going to happen than you should not have released the new chapter until the servers were upgraded.

    This is like charging rent when you can't move in or renting a car that you can't drive on certain days.

    ESO + should not be queued at all since they are paying to play.

    I can't play I don't pay or should not be charged.
    Edited by lurkin777 on May 27, 2020 2:20AM
  • zergbase_ESO
    zergbase_ESO
    ✭✭✭✭
    Nagastani wrote: »
    Elsonso wrote: »
    Nagastani wrote: »
    So... ummm... why do subscribers have to queue?

    The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

    Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I :)

    It's about priorities. And on that note, the game has been lagging forever so yeah... I understand you wanting to defend them but this is not a new problem.

    Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.
  • TineaCruris
    TineaCruris
    ✭✭✭✭✭
    Well, at least it is nice to see a dev admitting what we already know to be true. It's not the dev's fault, the accounting department appears to have decided not to make the investment into enough server to support their software.

    The only long term solution then is for Zenimax to actually pay the recurring costs of enough server to support their game. How do we convince Zenimax to do this?
  • zergbase_ESO
    zergbase_ESO
    ✭✭✭✭
    Well, at least it is nice to see a dev admitting what we already know to be true. It's not the dev's fault, the accounting department appears to have decided not to make the investment into enough server to support their software.

    The only long term solution then is for Zenimax to actually pay the recurring costs of enough server to support their game. How do we convince Zenimax to do this?

    It would fall on system admin team lead for this and whoever is in charge of ordering hardware. From what I understand but if system admins saw the issue and never reported it then it falls on them harshly.
  • nimander99
    nimander99
    ✭✭✭✭✭
    ✭✭
    With the best AAA MMO on the market you all should have been ready for this lmao!!!!!
    I AM UPDATING MY PRIVACY POLICY

    PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!

    ∽∽∽ 2 years of Elder Scrolls Online ∼∼∼
    "Give us money" = Box sales & monthly sub fees,
    "moar!" = £10 palomino horse,
    "MOAR!" = Switch to B2P, launch cash shop,
    "MOAR!!" = Charge for DLC that subs had already paid for,
    "MOAR!!!" = Experience scrolls and riding lessons,
    "MOARR!!!" = Vampire/werewolf bites,
    "MOAARRR!!!" = CS exclusive motifs,
    "MOOAARRR!!!" = Crown crates,
    "MOOOAAARRR!!!" = 'Chapter's' bought separately from ESO+,
    "MOOOOAAAARRRR!!!!" = ???

    Male, Dunmer, VR16, Templar, Aldmeri Dominion, Master Crafter & all Traits, CP450
  • Nagastani
    Nagastani
    ✭✭✭✭
    Well, at least it is nice to see a dev admitting what we already know to be true. It's not the dev's fault, the accounting department appears to have decided not to make the investment into enough server to support their software.

    The only long term solution then is for Zenimax to actually pay the recurring costs of enough server to support their game. How do we convince Zenimax to do this?

    It would fall on system admin team lead for this and whoever is in charge of ordering hardware. From what I understand but if system admins saw the issue and never reported it then it falls on them harshly.

    Right, IT management has to approve server upgrades. It's the server Director's job to approach management with recommendations and their plan to fix whatever the issue happens to be, then management has to decide to sign off on it or not. From what others have told me they can't justify the cost of upgrading the servers off mega server technology. It could be the price, it could be the technology itself however my best guess is it's a combination of the two.
  • TineaCruris
    TineaCruris
    ✭✭✭✭✭
    Please check the date of the original post. This is from 13 plus months ago. It sure seems like NA did not get any hardware upgrades in that time.
  • McGordon
    McGordon
    ✭✭✭
    These is from April 2019, if you dig deeper in forum, I remember similar problems in past. Smells like manual server scaling.
  • volkeswagon
    volkeswagon
    ✭✭✭✭✭
    stuff like this helps to teach patience, understanding, emphathy, etc so if you're flipping out and writing nasty comments then you have some things to work on.
  • Elsonso
    Elsonso
    ✭✭✭✭✭
    ✭✭✭✭✭
    Nagastani wrote: »
    Elsonso wrote: »
    Nagastani wrote: »
    So... ummm... why do subscribers have to queue?

    The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

    Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I :)

    My comment was that letting the subscribers past the queue is pointless because the server usage is likely to be from the people who are the subscribers. The queues are obviously to restrict how many people can be on the server, so turning around and letting most of those people back in does not really address the problem of too many people.
    Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

    No one here in the forum knows what ZOS is doing with the servers. They said they were going to upgrade them. They said they were preparing to upgrade them. They said they did do server upgrades. We will never know the specifics, timetables, and completion dates.

    A few people, you and me included, have jobs that allow us to have limited insight into what we don't know. This is why I am more cautious about pointing fingers.

    For example, it is looking like the launch day problems were not due to inadequate server capacity. Instead, it looks like problems with software and configuration caused the bulk of the problems. It is possible that the megaserver itself was the only thing that worked on Tuesday.

    This actually makes sense since it has become clear that almost all of the performance problems that we see, and definitely all of the bugs that we see, stem from software and configuration problems, not a lack of hardware.

    Still, for a little while, I fell into the "too many people, not enough server" mindset on Tuesday.
    Edited by Elsonso on May 28, 2020 11:35AM
    PC NA/EU: @Elsonso
    XBox EU/NA: @ElsonsoJannus
    X/Twitter: ElsonsoJannus
  • TineaCruris
    TineaCruris
    ✭✭✭✭✭
    Elsonso wrote: »
    Nagastani wrote: »
    Elsonso wrote: »
    Nagastani wrote: »
    So... ummm... why do subscribers have to queue?

    The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

    Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I :)

    My comment was that letting the subscribers past the queue is pointless because the server usage is likely to be from the people who are the subscribers. The queues are obviously to restrict how many people can be on the server, so turning around and letting most of those people back in does not really address the problem of too many people.
    Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

    No one here in the forum knows what ZOS is doing with the servers. They said they were going to upgrade them. They said they were preparing to upgrade them. They said they did do server upgrades. We will never know the specifics, timetables, and completion dates.

    A few people, you and me included, have jobs that allow us to have limited insight into what we don't know. This is why I am more cautious about pointing fingers.

    For example, it is looking like the launch day problems were not due to inadequate server capacity. Instead, it looks like problems with software and configuration caused the bulk of the problems. It is possible that the megaserver itself was the only thing that worked on Tuesday.

    This actually makes sense since it has become clear that almost all of the performance problems that we see, and definitely all of the bugs that we see, stem from software and configuration problems, not a lack of hardware.

    Still, for a little while, I fell into the "too many people, not enough server" mindset on Tuesday.

    The problem with your assessment is that its not congruent with the OP's, and the OP is a dev.

    Have we gotten the physical hardware upgrades that the OP said we needed last year or not? It's not a controversial question, or at least it shouldn't be.

    https://forums.elderscrollsonline.com/en/discussion/529530/april-2019-matt-firor-promised-server-hardware-upgrades-did-they-happen#latest
  • Elsonso
    Elsonso
    ✭✭✭✭✭
    ✭✭✭✭✭
    Elsonso wrote: »
    Nagastani wrote: »
    Elsonso wrote: »
    Nagastani wrote: »
    So... ummm... why do subscribers have to queue?

    The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

    Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I :)

    My comment was that letting the subscribers past the queue is pointless because the server usage is likely to be from the people who are the subscribers. The queues are obviously to restrict how many people can be on the server, so turning around and letting most of those people back in does not really address the problem of too many people.
    Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

    No one here in the forum knows what ZOS is doing with the servers. They said they were going to upgrade them. They said they were preparing to upgrade them. They said they did do server upgrades. We will never know the specifics, timetables, and completion dates.

    A few people, you and me included, have jobs that allow us to have limited insight into what we don't know. This is why I am more cautious about pointing fingers.

    For example, it is looking like the launch day problems were not due to inadequate server capacity. Instead, it looks like problems with software and configuration caused the bulk of the problems. It is possible that the megaserver itself was the only thing that worked on Tuesday.

    This actually makes sense since it has become clear that almost all of the performance problems that we see, and definitely all of the bugs that we see, stem from software and configuration problems, not a lack of hardware.

    Still, for a little while, I fell into the "too many people, not enough server" mindset on Tuesday.

    The problem with your assessment is that its not congruent with the OP's, and the OP is a dev.

    The important realization is that Firor is talking about different things, with different solutions, in the same message. One of those things is server capacity, and that was addressed by putting in additional hardware. We know that they did this, that issue is closed, and that they are monitoring it to see if they need to add more.

    Hardware is important to improve the ability of the servers to handle basic high concurrency. This is important because the game is at a high point in terms of concurrency. I expect this to be evident by the fact that they have more server resources that they can deploy to handle placement of characters into the game. To me, this means less crowded zones and ability to create more "instances" where they can place players in zones, houses, and dungeons.

    I am not expecting that this will resolve the other performance problems that are more related to software. This should have little to no effect on Cyrodiil, for example.

    As with most complex things, there is no single answer to what is going on with Performance, and that is where what Firor said merges with the overall Performance Plan that ZOS has been updating us on monthly. What is in that list, and is mentioned in passing by Firor, is a list of other fixes that are not hardware.

    It is those performance problems that I think we are all noticing. Input lag, desync, and all of that stuff. Some of this stuff has been going on for months and years, other stuff was introduced with Harrowstorm.
    PC NA/EU: @Elsonso
    XBox EU/NA: @ElsonsoJannus
    X/Twitter: ElsonsoJannus
  • Psiion
    Psiion
    ✭✭✭✭
    Greetings,

    Due to the age of this thread, and the fact the information within is outdated, we have closed this thread down. Thank you for your understanding.
    Staff Post
This discussion has been closed.