PC-EU Server Update - April 11

Elsonso

Nagastani wrote: »

So... ummm... why do subscribers have to queue?

The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

zergbase_ESO

ZOS_MattFiror wrote: »

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

So, here is what we are doing to fix the problem:

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

Matt

So you knew since January... Now is when you order hardware... [snip]. You saw the increase but held off on upgrades... If you ordered this hardware earlier like when you saw it your user base would not have this issue at all.

[edited to remove bashing]

Nagastani

Elsonso wrote: »

Nagastani wrote: »

So... ummm... why do subscribers have to queue?

The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I

It's about priorities. And on that note, the game has been lagging forever so yeah... I understand you wanting to defend them but this is not a new problem.

lurkin777

If the queue was for less than 5 minutes I might be able to stand it but with it being 41 minutes, and the timer does not change at all so is not working at all, so that is unacceptable. Seems that queue is not allowing players in at all if the population is at a certain number. This should not have happened at all. If you know this was going to happen than you should not have released the new chapter until the servers were upgraded.

This is like charging rent when you can't move in or renting a car that you can't drive on certain days.

ESO + should not be queued at all since they are paying to play.

I can't play I don't pay or should not be charged.

zergbase_ESO

Nagastani wrote: »

Elsonso wrote: »

Nagastani wrote: »

So... ummm... why do subscribers have to queue?

The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I

It's about priorities. And on that note, the game has been lagging forever so yeah... I understand you wanting to defend them but this is not a new problem.

Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

TineaCruris

Well, at least it is nice to see a dev admitting what we already know to be true. It's not the dev's fault, the accounting department appears to have decided not to make the investment into enough server to support their software.

The only long term solution then is for Zenimax to actually pay the recurring costs of enough server to support their game. How do we convince Zenimax to do this?

zergbase_ESO

TineaCruris wrote: »

Well, at least it is nice to see a dev admitting what we already know to be true. It's not the dev's fault, the accounting department appears to have decided not to make the investment into enough server to support their software.

The only long term solution then is for Zenimax to actually pay the recurring costs of enough server to support their game. How do we convince Zenimax to do this?

It would fall on system admin team lead for this and whoever is in charge of ordering hardware. From what I understand but if system admins saw the issue and never reported it then it falls on them harshly.

nimander99

With the best AAA MMO on the market you all should have been ready for this lmao!!!!!

Nagastani

zergbase_ESO wrote: »

TineaCruris wrote: »

Well, at least it is nice to see a dev admitting what we already know to be true. It's not the dev's fault, the accounting department appears to have decided not to make the investment into enough server to support their software.

The only long term solution then is for Zenimax to actually pay the recurring costs of enough server to support their game. How do we convince Zenimax to do this?

It would fall on system admin team lead for this and whoever is in charge of ordering hardware. From what I understand but if system admins saw the issue and never reported it then it falls on them harshly.

Right, IT management has to approve server upgrades. It's the server Director's job to approach management with recommendations and their plan to fix whatever the issue happens to be, then management has to decide to sign off on it or not. From what others have told me they can't justify the cost of upgrading the servers off mega server technology. It could be the price, it could be the technology itself however my best guess is it's a combination of the two.

TineaCruris

Please check the date of the original post. This is from 13 plus months ago. It sure seems like NA did not get any hardware upgrades in that time.

McGordon

These is from April 2019, if you dig deeper in forum, I remember similar problems in past. Smells like manual server scaling.

volkeswagon

stuff like this helps to teach patience, understanding, emphathy, etc so if you're flipping out and writing nasty comments then you have some things to work on.

Elsonso

Nagastani wrote: »

Elsonso wrote: »

Nagastani wrote: »

So... ummm... why do subscribers have to queue?

The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I

My comment was that letting the subscribers past the queue is pointless because the server usage is likely to be from the people who are the subscribers. The queues are obviously to restrict how many people can be on the server, so turning around and letting most of those people back in does not really address the problem of too many people.

zergbase_ESO wrote: »

Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

No one here in the forum knows what ZOS is doing with the servers. They said they were going to upgrade them. They said they were preparing to upgrade them. They said they did do server upgrades. We will never know the specifics, timetables, and completion dates.

A few people, you and me included, have jobs that allow us to have limited insight into what we don't know. This is why I am more cautious about pointing fingers.

For example, it is looking like the launch day problems were not due to inadequate server capacity. Instead, it looks like problems with software and configuration caused the bulk of the problems. It is possible that the megaserver itself was the only thing that worked on Tuesday.

This actually makes sense since it has become clear that almost all of the performance problems that we see, and definitely all of the bugs that we see, stem from software and configuration problems, not a lack of hardware.

Still, for a little while, I fell into the "too many people, not enough server" mindset on Tuesday.

TineaCruris

Elsonso wrote: »

Nagastani wrote: »

Elsonso wrote: »

Nagastani wrote: »

So... ummm... why do subscribers have to queue?

The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I

My comment was that letting the subscribers past the queue is pointless because the server usage is likely to be from the people who are the subscribers. The queues are obviously to restrict how many people can be on the server, so turning around and letting most of those people back in does not really address the problem of too many people.

zergbase_ESO wrote: »

Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

No one here in the forum knows what ZOS is doing with the servers. They said they were going to upgrade them. They said they were preparing to upgrade them. They said they did do server upgrades. We will never know the specifics, timetables, and completion dates.

A few people, you and me included, have jobs that allow us to have limited insight into what we don't know. This is why I am more cautious about pointing fingers.

For example, it is looking like the launch day problems were not due to inadequate server capacity. Instead, it looks like problems with software and configuration caused the bulk of the problems. It is possible that the megaserver itself was the only thing that worked on Tuesday.

This actually makes sense since it has become clear that almost all of the performance problems that we see, and definitely all of the bugs that we see, stem from software and configuration problems, not a lack of hardware.

Still, for a little while, I fell into the "too many people, not enough server" mindset on Tuesday.

The problem with your assessment is that its not congruent with the OP's, and the OP is a dev.

Have we gotten the physical hardware upgrades that the OP said we needed last year or not? It's not a controversial question, or at least it shouldn't be.

https://forums.elderscrollsonline.com/en/discussion/529530/april-2019-matt-firor-promised-server-hardware-upgrades-did-they-happen#latest

Elsonso

TineaCruris wrote: »

Elsonso wrote: »

Nagastani wrote: »

Elsonso wrote: »

Nagastani wrote: »

So... ummm... why do subscribers have to queue?

The majority of the players that are taking up all the time in the game are probably ESO Plus. Letting ESO Plus players in unrestricted would probably be just like not having a queue at all. Then, we would be back to where the game was earlier, where using a wayshrine took 5 minutes, if you did not get timed out and kicked off. Of course, I am not on NA right now, so I have no idea whether the queue fixed that.

Please consider what zergbase_ESO just said. Was going to post something else however they said it better than I

My comment was that letting the subscribers past the queue is pointless because the server usage is likely to be from the people who are the subscribers. The queues are obviously to restrict how many people can be on the server, so turning around and letting most of those people back in does not really address the problem of too many people.

zergbase_ESO wrote: »

Pretty much. Our hospital saw server resources slowly increasing as people worked from home. First thing I did was order hardware to increase our server limitations month by month until everything settled down. Now we have infrastructure to support all states clinics and hospitals to Work from home ((WFH))... You can see it happening as system admin on the backend... This what upsets me the most honestly. Like biggest F-U to your client base if you saw it in January and did not bother to do anything.

No one here in the forum knows what ZOS is doing with the servers. They said they were going to upgrade them. They said they were preparing to upgrade them. They said they did do server upgrades. We will never know the specifics, timetables, and completion dates.

A few people, you and me included, have jobs that allow us to have limited insight into what we don't know. This is why I am more cautious about pointing fingers.

For example, it is looking like the launch day problems were not due to inadequate server capacity. Instead, it looks like problems with software and configuration caused the bulk of the problems. It is possible that the megaserver itself was the only thing that worked on Tuesday.

This actually makes sense since it has become clear that almost all of the performance problems that we see, and definitely all of the bugs that we see, stem from software and configuration problems, not a lack of hardware.

Still, for a little while, I fell into the "too many people, not enough server" mindset on Tuesday.

The problem with your assessment is that its not congruent with the OP's, and the OP is a dev.

The important realization is that Firor is talking about different things, with different solutions, in the same message. One of those things is server capacity, and that was addressed by putting in additional hardware. We know that they did this, that issue is closed, and that they are monitoring it to see if they need to add more.

Hardware is important to improve the ability of the servers to handle basic high concurrency. This is important because the game is at a high point in terms of concurrency. I expect this to be evident by the fact that they have more server resources that they can deploy to handle placement of characters into the game. To me, this means less crowded zones and ability to create more "instances" where they can place players in zones, houses, and dungeons.

I am not expecting that this will resolve the other performance problems that are more related to software. This should have little to no effect on Cyrodiil, for example.

As with most complex things, there is no single answer to what is going on with Performance, and that is where what Firor said merges with the overall Performance Plan that ZOS has been updating us on monthly. What is in that list, and is mentioned in passing by Firor, is a list of other fixes that are not hardware.

It is those performance problems that I think we are all noticing. Input lag, desync, and all of that stuff. Some of this stuff has been going on for months and years, other stuff was introduced with Harrowstorm.

Psiion

Greetings,

Due to the age of this thread, and the fact the information within is outdated, we have closed this thread down. Thank you for your understanding.