Activity Finder & Server Performance Update

DirkRavenclaw

Thank you very much for this Update and Claws crossed, we will see more and more performance Improvemnst over the Course of next Year

pema

as much as I appreciate the update I notice my sinical "oww the server is down" tone is growing.
The game, especially us ont he EU server are suffering a lot... free play weekend while the game is in a poor state... this is not good for the player base this is not good for us as guilds.

Just a sorry isn't enough, several guildies have quit ESO+ due to all the issues and the lack of fate that ZOS has our best interest... and I can't blame them. The events are a nice things for long time players, something going on. It's also nice for guilds to use as excuse to put up events. Them not working and the way the game is made to easy in endgame (all but trails) or to die hard (vet trails) makes that people leave. I find it very disappointing, especially that there's nothing towards us a community to repent.

Overall some things have been dragging to long, and I appreciate the new flow of actual communication it isn't enough.

mayasunrising

Thank you so much for the update @ZOS_RichLambert , and for being transparent with the community. I personally love the game and believe it has gotten better and better through my 4 years as an ESO+ member. I appreciate the effort you all put in whether it's new content, housing, combat related changes, or fixing hardware and software issues. Keep up the awesome work!

itsfatbass

@ZOS_RichLambert @ZOS_GinaBruno

Just an fyi, the activity finder is still very broken even after the patch on 11/25.

It is unclear from your message whether you believe todays "patch" corrected the issue but it seems like its actually even worse. All the usual issues compounded with ready check bugs, fully formed groups unable to queue, and BG's never filling the roster fully.

Also unable to leave the queue once queued with fully formed group (we were gonna try to requeue). We had to disband. Friend earlier was soloing queuing for random normal dungeon as tank role and it never popped after waiting over an hour he says. I am currently testing the solo queue as both tank AND healer for random normal and it has been 20 minutes with nothing. Same issues, rinse and repeat.

SantieClaws

Many thanks for the update.

Just please don't break Fishmas!

If Brenda causes too much server load then just replace her with a Khajiit. Properly built for the cold weather yes.

Yours with paws
Santie Claws

Tiedän

Thank you for the transparency about the recent performance issues @ZOS_RichLambert

Always appreciate the dedication the team has toward making this game as amazing as it is.

akl77

You didn’t update anything on November improvement roadmap update yet.
All your post is talking about the new issue you guys created the queue finder mess.
What about your first sentence??

OhNoDaedra

Oh thanks now I can to stop buying Skyrim, already have 9 copies

RefLiberty

Idinuse wrote: »

ZOS_RichLambert wrote: »

It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem.

The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

This being the case, I think you've just provided the full argument for having a separate Eastern Europe / Russian server. IMHO.

But are you sure that you would like to see more fragmented regions and less populated servers. I'm fine to play with anyone as long as the game is solidly populated. I think that this is better approach, and would be to get the things running smoothly with current loads then the further player base fragmentation.

Darkhorse1975

Thank you to all at ZOS, this type of communication is much appreciated!

Idinuse

RefLiberty wrote: »

Idinuse wrote: »

ZOS_RichLambert wrote: »

It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem.

The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

This being the case, I think you've just provided the full argument for having a separate Eastern Europe / Russian server. IMHO.

But are you sure that you would like to see more fragmented regions and less populated servers. I'm fine to play with anyone as long as the game is solidly populated. I think that this is better approach, and would be to get the things running smoothly with current loads then the further player base fragmentation.

If what Rich says is accurate, and why wouldn't it, I estimate we would have one West European and one East European server, each roughly equal to the NA server, so yeah. I stand by it.

This sort of overpopulated server mayhem is not doing the game, nor anyone game wise frankly, any service.

kelseaux

@ZOS_RichLambert Please go in depth if at all possible if these things being worked on are going to help the blue screening and disconnecting issues we PS4 PvPers are having in Cyrodiil. It’s almost unplayable in Cyro due to the blue screens and dc’s we are all getting more lately since the Dragonhold DLC.

DragonRacer

kelseaux wrote: »

@ZOS_RichLambert Please go in depth if at all possible if these things being worked on are going to help the blue screening and disconnecting issues we PS4 PvPers are having in Cyrodiil. It’s almost unplayable in Cyro due to the blue screens and dc’s we are all getting more lately since the Dragonhold DLC.

And not just PvPers, although Cyrodiil sounds disproportionately hit, but the increase of blue screen frequency for simple PvE is off the charts. Entering a bank. Simply turning around. Running mounted down a street. Never know when it's gonna strike and it's at least a few times an hour.

MartiniDaniels

Responsiveness improved, bar swap speed really good. Like this patch so far, thanks, there is still hope!

RefLiberty

MartiniDaniels wrote: »

Responsiveness improved, bar swap speed really good. Like this patch so far, thanks, there is still hope!

CiliPadi

It's nice to see developers admit they were wrong.

PrimusNephilim

Thank you Rich

karekiz

My advise? Scrap the entire Automated Group finder system. All of it. Replace it with an Undaunted Board system to form premades with.

zyk

This sounds pretty hopeless. The new Activity Finder which has been in development for over a year shares bottlenecks with the rest of the game?

It's hard to imagine why this is so in 2019 and the fact it is so for a system that has been overhauled is dreadful. If this can't be done properly given so much time, I have little hope for the other technical issues the game has.

There's no reason for a matchmaker system to put strain on the rest of the game.

Meanwhile, before I knew there was a problem:

That might sound like an unreasonable thing to point out, but I don't think so. I think the reason the infrastructure for the rest of the game has suffered is because of the supreme focus on the Crown Store and support for its sprawling catalog, which is the most reliable part of the game.

MartiniDaniels

RefLiberty wrote: »

MartiniDaniels wrote: »

Responsiveness improved, bar swap speed really good. Like this patch so far, thanks, there is still hope!

Man, imo it is better when 80% game works, but 20% have long queue, other then when 100% of game is lag fest. And yes I am active user of group finder, so those long queue times hit me too.

TempPlayer

The main issue is not with the finder but with your concept of a group, or more precisely the fact that you only allow a group of 2DD/1H/1T configuration. This alone cause the DD queue to overflow, which take large amount of memory and the process need to sort through all those entries. And since DD queue takes forever, the DD will just quit and immediately re-qeueu cause they have the impression that this somehow work last time, which again add more load on the finder.
Trying to fix the current finder without rethinking this concept is like fixing a leaking bucket by adding water without patching the hole.
Let's the user choose whether they are okay with a 4DD group, or 3DD/1H, etc. Remove the daily random and just give the bonus to the first dungeon you clear. Show how much people are queueing on each dungeon so that they can choose the one that best fit their interest. Interesting that a lot of load can be delegated to the players and they will sort it out themselves, by just giving them a bit of freedom and info.

Lady_Linux

Idinuse wrote: »

ZOS_RichLambert wrote: »

It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem.

The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

This being the case, I think you've just provided the full argument for having a separate Eastern Europe / Russian server. IMHO.

THis indeed seems to be true. You dont have enough servers to hande the load, you need MOAR SERVERS


WHile i appreciate you saying the issue is with activity finder, and i have no cause to doubt your perception, it seems that the real solution is adding an eastern european/asian server system to handle the abundance of players. Clearly that is what is needed but since you keep trying to use the resources you have to write new code over and over it seems clear that you have either no desire or ability to provide the server resources that are truly needed. I do applaud you commitment to make this thing work with your mad coding skills... there may just not be enough coding skills in the world given the load you are trying to cope with..

woe

Thank you very much for the update.

oddbasket

Thanks, this is a post I appreciate.
Letting your players understand what's going on is more important than continuous fancy events, especially when the problems are prolonged and an explanation was long overdue in place of vague updates.

Abyssmol

Hellsbunnies wrote: »

Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.

Words are cheap! Just write them a fatty check to help with the server issue...

Aznarb

Thank for the update.

Pr0Skygon

It's because of instances like this that I'm mad at The Game Award for not nominating ESO for best multiplayer game.

You know what, I think The Game Award should make a new category, for best Multiplayer game studio, and at the very least give ZOS a nominee. There will surely be a ton of people disagree with my statement, but I'm standing my ground on this.

smee_z

@ZOS_RichLambert

Much appreciated, specially sharing the technical details.

Seems the AF/GF is getting overloaded because it's not as quick to form groups. The sooner you can form groups the sooner you release resources which I would guess spent on the deciding part of forming a group. Throwing more resources would likely still fail as the problem seems to be how to quickly form a group, or resolve AF/GF requests.

KageNin

Pr0Skygon wrote: »

It's because of instances like this that I'm mad at The Game Award for not nominating ESO for best multiplayer game.

You know what, I think The Game Award should make a new category, for best Multiplayer game studio, and at the very least give ZOS a nominee. There will surely be a ton of people disagree with my statement, but I'm standing my ground on this.

But they are bad in every aspect, reward for what, number of promises?

5cript

The honesty is commendable.

But this "oh we will eventually fix it, we are on it, dont worry" just doesnt quite satisfy my worries.

We get an ever clearer picture of what we can expect in the future. And beside content, the future ain't looking great.
Please hire talent like madmen! Please.

Beside that I agree in its entirety with 'zyk'.