Activity Finder & Server Performance Update

ZOS_RichLambert

It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem. The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

kojou

Thanks for the hard work guys! Fingers crossed.

Turelus

Thanks for the update Rich, some more behind the scenes information is appreciated.

Many if us want to be patient in regards to the issues, but without some communication as to why things are happening and the details of what has happened it can sometimes feel like the updates that things are being investigated and worked on are a little hollow.

Hoping the issues can all be fixed up soon and we can all get back to smashing dragons or one another in Cyrodiil.

Evarelia

Thank you so much for the thoughtful update and letting us know what's going on behind the scenes!

AgaTheGreat

There's nothing about console performance or colsole blue screens (crashes of the game). Shame.

jlboozer

This was a very insightful update, thanks for everything you guys do. Even though as players we don't always show it, your hard work put into this game is very appreciated!

pod88kk

Thank you for the update.

Is there anyway you could twist the Board of Directors' arms into releasing more money to invest back into the game, eg more server capacity, so that we can fully enjoy the game, rather than paying themselves juicy dividend?

CaffeinatedMayhem

This is 100000000% the reality of any software developemnt! Thank you for an excellent summary.

Hellsbunnies

Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.

redgreensunset

Thank you for the information and communication.

pod88kk

Hellsbunnies wrote: »

Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.

Lol and you wonder why they call us elitists

Ashtaris

I’ve worked with gaming servers a bit, but certainly I’m no expert at it. Many times you will have different parts of a system broken out into individual servers, like a authentication server for handling logins and game permissions, a game server for handling the main game, data server for updates, etc. All of these can be tied in to each other via the localhost, or actually put on seperate hardware in the Lan. Could something like this be done, for example, as a Group Finder server taking much of the load away from the main game megaserver? Anyway, just asking questions. I’m sure the problems are much more complex than many of us see here as armchair technicians

maboleth

ZOS_RichLambert wrote: »

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit.

@ZOS_RichLambert Thank you for providing the details on this. However, since Activity finder uses/shares system resources of the rest of the game... does it mean:

- You will upgrade your hardware?
- Make Activity Finder use different resources?

Because, to me as a rookie when it comes to servers, it sounds that you're trying to squeeze Activity Finder to work under heavy load without offering more resources? Is it possible after all? Maybe using some low level assembly language.

AngeredDaisy2

Thank you for the communication.

Idinuse

ZOS_RichLambert wrote: »

It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem.

The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

This being the case, I think you've just provided the full argument for having a separate Eastern Europe / Russian server. IMHO.

NeillMcAttack

Hellsbunnies wrote: »

Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.

So how many months of opinions does a base game purchase grant you? And how many crowns does it cost before ZOS are forced to implement any fixes I deem highest priority?

Just trying to get a handle on the monetary cost of ones opinions!

TheHsN

Thanks For Update.
But i will not be happy till i see all problem SOLVED. And i remember MANY OF US told you dont bring the PTS to live server cuz it has huge problems. But you did not listen to us like always.

Still nothing about,

Combat changes
Are you going to change your road Map or same like before ( one step forward then 2 steps back )
We dont want you to communicate with us when it is LAST RESORT...

If you want me to keep going with My sub and buy chapters, I want to listen all your plans for 2020... NO PRE ORDER ANYMORE OR SUB before hearing them all


Issues For me:

* Cyrodill Lag Issue
* Combat and Meta Changes Every 2-3 Months (what road map we should expect from you in 2020) Explanation PLS!!!
* And Wrong Combat Changes Back and Forth and as a result no changes and solved results after all.
* Class Balances
* BG finder
* Bugs that here since 2 - 3 years
* Communication Between Zos And Players.
* Skills that not Shot even presses after 2-3 times / Not Working Break Free ETC...

When we are going to be able to start to play game PROPERLY...

Cuz Every end of year we are hearing Sorry Posts From you. But it start to make me think that you make it to take our money for next year. And no FIX at all... Then 1 Year we wait to hear From you Rich Lambert.

ZOS_JessicaFolsom

AgaTheGreat wrote: »

There's nothing about console performance or colsole blue screens (crashes of the game). Shame.

We have fixes for some known memory management issues coming in the next console incremental patch that should help with this.

demerdecanswrath

ZOS_RichLambert wrote: »

We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

Appreciate the update. A lot.

Can I urge you for an update on the Mac executable as well?

Hasn't been maintained or updated for a long long time. MoltenVK update (Murkmire) also removed all window controls and there are countless I/O bugs.

Sandman929

Cyrodiil performance has been quite bad for a very long time, were the problems before the new group finder attributable to the old group finder?

Idinuse

Sandman929 wrote: »

Cyrodiil performance has been quite bad for a very long time, were the problems before the new group finder attributable to the old group finder?

Yes, and GF has been disabled in Cyrodiil. So what gives with the poor Cyrodiil performance?

Ufretin

So the Activity Finder was responsible for the login issues and loading screens on PC-EU?

That's good to know but it also raises one question:

Why don't you just deactivate the Activity Finder the next time the server collapses?

I know it's a desperate measure but not being able to queue for BGs or random dungeons is better than not being able to log into the game at all, isn't it?

Jolipinator

Jolipinator wrote: »

There's the communication we are looking for.

GrimTheReaper45

ZOS_RichLambert wrote: »

It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem. The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

I appreciate the post. It is actually very nice to get insight into the problem. For example I was under the impression that the recent spike in crashing was a result of the work on performance fixes. Knowing that just one malfunctioning system is the problem is a relief. Also Is helpful to know more about the internal testing process and where issues can arise there.

Also, hows that tattoo coming along rich?

VerboseQuips

Thank you for this update, I appreciate that you went a bit in-depth. It was indeed concerning, and knowing that the issues comes from one single place and not from systemic underinvestment does alleviate the concerns. I hope you can fix it all soon. I also only can guess how stressful the recent events must have been for the entire team, and I hope the coming holidays will allow you to rest and reload your batteries.

Mettaricana

I want to believe this is the case but 5 years and counting of never quite nailing the server problem...

YoshiPikachu

Nice update. Me and many other prknownyou are trying your best to fix the issues.

Hyzock

Yeah that's great and all but how about the cyrodiil performance issues that we've been having since, idk like at least 4 years. Or are we just pretending those aren't a thing anymore?

React

So can we expect the mayhem PVP event to be held in January? It is the ONLY pvp specific event of the year and with the last one being cancelled, pvp players are being generally disregarded when it comes to in game events. With the overall terrible performance in cyrodiil and the insane amount of combat related bugs in the game now, the least ZOS could do is run the event with a few extra campaigns to allow PVP players to do something refreshing and enjoyable.

Freddycruz89

Thanks for the update! I have full confidence the team will figure this out and fix it. Do not let the trolls get to you; the real community waits patiently and appreciates the communications. Good luck!

Rehdaun

That's the most honest and up front post I've seen from anyone on your team in a long time!!!
Thank you @ZOS_RichLambert