Activity Finder & Server Performance Update

ZOS_RichLambert
ZOS_RichLambert
Creative Director
It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem. The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.
Rich Lambert
Creative Director - The Elder Scrolls Online
Facebook | Twitter | Google+ | Tumblr | Pinterest | YouTube
Staff Post
  • Hellsbunnies
    Hellsbunnies
    ✭✭✭
    Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.
    Most known on PC EU server as The Samurai Pizza Cat.
    Twitter @HellsbunniesTV
  • redgreensunset
    redgreensunset
    ✭✭✭✭
    Thank you for the information and communication.
  • Ashtaris
    Ashtaris
    ✭✭✭✭✭
    I’ve worked with gaming servers a bit, but certainly I’m no expert at it. Many times you will have different parts of a system broken out into individual servers, like a authentication server for handling logins and game permissions, a game server for handling the main game, data server for updates, etc. All of these can be tied in to each other via the localhost, or actually put on seperate hardware in the Lan. Could something like this be done, for example, as a Group Finder server taking much of the load away from the main game megaserver? Anyway, just asking questions. I’m sure the problems are much more complex than many of us see here as armchair technicians :)
    Edited by Ashtaris on November 25, 2019 4:05PM
  • maboleth
    maboleth
    ✭✭✭✭✭
    The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit.

    @ZOS_RichLambert Thank you for providing the details on this. However, since Activity finder uses/shares system resources of the rest of the game... does it mean:

    - You will upgrade your hardware?
    - Make Activity Finder use different resources?

    Because, to me as a rookie when it comes to servers, it sounds that you're trying to squeeze Activity Finder to work under heavy load without offering more resources? Is it possible after all? Maybe using some low level assembly language. :wink:
  • AngeredDaisy2
    AngeredDaisy2
    ✭✭
    Thank you for the communication.
  • demerdecanswrath
    We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

    Appreciate the update. A lot.

    Can I urge you for an update on the Mac executable as well?

    Hasn't been maintained or updated for a long long time. MoltenVK update (Murkmire) also removed all window controls and there are countless I/O bugs.
    Characters:
    • Arien Larethian (High Elf Templar)
    • Relus Demerdecan (Redguard Dragonknight)
    • Elisia Bonaire (Breton Nightblade)

    Proudly playing and producing content on a 2018 15" MacBook Pro.
  • Sandman929
    Sandman929
    ✭✭✭✭✭
    ✭✭✭
    Cyrodiil performance has been quite bad for a very long time, were the problems before the new group finder attributable to the old group finder?
    Daggerfall Covenant - Xbox One NA
  • Idinuse
    Idinuse
    ✭✭✭✭✭
    Sandman929 wrote: »
    Cyrodiil performance has been quite bad for a very long time, were the problems before the new group finder attributable to the old group finder?

    Yes, and GF has been disabled in Cyrodiil. So what gives with the poor Cyrodiil performance?
    Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium dolorem que laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?
  • Jolipinator
    Jolipinator
    ✭✭✭✭✭
    There's the communication we are looking for.
    PS4 EU.
  • GrimTheReaper45
    GrimTheReaper45
    ✭✭✭✭
    It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

    Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

    The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem. The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

    So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

    I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

    Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

    We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

    I appreciate the post. It is actually very nice to get insight into the problem. For example I was under the impression that the recent spike in crashing was a result of the work on performance fixes. Knowing that just one malfunctioning system is the problem is a relief. Also Is helpful to know more about the internal testing process and where issues can arise there.

    Also, hows that tattoo coming along rich? :D
  • VerboseQuips
    VerboseQuips
    ✭✭✭✭✭
    Thank you for this update, I appreciate that you went a bit in-depth. It was indeed concerning, and knowing that the issues comes from one single place and not from systemic underinvestment does alleviate the concerns. I hope you can fix it all soon. I also only can guess how stressful the recent events must have been for the entire team, and I hope the coming holidays will allow you to rest and reload your batteries. :smile:
    My characters:
    Main and crafter: A Breton magicka templar named Erwann Sorril
    Alt 1: A Bosmer sorcerer named Tuuneleg
    Alt 2: An Imperial dragonknight named Gaius Tullius Hastifer
    Alt 3: An Argonian vampire/nightblade named Observe-le-Xanmeer
    Alt 4: A Nord werewolf/dragonknight named Sigurd Hurlevent
    Alt 5: A Breton sorcerer named Gilian Sorril (he's Erwann's younger brother)
    Alt 6: A Khajiit nightblade named Jolan-dar
    Alt 7: A Nord warden named Sigurmar Hurlevent (he's Sigurd's younger brother)
    Alt 8: An Altmer templar named Oioriel
    Alt 9: An Argonian stamina Warden named Danse-avec-les-Rainettes
    Alt 10: A Redguard templar named Neemokh af-Corelanya
    Alt 11: A Nord stamina sorcerer named Olga Écoute-Vent
    Alt 12: A Breton magicka Warden named Ian Sorril
    Alt 13: A Dunmer magicka necromancer named Ilmoran Dren
    Alt 14: An Orc stamina necromancer named Norgol gro-Borziel
    Main in NA (For collaborative events): A Breton magicka nightblade named Titouan Sorril (long-lost brother of Erwann and Gilian)
  • YoshiPikachu
    YoshiPikachu
    Soul Shriven
    Nice update. Me and many other prknownyou are trying your best to fix the issues.
  • Freddycruz89
    Freddycruz89
    ✭✭✭
    Thanks for the update! I have full confidence the team will figure this out and fix it. Do not let the trolls get to you; the real community waits patiently and appreciates the communications. Good luck! <3
    Octavius Maximus, Founder of The Maester' Order
    PS4 NA. Role Player. Housing Fanatic. Part Time Achievement Hunter.
    | Princely Dawnlight Palace |
Sign In or Register to comment.