Maintenance for the week of March 25:
• [COMPLETE] ESO Store and Account System for maintenance – March 28, 9:00AM EDT (13:00 UTC) - 12:00PM EDT (16:00 UTC)

Activity Finder & Server Performance Update

ZOS_RichLambert
ZOS_RichLambert
Creative Director
It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem. The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.
Rich Lambert
Creative Director - The Elder Scrolls Online
Facebook | Twitter | Google+ | Tumblr | Pinterest | YouTube
Staff Post
  • kojou
    kojou
    ✭✭✭✭✭
    ✭✭
    Thanks for the hard work guys! Fingers crossed.
    Playing since beta...
  • Turelus
    Turelus
    ✭✭✭✭✭
    ✭✭✭✭✭
    Thanks for the update Rich, some more behind the scenes information is appreciated.

    Many if us want to be patient in regards to the issues, but without some communication as to why things are happening and the details of what has happened it can sometimes feel like the updates that things are being investigated and worked on are a little hollow.

    Hoping the issues can all be fixed up soon and we can all get back to smashing dragons or one another in Cyrodiil.
    Edited by Turelus on November 25, 2019 3:58PM
    @Turelus - EU PC Megaserver
    "Don't count on others for help. In the end each of us is in this alone. The survivors are those who know how to look out for themselves."
  • Evarelia
    Evarelia
    ✭✭✭
    Thank you so much for the thoughtful update and letting us know what's going on behind the scenes!
  • AgaTheGreat
    AgaTheGreat
    ✭✭✭✭✭
    There's nothing about console performance or colsole blue screens (crashes of the game). Shame.
    PS4 EU Aga_The_Grey - retired | PC EU AgaTheGreat
  • jlboozer
    jlboozer
    ✭✭✭✭✭
    This was a very insightful update, thanks for everything you guys do. Even though as players we don't always show it, your hard work put into this game is very appreciated!
    Edited by jlboozer on November 25, 2019 3:55PM
  • pod88kk
    pod88kk
    ✭✭✭✭✭
    ✭✭
    Thank you for the update.

    Is there anyway you could twist the Board of Directors' arms into releasing more money to invest back into the game, eg more server capacity, so that we can fully enjoy the game, rather than paying themselves juicy dividend?
  • CaffeinatedMayhem
    CaffeinatedMayhem
    ✭✭✭✭✭
    ✭✭✭
    This is 100000000% the reality of any software developemnt! Thank you for an excellent summary.
  • Hellsbunnies
    Hellsbunnies
    ✭✭✭
    Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.
    Most known on PC EU server as The Samurai Pizza Cat.
    Twitter @HellsbunniesTV
  • redgreensunset
    redgreensunset
    ✭✭✭✭✭
    Thank you for the information and communication.
  • pod88kk
    pod88kk
    ✭✭✭✭✭
    ✭✭
    Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.

    Lol and you wonder why they call us elitists :D
  • Ashtaris
    Ashtaris
    ✭✭✭✭✭
    ✭✭
    I’ve worked with gaming servers a bit, but certainly I’m no expert at it. Many times you will have different parts of a system broken out into individual servers, like a authentication server for handling logins and game permissions, a game server for handling the main game, data server for updates, etc. All of these can be tied in to each other via the localhost, or actually put on seperate hardware in the Lan. Could something like this be done, for example, as a Group Finder server taking much of the load away from the main game megaserver? Anyway, just asking questions. I’m sure the problems are much more complex than many of us see here as armchair technicians :)
    Edited by Ashtaris on November 25, 2019 4:05PM
  • maboleth
    maboleth
    ✭✭✭✭✭
    The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit.

    @ZOS_RichLambert Thank you for providing the details on this. However, since Activity finder uses/shares system resources of the rest of the game... does it mean:

    - You will upgrade your hardware?
    - Make Activity Finder use different resources?

    Because, to me as a rookie when it comes to servers, it sounds that you're trying to squeeze Activity Finder to work under heavy load without offering more resources? Is it possible after all? Maybe using some low level assembly language. :wink:
  • AngeredDaisy2
    AngeredDaisy2
    ✭✭
    Thank you for the communication.
  • Idinuse
    Idinuse
    ✭✭✭✭✭
    It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

    Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

    The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem.
    The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

    So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

    I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

    Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

    We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

    This being the case, I think you've just provided the full argument for having a separate Eastern Europe / Russian server. IMHO.
    Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium dolorem que laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?
  • NeillMcAttack
    NeillMcAttack
    ✭✭✭✭✭
    Don't worry as they say, to err is to be human, this explains why so many people seem to join the Ebonheart Pact! You've done as you could do and I think cancelling the Undaunted event was a wise (and brave) thing to do. I think most people that were having a hissy fit were people that had bought the game and then played without ever giving another penny. I know this will be an unpopular view but I feel that doesn't give them the right to an opinion (unless they'd only just bought the game) but makes them actually part of the problem, draining resources without contributing to its upkeep.

    So how many months of opinions does a base game purchase grant you? And how many crowns does it cost before ZOS are forced to implement any fixes I deem highest priority?

    Just trying to get a handle on the monetary cost of ones opinions!
    PC EU - NoCP PvP, is real PvP
    Tiidehunter Nord StamDK EP PvP Main
    Legion Commander Tresdin Stamplar DC PvE Main
    Sephirith Altmer MagPlar EP Gondar the Bounty Hunter Khajiit StamBlade DC
    The Dirge Redguard StamNecro EP Disruptor Stormcrafter Nord StamSorc AD
    Lone Druid Bosmer Stam Warden EP Necro-Phos Argonian MagBlade AD
    @ McAttack in game
    Played since beta, and then on console at release, until the game became unplayable on console.
  • TheHsN
    TheHsN
    ✭✭✭✭✭
    Thanks For Update.
    But i will not be happy till i see all problem SOLVED. And i remember MANY OF US told you dont bring the PTS to live server cuz it has huge problems. But you did not listen to us like always.

    Still nothing about,

    Combat changes
    Are you going to change your road Map or same like before ( one step forward then 2 steps back )
    We dont want you to communicate with us when it is LAST RESORT...

    If you want me to keep going with My sub and buy chapters, I want to listen all your plans for 2020... NO PRE ORDER ANYMORE OR SUB before hearing them all


    Issues For me:

    * Cyrodill Lag Issue
    * Combat and Meta Changes Every 2-3 Months (what road map we should expect from you in 2020) Explanation PLS!!!
    * And Wrong Combat Changes Back and Forth and as a result no changes and solved results after all.
    * Class Balances
    * BG finder
    * Bugs that here since 2 - 3 years
    * Communication Between Zos And Players.
    * Skills that not Shot even presses after 2-3 times / Not Working Break Free ETC...

    When we are going to be able to start to play game PROPERLY...

    Cuz Every end of year we are hearing Sorry Posts From you. But it start to make me think that you make it to take our money for next year. And no FIX at all... Then 1 Year we wait to hear From you Rich Lambert.
    Edited by TheHsN on November 25, 2019 6:37PM
    Plays:
    Magicka SORC - PvE/PvP
    Stamina NB - PvE/PvP
    Magicka NB - PvE/PvP
    Magicka Templar - PvE
    Stamina Templar - PvP
    Magicka DK - PvE
    Stamina DK - PvE
  • ZOS_JessicaFolsom
    ZOS_JessicaFolsom
    Community Manager
    There's nothing about console performance or colsole blue screens (crashes of the game). Shame.

    We have fixes for some known memory management issues coming in the next console incremental patch that should help with this.
    Edited by ZOS_JessicaFolsom on November 25, 2019 4:09PM
    Jessica Folsom
    Lead Community Manager - ZeniMax Online Studios
    Facebook | Twitter | Twitch | Tumblr | Instagram | YouTube | Support
    Staff Post
  • demerdecanswrath
    demerdecanswrath
    ✭✭✭
    We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

    Appreciate the update. A lot.

    Can I urge you for an update on the Mac executable as well?

    Hasn't been maintained or updated for a long long time. MoltenVK update (Murkmire) also removed all window controls and there are countless I/O bugs.
    Characters:
    • Arien Larethian (High Elf Templar)
    • Relus Demerdecan (Redguard Dragonknight)
    • Elisia Bonaire (Breton Nightblade)

    Proudly playing and producing content on a 2018 15" MacBook Pro.
  • Sandman929
    Sandman929
    ✭✭✭✭✭
    ✭✭✭✭✭
    Cyrodiil performance has been quite bad for a very long time, were the problems before the new group finder attributable to the old group finder?
  • Idinuse
    Idinuse
    ✭✭✭✭✭
    Sandman929 wrote: »
    Cyrodiil performance has been quite bad for a very long time, were the problems before the new group finder attributable to the old group finder?

    Yes, and GF has been disabled in Cyrodiil. So what gives with the poor Cyrodiil performance?
    Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium dolorem que laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?
  • Ufretin
    Ufretin
    ✭✭✭✭✭
    So the Activity Finder was responsible for the login issues and loading screens on PC-EU?

    That's good to know but it also raises one question:

    Why don't you just deactivate the Activity Finder the next time the server collapses?


    I know it's a desperate measure but not being able to queue for BGs or random dungeons is better than not being able to log into the game at all, isn't it?
  • Jolipinator
    Jolipinator
    ✭✭✭✭✭
    ✭✭
    There's the communication we are looking for.
    PS5 EU.
  • GrimTheReaper45
    GrimTheReaper45
    ✭✭✭✭✭
    It’s time for the November update to our performance improvements roadmap. Before giving the update, we wanted to shed some more light on the recent issues related to the new Activity Finder so that you all have a better understanding of why these issues are happening. Note that completely replacing the old Activity Finder was something we had to do. It was not able to be easily updated and could not handle the load we see on the live servers.

    Before launching the new Activity Finder, the team tried to put the system to as much internal load testing as possible by using automated bots. Even though bots cannot truly simulate player behavior, they are a good tool for identifying basic load problems, and we were able to get significantly more testing bots into the system than players on the live servers and still have the system run smoothly. This gave us confidence to push it to PTS for Update 24. On PTS, new problems were discovered and fixed, but we never saw player concurrencies on PTS as compared to Live. Once the system hit Live, well, we all know that what happened wasn’t good.

    The core issue we’ve been having with the new Activity Finder is that it shares many of the same system resources as the rest of the game. This means that when the Activity Finder gets overloaded, everything else on the server starts to bog down – which is why players experience long loading times, timeouts/disconnects, and sluggish server response at these times. It’s been a challenge for us to identify and fix these issues as each time we go through a cycle of overload, we fix that particular problem, but then discover a new one as new concurrency levels are hit. It is very important to note that almost all of the server issues encountered over the last month were caused by the Activity Finder – it is not a hardware issue or a more widespread problem. The reason why this seems to happen the most on the PC EU server is because it’s our most populated server where people generally all play during the same core hours.

    So, when the most recent Undaunted event started, we started to see server performance issues, even though we had already found and fixed Activity Finder problems during the Witches Festival event. We found and fixed issue after issue during the Undaunted event, but new ones kept cropping up. So we postponed the event and are taking our time now to ensure that we resolve this problem once and for all. We are testing a new build internally, and we fully expect to have this issue resolved.

    I, along with everyone else, appreciate the irony of stating just last year that we wouldn’t have another in-game event focused on the Activity Finder until we were “100% sure” it would work. With the new Activity Finder, we were a lot more confident than with the old one. Obviously, we were wrong.

    Even though it is a small consolation, I do want to make sure everyone understands that the issues you have been seeing are caused by one system going haywire and not a deep systemic or hardware problem. This Activity Finder issue has been plaguing us for over a year now and it is long past time that we fix it.

    We all appreciate everyone’s patience on this, and hopefully we are approaching the end of this process. We haven’t exactly covered ourselves in glory--we know it, and we are doing everything we can to make it better.

    I appreciate the post. It is actually very nice to get insight into the problem. For example I was under the impression that the recent spike in crashing was a result of the work on performance fixes. Knowing that just one malfunctioning system is the problem is a relief. Also Is helpful to know more about the internal testing process and where issues can arise there.

    Also, hows that tattoo coming along rich? :D
  • VerboseQuips
    VerboseQuips
    ✭✭✭✭✭
    Thank you for this update, I appreciate that you went a bit in-depth. It was indeed concerning, and knowing that the issues comes from one single place and not from systemic underinvestment does alleviate the concerns. I hope you can fix it all soon. I also only can guess how stressful the recent events must have been for the entire team, and I hope the coming holidays will allow you to rest and reload your batteries. :smile:
    My characters:
    Main and crafter: A Breton magicka templar named Erwann Sorril
    Alt 1: A Bosmer sorcerer named Tuuneleg
    Alt 2: An Imperial dragonknight named Gaius Tullius Hastifer
    Alt 3: An Argonian vampire/nightblade named Observe-le-Xanmeer
    Alt 4: A Nord werewolf/dragonknight named Sigurd Hurlevent
    Alt 5: A Breton sorcerer named Gilian Sorril (he's Erwann's younger brother)
    Alt 6: A Khajiit nightblade named Jolan-dar
    Alt 7: A Nord warden named Sigurmar Hurlevent (he's Sigurd's younger brother)
    Alt 8: An Altmer templar named Oioriel
    Alt 9: An Argonian stamina Warden named Danse-avec-les-Rainettes
    Alt 10: A Redguard templar named Neemokh af-Corelanya
    Alt 11: A Nord stamina sorcerer named Olga Écoute-Vent
    Alt 12: A Breton magicka Warden named Ian Sorril
    Alt 13: A Dunmer magicka necromancer named Ilmoran Dren
    Alt 14: An Orc stamina necromancer named Norgol gro-Borziel
    Alt 15: A Nord magicka necromancer named Thorgen Givresang
    Alt 16: An Imperial magicka dragonknight named Publius Valeirus Hastifer (Just call him "Valerio" - he's Gaius younger troublemaker of a brother)
    Main in NA (For collaborative events): A Breton magicka nightblade named Titouan Sorril (long-lost brother of Erwann and Gilian)
  • Mettaricana
    Mettaricana
    ✭✭✭✭✭
    ✭✭✭
    I want to believe this is the case but 5 years and counting of never quite nailing the server problem...
  • YoshiPikachu
    Nice update. Me and many other prknownyou are trying your best to fix the issues.
  • Hyzock
    Hyzock
    ✭✭✭
    Yeah that's great and all but how about the cyrodiil performance issues that we've been having since, idk like at least 4 years. Or are we just pretending those aren't a thing anymore?
    PC EU - Frank the Potato - Stam DK
  • React
    React
    ✭✭✭✭✭
    ✭✭✭✭✭
    So can we expect the mayhem PVP event to be held in January? It is the ONLY pvp specific event of the year and with the last one being cancelled, pvp players are being generally disregarded when it comes to in game events. With the overall terrible performance in cyrodiil and the insane amount of combat related bugs in the game now, the least ZOS could do is run the event with a few extra campaigns to allow PVP players to do something refreshing and enjoyable.

    @ReactSlower - PC/NA - 2000+ CP
    React Faster - XB/NA - 1500+ CP
    Content
    Twitch.tv/reactfaster
    Youtube.com/@ReactFaster
  • Freddycruz89
    Freddycruz89
    ✭✭✭
    Thanks for the update! I have full confidence the team will figure this out and fix it. Do not let the trolls get to you; the real community waits patiently and appreciates the communications. Good luck! <3
    Octavius Maximus, Founder of The Maester' Order
    PS4 NA. Role Player. Housing Fanatic. Part Time Achievement Hunter.
    | Princely Dawnlight Palace |
  • Rehdaun
    Rehdaun
    ✭✭✭✭✭
    That's the most honest and up front post I've seen from anyone on your team in a long time!!!
    Thank you @ZOS_RichLambert
Sign In or Register to comment.