The Gold Road Chapter – which includes the Scribing system – and Update 42 is now available to test on the PTS! You can read the latest patch notes here: https://forums.elderscrollsonline.com/en/discussion/656454/
Maintenance for the week of April 15:
• [COMPLETE] ESO Store and Account System for maintenance – April 16, 8:00AM EDT (12:00 UTC) - 6:00PM EDT (22:00 UTC)

How do all these bugs get past the PTS?

SilverBride
SilverBride
✭✭✭✭✭
✭✭✭✭✭
I have always had a lot of respect for ZoS, and I still do, but I am scratching my head about how so many major bugs made it to live.

I thought the purpose of the PTS was not only to get player feedback but mostly to make sure that all changes are working as intended and all bugs worked out before going live with it. So how does this happen?
PCNA
  • kojou
    kojou
    ✭✭✭✭✭
    ✭✭
    Time to resolve bugs > Number of Developers x Number of Hours Developers Work on ESO
    Playing since beta...
  • Elsonso
    Elsonso
    ✭✭✭✭✭
    ✭✭✭✭✭
    kojou wrote: »
    Time to resolve bugs > Number of Developers x Number of Hours Developers Work on ESO bug fixes

    hiefpuiqd5dy.gif

    With one edit, from a position of experience, this is often the case.

    It is also helpful if the people creating the bugs don't do that. :smile:
    Edited by Elsonso on March 11, 2024 7:29PM
    PC NA/EU: @Elsonso
    XBox EU/NA: @ElsonsoJannus
    X/Twitter: ElsonsoJannus
  • jaws343
    jaws343
    ✭✭✭✭✭
    ✭✭✭✭✭
    Unfortunately, development work across different environments is never foolproof.

    Any little thing could cause an unforeseen error. Player load, database size, different coding for each environment, different connections to servers, etc. Since the PTS is not the live server, there is almost no way to account for every possible way the new update will interact with that specific environment.

    A personal example here. My company runs software changes through 3 environemnts. Dev, UAT, Live. The Dev environment has a ton of stuff going on. Not only is it running Live build stuff, it is running various legacy, current, and upcoming features for testing purposes. The UAT is running things that have been deemed functional in Dev but now need user testing. But since the differences between UAT and Dev are vast, feature wise, the chance of bugs or issues is high. And, even though UAT is as close to live as possible, for pure functionality reasons, UAT does not contain the entirety of the live Database. A databases which is adding thousands of entries a day to it. So, no matter what testing happens in UAT, there is always a chance that the database load from Live will cause additional issues once the feature has gone live. And there is a chance that everything will be fine as well. But since none of the 3 environments are exact clones of one another, it is impossible to know until the update is pushed out.

    Basically, same here. PTS and Live are not the same environment. And sometimes, there is just no way to know if something will break on live after PTS testing.
    Edited by jaws343 on March 11, 2024 7:31PM
  • tomofhyrule
    tomofhyrule
    ✭✭✭✭✭
    ✭✭✭
    1. The PTS has a low userbase. Not many players use the PTS, and those that do tend to be content creators showing off the new stuff or players interested in leaving feedback on the new changes. Very few people are going in specifically to bugtest.
    2. PTS bugtesters are doing it out of the goodness of their hearts. And lately, there's been a lot less dev communication on the PTS, so many people who used to don't care to do so anymore
    3. People *have* caught some bugs that did get fixed. There are just a lot more bugs than people expect, and a team of volunteers can't be expected to find all of them.
    4. There is no reason to test some things that didn't get changed, which means that unexpected bugs that pop up there weren't found because why would someone think to run an 8.5 year old zone story for an update that just added a new dungeon on the other side of the map?
    5. It's a bit disheartening to run difficult group content on PTS because not many people are there, and nobody wants to tear their hair out with something super difficult and get no reward for it.
    6. PvP must be planned on PTS because again there are not many people who run it, so the only PvP that tends to get tested is two high-level players in a duel. Cyro/IC/BG PvP content just doesn't have the numbers to get reasonable tests.
    7. There are also some changes that may occur between PTS week 5 and release that didn't go through PTS.

    Basically, a lot of us who use PTS would welcome more people to come in and test things. I know that I personally got very discouraged this cycle due to the lack of communication, and as a result I ended up not retesting a bug I knew about and reported because I couldn't be bothered to spend more time on it. At times, it really does feel like a "hey Streamers! Come show off the new stuff to advertise the new content!" more than a testing server, and that was especially clear this round.

    So to everyone else, please by all means come to help us on PTS. We'd really like more eyes on everything. And we'd absolutely love more dev interaction there.
  • Elsonso
    Elsonso
    ✭✭✭✭✭
    ✭✭✭✭✭
    I do not consider the purpose of PTS to be for bug testing. I consider that the bug testing should have been done before it got to PTS, and late bug fixes will meet up when it releases. Honestly, that is what they get paid to do, and we do not. :smile:

    As far as I am concerned, PTS is for testing the final design and tuning in user hands. It is the opportunity for users to try to break the design, and chime in on things that they want different. Not that ZOS is obligated to follow said suggestions. :neutral:

    Edited by Elsonso on March 11, 2024 7:38PM
    PC NA/EU: @Elsonso
    XBox EU/NA: @ElsonsoJannus
    X/Twitter: ElsonsoJannus
  • sarahthes
    sarahthes
    ✭✭✭✭✭
    Elsonso wrote: »
    I do not consider the purpose of PTS to be for bug testing. I consider that the bug testing should have been done before it got to PTS, and late bug fixes will meet up when it releases. Honestly, that is what they get paid to do, and we do not. :smile:

    PTS is for testing the final design and tuning in user hands. It is the opportunity for users to try to break the design, and chime in on things that they want different. Not that ZOS is obligated to follow said suggestions. :neutral:

    Agree with this. Most of the bugs that are found on live are only found because of the sheer diversity of client configurations/builds and playerbase. The likelihood of finding many of them on the pts is very low, and when its a really weird interaction the likelihood of finding it in QA is even lower.
  • jaws343
    jaws343
    ✭✭✭✭✭
    ✭✭✭✭✭
    sarahthes wrote: »
    Elsonso wrote: »
    I do not consider the purpose of PTS to be for bug testing. I consider that the bug testing should have been done before it got to PTS, and late bug fixes will meet up when it releases. Honestly, that is what they get paid to do, and we do not. :smile:

    PTS is for testing the final design and tuning in user hands. It is the opportunity for users to try to break the design, and chime in on things that they want different. Not that ZOS is obligated to follow said suggestions. :neutral:

    Agree with this. Most of the bugs that are found on live are only found because of the sheer diversity of client configurations/builds and playerbase. The likelihood of finding many of them on the pts is very low, and when its a really weird interaction the likelihood of finding it in QA is even lower.

    Yeah, I think people don't consider the "steps" aspect to some bugs. Like, when your sample size is 500 people, and a bug isn't found on PTS because none of those 500 people performed the X, Y, Z steps to make the bug happen. And then on live, with 10K players, a handful always jump in the air twice before getting on their mount, and suddenly the game crashes. Of course that is something that no one is going to test on PTS because why would they.
  • WhiteCoatSyndrome
    WhiteCoatSyndrome
    ✭✭✭✭✭
    ✭✭✭✭
    1. It's a bit disheartening to run difficult group content on PTS because not many people are there, and nobody wants to tear their hair out with something super difficult and get no reward for it.

    This. I tried soloing the new dungeons on normal but that isn’t anywhere close to the usual dungeon experience, on top of being very slow and tedious.

    That in mind, if anyone wants to sign up to help test the next Chapter’s Trial when it gets released to PTS in a month or so, I’ll open a thread for it right now. I’d love to help test the group content more thoroughly but I’m not a group. 🙃
    #proud2BAStarObsessedLoony
    PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!
    A useful explanation for how RNG works

    PC/NA ROLLBACKS AND BAN NOTIFICATIONS ANNOUNCEMENT.
  • Veryamedliel
    Veryamedliel
    ✭✭✭
    jaws343 wrote: »
    And then on live, with 10K players, a handful always jump in the air twice before getting on their mount, and suddenly the game crashes. Of course that is something that no one is going to test on PTS because why would they.
    I agree on the sentiment. Of course the first question should not be 'why do people jump twice before mounting, but 'why would/should/does that break anything?'

    I understand you can't catch every bug before it bites. I've been in the software business too long for that. But.. and I'll pick one at random here.. The disappearing achievements bug. This is something that does not depends on a player. There is nothing a player should be able to do to cause such a bug. It cannot and should not depends on hardware or add-ons either. Point is, if you mess with something, say a database, you check it for possible errors to whatever relates and depends on it. That's not something you do by hand, you can easily automate checks like that. Bugs like this can easily be avoided with some basic gameplay. You log in, go through a checklist you made beforehand (you know what's changed, so it can't be that hard to make), do a dungeon and trial, check if the drops drop as they should, if mechanics work as they should.. the basic stuff every designer should test before it even hits a PTS server. This is basic quality control.
    Edited by Veryamedliel on March 11, 2024 8:00PM
  • Sleep724
    Sleep724
    ✭✭✭✭✭
    PTS is the Alpha test. Now you’re on the beta test. Thanks for helping with the bug reports for the full release! Coming soon!
  • jaws343
    jaws343
    ✭✭✭✭✭
    ✭✭✭✭✭
    jaws343 wrote: »
    And then on live, with 10K players, a handful always jump in the air twice before getting on their mount, and suddenly the game crashes. Of course that is something that no one is going to test on PTS because why would they.
    I agree on the sentiment. Of course the first question should not be 'why do people jump twice before mounting, but 'why would/should/does that break anything?'

    I understand you can't catch every bug before it bites. I've been in the software business too long for that. But.. and I'll pick one at random here.. The disappearing achievements bug. This is something that does not depends on a player. There is nothing a player should be able to do to cause such a bug. It cannot and should not depends on hardware or add-ons either. Point is, if you mess with something, say a database, you check it for possible errors to whatever relates and depends on it. That's not something you do by hand, you can easily automate checks like that. Bugs like this can easily be avoided with some basic gameplay. You log in, go through a checklist you made beforehand (you know what's changed, so it can't be that hard to make), do a dungeon and trial, check if the drops drop as they should, if mechanics work as they should.. the basic stuff every designer should test before it even hits a PTS server. This is basic quality control.

    Yeah, my jumping and mounting example was a bit hyperbolic, intended to represent the unpredictability of end-user engagement with the content. Which makes identifying bugs ahead of time challenging. Developer A expects a specific interaction with content, User A does something unexpected. Now there is a bug to figure out.

    As to the achievement stuff, I agree it likely isn't user related. And probably a lot more to do with the differences between the Live and PTS environment. They are not exact clones of one another, by the very nature of pointing to different servers. So while something as simple as a line of code telling the PTS server to read database A to pull in achievements functions in PTS alongside the other changes, a single variable of that might not function when ported to Live. Or, that call to database A fails under the load of live, where it wouldn't fail in PTS due to a much lower # of players participating.

    It is also important to note that going through expected changes is exactly how bugs get missed. You make a change to feature X. So you go in and test Feature X, and hey, everything is great. Feature X is awesome. Unfortunately, Function Y had a dependency downstream in a system that had to be changed slightly to accommodate Feature X. Well, no one tested Function Y, because it wasn't supposed to be changed. And, since the user base on PTS is low, none of the users on PTS actually use Function Y in the way that would cause the bug that shows up on live.

  • tsaescishoeshiner
    tsaescishoeshiner
    ✭✭✭✭✭
    Some of the bugs were not on the PTS as far as I'm aware, like the achievement and outfit slot issues. I didn't notice them there.

    Concerns like the font UI were brought up by some people, but I'd assume it's a change they want to move forward with anyway.

    Things like the new Gold Road screen didn't come up--I actually think most people like it or don't mind it. Some complaints get noticed more when they hit Live.
    PC-NA
    in-game: @tsaescishoeshiner
  • chessalavakia_ESO
    chessalavakia_ESO
    ✭✭✭✭✭
    I have always had a lot of respect for ZoS, and I still do, but I am scratching my head about how so many major bugs made it to live.

    I thought the purpose of the PTS was not only to get player feedback but mostly to make sure that all changes are working as intended and all bugs worked out before going live with it. So how does this happen?

    The PTS has a relatively low player count that are generally interested in specific areas of the game.

    As a result, it's easy for the PTS testers to miss things that either aren't in areas where they are interested or don't always occur.

    You can also hit situations where it's hard to be sure what counts as a bug vs decision and you don't want to be the person that complains about every little thing under the sun.
  • Grega
    Grega
    ✭✭✭✭✭
    I have always had a lot of respect for ZoS, and I still do, but I am scratching my head about how so many major bugs made it to live.

    I thought the purpose of the PTS was not only to get player feedback but mostly to make sure that all changes are working as intended and all bugs worked out before going live with it. So how does this happen?

    It’s simple. Quality is either good or bad. Sometimes is good. In this case, it’s bad. 🤷‍♂️ such is life.

    I go to Panda Express usually about 6 times a month. 5 times is usually good, once or twice maybe bad.

    Was chef lazy that once? Maybe. Maybe sick and someone else subbed in who didn’t know better. Could be hundreds of reasons. But I don’t really stop and waste my time with how/why. It’s done anyway. Best to move on 🙂
    Edited by Grega on March 11, 2024 8:53PM
  • Soarora
    Soarora
    ✭✭✭✭✭
    ✭✭✭
    Somehow, most of these bugs weren’t on the PTS. Or if they were, I never ran into them despite playing most PTS weeks. I did report bugs that never got resolved, though.
    PC/NA Dungeoneer (Tank/DPS/Heal), Trialist (DPS/Tank/Heal), and amateur Battlegrounder (DPS) with a passion for The Elder Scrolls lore
  • duagloth
    duagloth
    ✭✭✭✭
    Pts has been historically just a way for developers too let players know what's going live. Feedback is ignored anyways.
Sign In or Register to comment.