Does ZOS Test The Game?

SaintJohnHM · November 2022

Does ZOS test the game internally before release? It seems they might not have the resources to check to see if basic functions such as blocking work, and many folks are still getting game crashes after a few days of this release.

I think many of us wouldn't be so disappointed in these broken releases if we could set our expectations accordingly. From now on, I will know to unschedule any trials and game events until two weeks after a new patch assuming the game will be broken again.

Freelancer_ESO · November 2022

My impression is that they test in sections but, do not test with everything combined very much/at all.

For example, the PTS for Firesong was bugged so that players could not complete the main quest at launch which means they never did a full run through the main quest with the final changes before pushing it out or they decided to do it anyways with it being bugged.

A PTS update that fixed the problem blocking completion of the main quest also had fixes to quests after the quest that blocked progress which means that they had to internally have found the bugs as players could not progress to that point indicating they do test some things internally.

Bucky_13 · November 2022

One option could be that QA catches most of these bugs, informs everyone else internally about it and then gets ignored because fancy features and release schedule is more important than quality and customer satisfaction. Wouldn't be the only company that does this. Considering how ZOS ignores feedback about bugs and other issues from users on PTS, I'd guess this could apply to a lot of the bugs.

SaintJohnHM · November 2022

Ah, so at least they found some of the smaller bugs.

Glaring bugs in fundamental game mechanics like blocking seem like they would have been found easily if someone at ZOS was testing the patch.

I just figured they might be too small of a company to have a testing team.

Danikat · November 2022

They're hiring more QA staff, so they do have a QA department that does test the game.

There's lots of other reasons bugs can get through. For example it might be something that's difficult to fix and isn't serious enough that it's considered worth delaying the release (I suspect that would mainly be things that would fail console certification - because if that happens they can't release the console versions), or it happens one in 1,000 times so it's hard to catch and diagnose the problem (or may not come up at all in internal testing) but once tens of thousands of people are playing at once it seems to come up a lot.

Or it might be something that doesn't happen at all during testing. It sometimes seems like the EU servers have more problems than the US ones, and if all their internal testers are in the USA (likely, because employing people who live in a foreign country adds complications which may not be considered worth dealing with for jobs lots of Americans can do) and their test server is in the USA it's not testing the connection to the EU servers.

Or sometimes there's things that just don't come up. I've seen that with other software in my work. Most memorably once we had a new database which was tested by all 7 end users on the same PCs we'd use for the real version. All we did after that was move it to a different location on the same server and somehow that introduced a bug where clicking on an empty space would ask if you wanted to save and if you clicked yes the program would crash.

Ragnarok0130 · November 2022

I do wonder about this myself. U35 was super buggy and then the bug fixing U36 introduced even more bugs. I'm honestly baffled at how QA keeps missing this stuff.

SaintJohnHM · November 2022

Ragnarok0130 wrote: »

I do wonder about this myself. U35 was super buggy and then the bug fixing U36 introduced even more bugs. I'm honestly baffled at how QA keeps missing this stuff.

Yep. I totally understand if times are tough and they're struggling as a company, I just need to lower my expectations for future releases and plan accordingly.

virtus753 · November 2022

Ragnarok0130 wrote: »

I do wonder about this myself. U35 was super buggy and then the bug fixing U36 introduced even more bugs. I'm honestly baffled at how QA keeps missing this stuff.

There’s an article out there about QA staff working on Fallout 76 and how many of the bugs they identified were known to the developers but not fixed by them in time for release. The process involves many people, and while it clearly failed to deliver a smooth experience here, we have no way of knowing where the process broke down. For all we know, QA caught all of these bugs and crashes. They’re there to report issues to be fixed, but they can’t fix anything themselves.

deleted221106-002999 · November 2022

Any kind of regression testing would appear to be conspicuous in its absence.

jaws343 · November 2022

It's also important to note that in plenty of coding related work, there a significant differences between a Dev environment, a UAT environment, and a live environment, including player load, additional coding for unreleased features, player accounts and world interactions that may not be present in Dev or testing.

All to say, just because something is functioning properly in the Dev or UAT environment, doesn't necessarily mean there won't be problems in live.

I've personally done Dev and UAT testing on features for my company's software only for thing thing I was testing to not work properly in live due to a few minor differences in the environments relating to unreleased features.

blktauna · November 2022

Comrade_Ogilvy wrote: »

Any kind of regression testing would appear to be conspicuous in its absence.

exactly which is a shame.

When I was developing, we could not release until QA signed off and catastrophic bugs ere corrected. Cosmetics were allowed to slide through because they were generally simple and non impactful. I am continually shocked at what happens on releases here.

Maybe be a bit less ambitious?

TechMaybeHic · November 2022

We see PTS versions. Truth is, they don't get enough participation to see everything. Nothing really happens in Cyrodiil during testing as an example. People usually don't have add-ons loaded for PTS is another thing.

I've seen games where PTS is persistent so some players just played there and would notice as there winds up being a community doing regular things rather than just jumping on to see specific things. They could do that here, but there is a problem where even stuff that is tested and reported to ZOS, just doesn't get addressed and I think with their aggressive patch schedule, they are a bit thin on Devs who can keep up with it all within the PTS time frame.

At any rate; I don't think it's anything nefarious or incompetence as much as it is resources. Now their decision making...that's another story

SaintJohnHM · November 2022

Yeah, maybe they do have a QA team, but then the devs just don't do anything about the bugs reported. In any case, it seems we need to lower our expectations of their product.

It's been a few days or major bug reports from players and they still haven't fixed them. If they would just give us back U35 in the meantime we could at least play the game reliably again.

virtus753 · November 2022

TechMaybeHic wrote: »

We see PTS versions. Truth is, they don't get enough participation to see everything. Nothing really happens in Cyrodiil during testing as an example. People usually don't have add-ons loaded for PTS is another thing.

I've seen games where PTS is persistent so some players just played there and would notice as there winds up being a community doing regular things rather than just jumping on to see specific things. They could do that here, but there is a problem where even stuff that is tested and reported to ZOS, just doesn't get addressed and I think with their aggressive patch schedule, they are a bit thin on Devs who can keep up with it all within the PTS time frame.

At any rate; I don't think it's anything nefarious or incompetence as much as it is resources. Now their decision making...that's another story

While some players don’t use add-ons or don’t use as many add-ons on the PTS, the crashes due to NodeDetection were actually reported on PTS this cycle. Only ZOS knows who saw those reports and how far they got in the process of prioritizing, testing internally, and fixing. But it was reported in this case, which speaks to your point about resources and decision making when the developers are given reports from testing.

Destai · November 2022

TechMaybeHic wrote: »

We see PTS versions. Truth is, they don't get enough participation to see everything. Nothing really happens in Cyrodiil during testing as an example. People usually don't have add-ons loaded for PTS is another thing.

I've seen games where PTS is persistent so some players just played there and would notice as there winds up being a community doing regular things rather than just jumping on to see specific things. They could do that here, but there is a problem where even stuff that is tested and reported to ZOS, just doesn't get addressed and I think with their aggressive patch schedule, they are a bit thin on Devs who can keep up with it all within the PTS time frame.

At any rate; I don't think it's anything nefarious or incompetence as much as it is resources. Now their decision making...that's another story

They'd probably get more participation if they didn't require people to buy the newest expansion for PTS access. I'd do a lot of PTS feedback if it didn't require me to buy the newest expansion again, despite having bought it on console. Even with that, there's too many simple things that get posted on PTS that it's value isn't being full realized. Goes back to how community management is run.

Hurbster · November 2022

Get players to do the unpaid testing for them and ignore the reports by the looks of it.

As for bug testing. well I dunno, ZoS is only a huge AAA company so I can understand how it might be tough for them.

SaintJohnHM · November 2022

virtus753 wrote: »

While some players don’t use add-ons or don’t use as many add-ons on the PTS, the crashes due to NodeDetection were actually reported on PTS this cycle. Only ZOS knows who saw those reports and how far they got in the process of prioritizing, testing internally, and fixing. But it was reported in this case, which speaks to your point about resources and decision making when the developers are given reports from testing.

And there are plenty of major U36 bugs that aren't addon related also. I could see if maybe they didn't test all addons and libraries or fix all addon related issues, but fundamental features such as a major block bug should have been caught. Did they change the code that broke blocking after players tested on PTS?

virtus753 · November 2022

SaintJohnHM wrote: »

virtus753 wrote: »

While some players don’t use add-ons or don’t use as many add-ons on the PTS, the crashes due to NodeDetection were actually reported on PTS this cycle. Only ZOS knows who saw those reports and how far they got in the process of prioritizing, testing internally, and fixing. But it was reported in this case, which speaks to your point about resources and decision making when the developers are given reports from testing.

And there are plenty of major U36 bugs that aren't addon related also. I could see if maybe they didn't test all addons and libraries or fix all addon related issues, but fundamental features such as a major block bug should have been caught. Did they change the code that broke blocking after players tested on PTS?

They do change things in the last two weeks, so it’s quite possible. Sometimes they publish those fixes in the live notes; sometimes they don’t, especially if they think it’s backend work that won’t affect us much. Unfortunately I suspect this means their testing teams don’t get a good look at those changes, or they do find them in testing and submit reports but there isn’t time to fix it before live.

I would be very curious to see a list of how many of these issues were known to the developers before release, with plans in place to fix them afterwards. I cannot imagine all the bugs and crashes were completely unknown to anyone at ZOS until we saw them on live. But fundamentally our knowing that won’t change when they actually get fixed, so it’s probably moot.

Dr_Con · November 2022

they're hiring if you're interested.

https://www.zenimaxonline.com/careers

contract qa tester

xDeusEJRx · November 2022

They do test this game quite extensively.

On the live servers where we get to suffer for 2/3 weeks post-launch.

Kingsindarkness · November 2022

SaintJohnHM wrote: »

Ragnarok0130 wrote: »

I do wonder about this myself. U35 was super buggy and then the bug-fixing U36 introduced even more bugs. I'm honestly baffled at how QA keeps missing this stuff.

Yep. I totally understand if times are tough and they're struggling as a company, I just need to lower my expectations for future releases and plan accordingly.

Zenimax is hardly struggling they are owned by the biggest software company in the world Microsoft

They are still doing most of their collaborative work via Zoom and are just now getting back into the office....and honestly given the fact they have turned out two incredibly successful expansion packs and 24 months of content...I would say they are doing pretty well.

I mean look at the utter buggy crap Blizzard has given us by comparison? Or the complete disaster that was/is New World?

I could list a litany of Bugs and facepalm disasters with most other games during this time but I won't bother.

Honestly, Zenimax is no better or worse than most companies today.

BenTSG · November 2022

I am curious the process of testing and PTS feedback works. Surely with everyone testing on the PTS, they would have seen that one quest marker is stuck below the ground making it impossible to do without logging in and out, a NPC breaking the laws of gravity and walking up to a roof, on top of this DK big I have heard about and the blocking, as well as general gameplay change feedback.

Either the update somehow breaks during shipping (I won't pretend like I know the technical side to this to deny it), or the feedback and bugs are indeed ignored.

In any case, this is the last time I myself give them the benefit of the doubt. Covid did cause some havoc back when it was going around, understandably causing some issue with the development of last expansion and maybe this too, but I believe those excises have now run their cause. To me, they've this next year to make it up, and if they don't? Then something is internally wrong with the company, no excuses anymore. They should be back up to full strength again, or at least communicate the reasons why not.

Jaimeh · November 2022

I don't know if they have the resources to test extensively, but even for simple things unrelated to gameplay, like crown store items, we see a lot of errors making their way to the game. For eg., recently there was a piece of furniture in the luxury vendor where the light fell on the opposite side from where it was supposed to. It's a very obvious thing to notice once you place the furnishing because it was a huge beam, and still it made its way to the vendor like that. There's a bunch of examples like this for many items, so based on that I'm inclined to say the process needs more attention. Then for gameplay testing I'm sure they run a lot of tests, but again I can't speak to the extent, because for eg., did no one test Dragoknight in the last PTS? Overall, the game would definitely benefit from more (dev) testing and quality control in design.

SaintJohnHM · November 2022

xDeusEJRx wrote: »

They do test this game quite extensively.

On the live servers where we get to suffer for 2/3 weeks post-launch.

Ha! That seems true. I can't help test, it's too frustrating to play right now and the communication about fixes is really bad so it doesn't seem worth it anyway.

SaintJohnHM · November 2022

Kingsindarkness wrote: »

Honestly, Zenimax is no better or worse than most companies today.

You're right. I was hoping ESO was better than the other games you mentioned, but if the game will be repeatedly broken for weeks at a time, I will lower my expectations.

ZOS_Icy · November 2022

Greetings,

After further review we have decided it best to close this thread as it provides non-constructive commentary. When making threads, we ask that they be civil, constructive, and kept within the Community Guidelines. If there may be any questions in regards to the rules, please feel free to review them here.

Thank you for your understanding.