Were There Really 100+ QA Testers?

Gabriel_H · July 16

So, a few days ago it seems a FOIA request on the layoffs at ZOS and ZM was published. It included the job title of those affected. Out of the 379 layoffs at both companies ~100 of these were QA Testers - I cannot wrap my head around that number.

I am a content s***. I LOVE new content. I will chew my way through it at breakneck speeds. A Chapter is about 6 hours, a DLC is less. And when I say chew through I mean chew: Story quest, 100% map completion, all WBs, all delves, PD, side quests, all achievements (except the 30 x daily quest ones). I long ago developed an efficient method for taking on content.

In that play-through I'll come across bugs. I report these, from the major to the minor. Now, if I was working as a QA tester, things would be slightly different. The play-through would last longer as I'd be even more meticulous, and writing up the bug reports would also be more lengthy, but I still won't see it taking me more than 12 hours, or 16 hours at the most - 2 work days. That play-through would then need to be repeated multiple times from a QA point of view. Lets say 10 times. That's 20 work days. Obviously I'm only talking non-dungeon content here, but the pattern holds as typically dungeons release at a different time than Overland. Trial have some overlap which I'll circle back to. Both of which take a lot less than 6 hours to clear and see everything.

There are typically months between content releases. Lets take a reasonable value of 3 months with the new model. The old Chapter model was a year/half-year. At 3 months that's 60 work days.With 100 QA Testers that's 6,000 work days. Coming back to trials, obviously you need at least 12 people for a trial. So lets take my 20 work days and multiply - that's 240 work days.

Lets have two groups of testers doing trials, that's 24 people and 480 work days. Alongside them, lets have 12 people on overland and 240 work days. Lets be nice and have a separate dungeon team, in fact lets have 6 of them, being 24 people and 480 work days. All for a grand total 60 people and we are only just over halfway to the 100 and we have a lot, and I mean a lot, of slack in there.

I have a pretty good idea of what a QA tester does. I don't know what else the QA testers were doing. But 100+ does seem like an awful lot of people, and an astonishingly large amount considering how many major bugs make it to Live.

Avran_Sylt · July 16

If you think you know what a QA tester does, then you of course are familiar with regression testing, mate.

You would also be familiar with the advent of massive offshoring/automation efforts in that area that requires automation engineers (and competent ones that still go back and verify the validity of their automation every now and then), but also introduced language barriers and staggard timezones resulting in scheduling conflicts.

You would then hopefully have QA testers that are given agency to try to rectify simple problems, but the more offshoring you do, the more turnover there is, the less likely they are to have such agency/have a desire to do so.

My guess is they axed the highest cost QA testers (seniority/high cost of living area residents), and are aiming to hire additional H1B workers where necessary, offshoring the rest where desired. If it's anything like I've experienced, massive knowledge loss of existing automated tests that eventually stop working/just read green when in reality if the test cases were opened up its obvious they'd be failing every time and it's just an extra PITA to clean up the repository.

I don't know how much of their QA team was actually in this kind of setup before, or not. But I expect there to be a greater number of bugs encountered with every release moving forward, and likely older content quietly failing until caught by players.

Gabriel_H · July 16

AScarlato wrote: »

Gabriel_H wrote: »

AScarlato wrote: »

These armchair developer posts are starting to really be something.

If that's your take, you missed the point.

You literally wrote some long rant about how you would allocate their teams. While saying you don't even know what they do or understand their role in the same post.

What is your point in all of these threads exactly? I'm glad you at least changed your mind from stating a document you haven't read or understood was fake and now using it as your source the next day.

Giving an example of necessary manpower is not even close to allocating teams. I also qualified those statements, with a small caveat that doesn't impact their main roles.

Changed my mind? Where did I say the document was fake? Oh wait, I didn't. I questioned its sourcing given where it appeared, because I think and question before accepting things at face value.

Anyway, on topic. Do you think that the near 50% of the 213 layoffs will actually impact development time, or could it just be they removed unnecessary overhead. Thoughts?

AScarlato · July 16

Having seen QA testers comment on these issues, they often simply lack authority to fix bugs themselves in a timely manner. It's why so many games have "known issues" where patches are pushed out with bugs that someone with actual decision making permits.

We can only guess as to the reasons why. Perhaps in fixing some bugs they aren't able to test if it would create other issues. I'm not a game dev so I won't pretend to know anything for sure, let alone about the dynamics of a specific team.

But it's not always "QA bad at their job because there are bugs." A lot is outside their direct control.

AScarlato · July 16

Gabriel_H wrote: »

Anyway, on topic. Do you think that the near 50% of the 213 layoffs will actually impact development time, or could it just be they removed unnecessary overhead. Thoughts?

You wrote a post that said "The number said 379 - that tells you all you need to know!" without understanding what that number even meant. You dismissed it and then followed up with a bunch of guesses and conjecture that you pulled from thin air.

I do think it will impact development time because it's going to take them weeks according to a prior Mod post to even have a new roadmap. Objectively it's already impacted.

How it plays out, who is still there as opposed to just a number of people, whether they have the same experience and skill set as the team who made Summerset is beyond my knowledge and yours. I guess we will see once they start releasing their new work product. I am skeptical.

hiyde · July 16

That line about having the same size team as the Wrothgar>Summerset era made headlines everywhere. The implication was clear that they are still capable of producing content at that level. It was also a second-hand quote posted by a (highly reliable) tavern attendee. Note that, to this day, it's not being said directly by any ZOS employee on any ZOS platform.

(Zenimax sister-company ID put out a similar statement, but on official platforms.)

If that turns out to be true and we see another amazing era like that, awesome.

If that turns out to not be true and we see new content diminish and diminish, that line will get resurrected everywhere and used as ammunition.

My expectation has been that we'll see less content in the future and I'll be sticking around to see how it all turns out for as long as the game is enjoyable and my 2014-launched guild still wants to be here. The quote above is implying something far grander that I hope they can deliver on (and if not, clarify).

hiyde · July 16

Mattymoo92 wrote: »

[

So there really is no need for them, understandable layoff

Sure, maybe.

Or maybe they need 100 less QA employees because they're going to offshore that work.
Or maybe they need 100 less QA employees because there will be a lot less new content to test moving forward.
Or maybe this is one of the areas they intend to shift to AI.

All guesses, because we, the customer, have no way of knowing unless the company shares that vision with us (which they have no obligation to do).

AScarlato · July 16

Mattymoo92 wrote: »

AScarlato wrote: »

Having seen QA testers comment on these issues, they often simply lack authority to fix bugs themselves in a timely manner. It's why so many games have "known issues" where patches are pushed out with bugs that someone with actual decision making permits.

We can only guess as to the reasons why. Perhaps in fixing some bugs they aren't able to test if it would create other issues. I'm not a game dev so I won't pretend to know anything for sure, let alone about the dynamics of a specific team.

But it's not always "QA bad at their job because there are bugs." A lot is outside their direct control.

So there really is no need for them, understandable layoff

I didn't say that at all. They are probably still needed to find and help diagnose bugs to begin with.

As I said, most games make the decisions to release patches with bugs because it's better to release new content for their player base sooner than it is to wait for things to be perfect and in the meantime lose people to boredom. Compex games are rarely bug free. FFXIV releases a "Known issues" section of nearly all their patch notes.

Others speculate that the fact that they unionized led to their convenient removal - whether there is something there or a tinfoil hat is another issue.

Gabriel_H · July 16

hiyde wrote: »

Mattymoo92 wrote: »

[

So there really is no need for them, understandable layoff

Sure, maybe.

Or maybe they need 100 less QA employees because they're going to offshore that work.
Or maybe they need 100 less QA employees because there will be a lot less new content to test moving forward.
Or maybe this is one of the areas they intend to shift to AI.

All guesses, because we, the customer, have no way of knowing unless the company shares that vision with us (which they have no obligation to do).

Or maybe having nearly 25% of the workforce comprising of QA testers was a massive overkill. Which is the wider point. There seems to have been ~100 let go. We don't know how many are left. But over 100 QA Testers for a game is well beyond industry standards.

Even AAA games on the lead up to release barely reach that number. An established game, with small (compared to the existing game) additions, which have been made even smaller with moving to the Season model, does not need that level of QA testing.

hiyde · July 16

Gabriel_H wrote: »

hiyde wrote: »

Mattymoo92 wrote: »

[

So there really is no need for them, understandable layoff

Sure, maybe.

Or maybe they need 100 less QA employees because they're going to offshore that work.
Or maybe they need 100 less QA employees because there will be a lot less new content to test moving forward.
Or maybe this is one of the areas they intend to shift to AI.

All guesses, because we, the customer, have no way of knowing unless the company shares that vision with us (which they have no obligation to do).

Or maybe having nearly 25% of the workforce comprising of QA testers was a massive overkill. Which is the wider point. There seems to have been ~100 let go. We don't know how many are left. But over 100 QA Testers for a game is well beyond industry standards.

Even AAA games on the lead up to release barely reach that number. An established game, with small (compared to the existing game) additions, which have been made even smaller with moving to the Season model, does not need that level of QA testing.

so many maybes!

Note: The 100 number is a combined number for both ZOS and ZM and there is no data on how many of them worked specifically on ESO.

Eliahnus · July 16

Having been in charge myself of R&D in a big, worldwide European company for many years, I will only say that I concur with OP’s conclusion.
No further comment on the matter.

Syldras · July 16

Gabriel_H wrote: »

I LOVE new content. I will chew my way through it at breakneck speeds. A Chapter is about 6 hours, a DLC is less. And when I say chew through I mean chew: Story quest, 100% map completion, all WBs, all delves, PD, side quests, all achievements (except the 30 x daily quest ones). I long ago developed an efficient method for taking on content.
In that play-through I'll come across bugs. I report these, from the major to the minor. Now, if I was working as a QA tester, things would be slightly different. The play-through would last longer as I'd be even more meticulous, and writing up the bug reports would also be more lengthy, but I still won't see it taking me more than 12 hours, or 16 hours at the most - 2 work days.

I'm curious: What is that efficient method you developed to finish a whole chapter, including all main and side quests, 100% zone completion, and all achievements, meticulously (to quote your last post), in 6 hours?

I've extensively documented my Solstice playthrough last year, and from my personal experience, the main questline alone took over 5 hours (circa 30 minutes per quest) if one listened to all npc dialogue. Just following the questline normally, not trying to find possible glitches, and without taking notes or writing reports (that might double the total amount of time).

ImmortalCX · July 16

hiyde wrote: »

That line about having the same size team as the Wrothgar>Summerset era made headlines everywhere. The implication was clear that they are still capable of producing content at that level. It was also a second-hand quote posted by a (highly reliable) tavern attendee. Note that, to this day, it's not being said directly by any ZOS employee on any ZOS platform.

(Zenimax sister-company ID put out a similar statement, but on official platforms.)

If that turns out to be true and we see another amazing era like that, awesome.

If that turns out to not be true and we see new content diminish and diminish, that line will get resurrected everywhere and used as ammunition.

My expectation has been that we'll see less content in the future and I'll be sticking around to see how it all turns out for as long as the game is enjoyable and my 2014-launched guild still wants to be here. The quote above is implying something far grander that I hope they can deliver on (and if not, clarify).

The quote could be BS (incorrect), or they may have lost their "fire" and are unable to produce at their former pace.

I am optimistic and believe that if well managed, they could deliver excellent yearly content, given the team is similar size to Orsinium era.

MMOs are a dying breed, and having returned to the game near the end of last year, gave me a fresh perspective. The massive quantity of voice acted questlines and deep systems means that ESO could have something to offer anyone who enjoys this genre. The graphics are good enough to not be offputting (with exception of character creator) and improving.

They could actually make good use of AI to improve the product if they are open to that. For instance, I don't know the legality, but they could use AI to rewrite or extend voiced questlines. If they are able to do that, ESO already has a treasure chest of resources rife to be "exploited".

Orbital78 · July 16

The Writhing Wall and various other broken main story lines didn't feel like it over the years. Maybe they didn't have a diverse enough amount of systems or did play like a player. Hard to say now.

Sluggy · July 16

Maybe I'm mistaken here but I thought Zenimax the parent company managed a shared pool of QA testers that were allocated across all projects for all studios and teams? I suspect ZoS would also have a much smaller team of permanent and dedicated in-house testers they manage as well.

ZOS_CouchTato · July 16

We have removed some back and forth from this thread. Please ensure you are all doing your part to keep conversation respectful and constructive, and keep in mind our Community Rules.

baratron · July 16

Gabriel_H wrote: »

hiyde wrote: »

Mattymoo92 wrote: »

[

So there really is no need for them, understandable layoff

Sure, maybe.

Or maybe they need 100 less QA employees because they're going to offshore that work.
Or maybe they need 100 less QA employees because there will be a lot less new content to test moving forward.
Or maybe this is one of the areas they intend to shift to AI.

All guesses, because we, the customer, have no way of knowing unless the company shares that vision with us (which they have no obligation to do).

Or maybe having nearly 25% of the workforce comprising of QA testers was a massive overkill. Which is the wider point. There seems to have been ~100 let go. We don't know how many are left. But over 100 QA Testers for a game is well beyond industry standards.

Even AAA games on the lead up to release barely reach that number. An established game, with small (compared to the existing game) additions, which have been made even smaller with moving to the Season model, does not need that level of QA testing.

I might be misremembering, but I'm sure I read somewhere that ESO was going to gain QA testers from another studio. Can I remember where I read it or which other studio it was? Nope. There have been a lot of news articles about the layoffs over the past week.

But I can certainly believe that expensive unionised ZOS employees could be laid off in favour of cheaper, un-unionised employees from another studio. I have not independently verified this information, but someone who looked at the complete list of laid-off positions said that all but one member of the Zenimax Media union was affected. Which is certainly "interesting" and wouldn't be the least bit legal in any part of Europe whose labour laws I am familiar with...

Gabriel_H · July 17

hiyde wrote: »

Which is certainly "interesting" and wouldn't be the least bit legal in any part of Europe whose labour laws I am familiar with...

I have a friend who works in HR for an American owned company which has European branches. Their days are ... interesting.

virtus753 · July 17

Sluggy wrote: »

Maybe I'm mistaken here but I thought Zenimax the parent company managed a shared pool of QA testers that were allocated across all projects for all studios and teams? I suspect ZoS would also have a much smaller team of permanent and dedicated in-house testers they manage as well.

That was certainly my impression from this interview with people working on Fallout 76, who were described as being assigned to work on FO76 but who are identified as ZeniMax (Media) employees, not Bethesda specifically.

https://kotaku.com/bethesda-zenimax-fallout-76-crunch-development-1849033233

Indeed also has a job listing for QA tester at ZeniMax Media: https://www.indeed.com/cmp/Zenimax-Media/job-titles/Quality-Assurance-Tester

Testers there attest to being assigned to specific games while being employed by ZeniMax Media, but ZeniMax Media doesn't directly make any, as far as I'm aware. So they must be hired by ZeniMax Media and then allocated (and reallocated as necessary) to studios under that umbrella, as you mentioned.

YandereGirlfriend · July 18

Regardless of the numbers and specifics, the game has been plagued by a ridonk number of high-profile bugs for as long as I have been playing it. So something in the QA process was not working properly and probably needed to change (though whether or not this particular change is actually for the better... seems potentially dubious...).

To wit, the ancient Seeker Synthesis set bug is still in the game and still destroying characters that use the set (you get permanently disabled from taking potions unless you follow an incredibly Byzantine recipe for unwinding the bug).

It has been reported since the Apocrypha PTS and never acknowledged or fixed! In four years! The set was a paid feature of the expansion and it literally kills core gameplay on characters that use it! How is that okay??!

Whatever process allows that situation to persist for four years definitely needs to be changed.

NoticeMeArkay · July 18

YandereGirlfriend wrote: »

Regardless of the numbers and specifics, the game has been plagued by a ridonk number of high-profile bugs for as long as I have been playing it. So something in the QA process was not working properly and probably needed to change (though whether or not this particular change is actually for the better... seems potentially dubious...).

To wit, the ancient Seeker Synthesis set bug is still in the game and still destroying characters that use the set (you get permanently disabled from taking potions unless you follow an incredibly Byzantine recipe for unwinding the bug).

It has been reported since the Apocrypha PTS and never acknowledged or fixed! In four years! The set was a paid feature of the expansion and it literally kills core gameplay on characters that use it! How is that okay??!

Whatever process allows that situation to persist for four years definitely needs to be changed.

Hold up, Seeker Synthesis messes up THIS bad?

YandereGirlfriend · July 18

NoticeMeArkay wrote: »

YandereGirlfriend wrote: »

Regardless of the numbers and specifics, the game has been plagued by a ridonk number of high-profile bugs for as long as I have been playing it. So something in the QA process was not working properly and probably needed to change (though whether or not this particular change is actually for the better... seems potentially dubious...).

To wit, the ancient Seeker Synthesis set bug is still in the game and still destroying characters that use the set (you get permanently disabled from taking potions unless you follow an incredibly Byzantine recipe for unwinding the bug).

It has been reported since the Apocrypha PTS and never acknowledged or fixed! In four years! The set was a paid feature of the expansion and it literally kills core gameplay on characters that use it! How is that okay??!

Whatever process allows that situation to persist for four years definitely needs to be changed.

Hold up, Seeker Synthesis messes up THIS bad?

Yep!

As a weird build enthusiast, I've circled back to the set several times over the years and the result is always the same: very quickly upon entering combat, your character will have greyed-out potions permanently and you will be unable to use any additional potions, as if you are stuck in a permanent potion cooldown (likely some timing variable goes negative or some such and breaks the cooldown function).

This will persist even if you un-equip the set and your character will essentially be broken forever unless you summon Customer Service to manually reset you or you unearth the ancient texts required to reverse the bug yourself.

It is beyond wild that your character can become permanently jeopardized just from using a basic set. Yet here we are.

Marto · July 18

I think you're massively understating how bad a game with little or no QA would look like. Any bug or issue you have ever experienced in ESO would be a hundred times worse if the QA team was only 10 people.

The game wouldn't just perform poorly, it would just crash. An item set wouldn't just fail to trigger, it would permanently change your stats. A quest wouldn't just fail to complete, your inventory would get deleted.

Imagining a game without QA is like imagining a city without plumbing. It wouldn't just be bad. It wouldn't exist.

If you ever played a videogame that boots after an update, you're witnessing the work of dozens if not hundreds of QA devs.

Sluggy · July 18

YandereGirlfriend wrote: »

Regardless of the numbers and specifics, the game has been plagued by a ridonk number of high-profile bugs for as long as I have been playing it. So something in the QA process was not working properly and probably needed to change (though whether or not this particular change is actually for the better... seems potentially dubious...).

To wit, the ancient Seeker Synthesis set bug is still in the game and still destroying characters that use the set (you get permanently disabled from taking potions unless you follow an incredibly Byzantine recipe for unwinding the bug).

It has been reported since the Apocrypha PTS and never acknowledged or fixed! In four years! The set was a paid feature of the expansion and it literally kills core gameplay on characters that use it! How is that okay??!

Whatever process allows that situation to persist for four years definitely needs to be changed.

And that's the thing. This is a management-level problem. Either a process that is in use is completely ineffective or someone at a higher level is either not keeping their teams on track or pushing them to focus on other issues. It could also be that they are very backlogged with work which tends to have a compounding effect until the whole thing just grinds to a halt. Either way, the issue should have been identified a LONG long time ago and corrected.

Sluggy · July 18

YandereGirlfriend wrote: »

NoticeMeArkay wrote: »

YandereGirlfriend wrote: »

Regardless of the numbers and specifics, the game has been plagued by a ridonk number of high-profile bugs for as long as I have been playing it. So something in the QA process was not working properly and probably needed to change (though whether or not this particular change is actually for the better... seems potentially dubious...).

To wit, the ancient Seeker Synthesis set bug is still in the game and still destroying characters that use the set (you get permanently disabled from taking potions unless you follow an incredibly Byzantine recipe for unwinding the bug).

It has been reported since the Apocrypha PTS and never acknowledged or fixed! In four years! The set was a paid feature of the expansion and it literally kills core gameplay on characters that use it! How is that okay??!

Whatever process allows that situation to persist for four years definitely needs to be changed.

Hold up, Seeker Synthesis messes up THIS bad?

Yep!

As a weird build enthusiast, I've circled back to the set several times over the years and the result is always the same: very quickly upon entering combat, your character will have greyed-out potions permanently and you will be unable to use any additional potions, as if you are stuck in a permanent potion cooldown (likely some timing variable goes negative or some such and breaks the cooldown function).

This will persist even if you un-equip the set and your character will essentially be broken forever unless you summon Customer Service to manually reset you or you unearth the ancient texts required to reverse the bug yourself.

It is beyond wild that your character can become permanently jeopardized just from using a basic set. Yet here we are.

Yeah, there are numerous (semi)permanent stat altering bugs that have happened over the years. At one point I had a character with a permanent 10%-ish speed boost at all times and another one still to this day has a weird bug were certain weapon types give them a slight increase in stamina regen on the front bar. There are various others too but they are usually so minor that I don't think much about them until I go to make a build and I'm trying to figure out what passive is giving me an unexpected boost or a stat is a little lower than I was aiming for and I can't understand why.