Again I ask, why are player reported PTS bugs not fixed?

LonePirate · November 2014

I spent some time reviewing and testing all of my bugs from the PTS 1.5 Bug Compilation thread. Amazingly, pretty much every single bug I logged in game and recorded in that thread are still present in the game after 1.5 went live. Bugs written on brand new content such as facial animation, Undaunted pledges and crafting certifications\writs as well as bugs written on items that worked in 1.4 but failed in 1.5 are all still present in the game today, even though many of the issues were reported on the very first day that Update 5 was pushed to PTS server (3 weeks before it went live).

@ZOS_JessicaFolsom @ZOS_GinaBruno What does it take for ZOS to actually fix bugs players report on the PTS? Why should players continue to test content (both old and new) on the PTS when you repeatedly refuse to fix those bugs release after release? For that matter, why do you even have a PTS if you're not going to fix bugs reported by players?

Mumyo · November 2014

!

Zed · November 2014

Third actually but, yeah.

shiva7663 · November 2014

Also, why is the final patch not tested on PTS before release to the Live servers?

Sharee · November 2014

Why do you even have to ask.

If ZOS waited till every single bug ever present on the PTS was fixed before making the patch live, we would get one update every half year, if that. Since content has to be released on a schedule to keep players happy, the patch is going live once it is deemed 'good enough', then more fixes are applied to the live build as they are ready.

Kego · November 2014

The truth?
ZOS just don`t care. In reality they don`t want and need your PTS help. They just do what they want and uses their own QA (lol) for Bugfixing.

KhajitFurTrader · November 2014

It basically goes like this:

Bugs get reported by player(s).
Bug reports get entered into the bug tracking database (BTDB), either as new bugs, or as an existing ones, depending on clarity/quality of players' bug reports.
New bugs are marked as unconfirmed and classified into a severity category depending on actual (not perceived) impact on gameplay and frequency of reports. The more testers report it, the better the chances for a bug getting a higher priority, depending on impact.
New, unconfirmed bugs are scheduled for confirmation by QA, depending on their priority. Again, a detailed bug report with a concise list of steps on how to reproduce it helps with prioritization. A detailed description of how the bug can be triggered also helps a ton with locating the bug in the code base.
Confirmed bugs are reassessed into a severity category, mostly depending on severity of gameplay impact and (potential) number of players affected. Also, if bugs affect new content that's going to be pushed out of the door, they'll get more attention.
Fix development is assigned to developers depending on severity classification. New fixes are collected in batches, which comprise a new build, which in turn is then pushed on internal development/test servers, before going to the PTS.
New builds are scheduled for testing by QA in order to verify that the fixes correctly address the issues they've been programmed for. N.B.: due to time constrictions, not each and every possible side-effect might be detected at this time.
Repeat steps 5. - 7. as necessary.
Bugs that are deemed to be squashed get marked as resolved in the BTDB.
New build gets pushed out on live.
Restart at step 1.

Some things to keep in mind:

This is an ongoing process.
It happens all the time, not just when a new major content patch hits live.
At any time, there are virtually millions of entries in the BTDB (most of which the players have no clue about).
It is simple statistics that hundreds of thousands of players on live can put in more hours into playing and thus are more likely to encounter bugs, than a QA team with a few dozen heads.
Some bugs only occur on live, where concurrency goes into the millions and systems are under constant stress. These conditions are hard to simulate on a test server with a team of less than a hundred people.
Most importantly: there are schedules and deadlines to be met. The reason why a build goes live that still has known bugs, is that development on an MMO never, ever, stops. It'll only stop when the servers will be switched off for good.

Erock25 · November 2014

LonePirate wrote: »

I spent some time reviewing and testing all of my bugs from the PTS 1.5 Bug Compilation thread. Amazingly, pretty much every single bug I logged in game and recorded in that thread are still present in the game after 1.5 went live. Bugs written on brand new content such as facial animation, Undaunted pledges and crafting certifications\writs as well as bugs written on items that worked in 1.4 but failed in 1.5 are all still present in the game today, even though many of the issues were reported on the very first day that Update 5 was pushed to PTS server (3 weeks before it went live).

@ZOS_JessicaFolsom @ZOS_GinaBruno What does it take for ZOS to actually fix bugs players report on the PTS? Why should players continue to test content (both old and new) on the PTS when you repeatedly refuse to fix those bugs release after release? For that matter, why do you even have a PTS if you're not going to fix bugs reported by players?

It is pretty disheartening, isn't it? They should honestly just delete PTS if they're going to ignore bug reporting here. They even ignore little, what I assume to be, quick fix items like fixing the wording on the 2h passive that replaced Arcane Fighter.

LonePirate · November 2014

Erock25 wrote: »

LonePirate wrote: »

I spent some time reviewing and testing all of my bugs from the PTS 1.5 Bug Compilation thread. Amazingly, pretty much every single bug I logged in game and recorded in that thread are still present in the game after 1.5 went live. Bugs written on brand new content such as facial animation, Undaunted pledges and crafting certifications\writs as well as bugs written on items that worked in 1.4 but failed in 1.5 are all still present in the game today, even though many of the issues were reported on the very first day that Update 5 was pushed to PTS server (3 weeks before it went live).

@ZOS_JessicaFolsom @ZOS_GinaBruno What does it take for ZOS to actually fix bugs players report on the PTS? Why should players continue to test content (both old and new) on the PTS when you repeatedly refuse to fix those bugs release after release? For that matter, why do you even have a PTS if you're not going to fix bugs reported by players?

It is pretty disheartening, isn't it? They should honestly just delete PTS if they're going to ignore bug reporting here. They even ignore little, what I assume to be, quick fix items like fixing the wording on the 2h passive that replaced Arcane Fighter.

I am starting to wonder if ZOS actually wants players to report bugs on the PTS. To me, it seems like the PTS is a focus group tool that allows players to provide positive/negative feedback on the content which ZOS largely ignores when it is negative. I do believe ZOS would address extreme game breaking issues if they were reported on the PTS - issues like an install failure or a rampant game crash. Pretty much everything else is put on the back burner until after the update goes live.

It's as if the PTS exists to prove/test the deployment of the update rather than to prove the content actually works how players expect it to work. The PTS is definitely an ideal deployment test tool and in that regard it is successful. Unfortunately, it is a failure if it is intended to serve as a means for players to test the game and its content so that player experience issues can be fixed before release.

I don't think anyone feels ZOS should or could fix all player reported bugs in the window of time between an update's deployment on the PTS and its release to the live server. However, they do need to increase the dismal percentage of player reported bugs they fix before an update goes live. I'm guessing they fix fewer than 5% of player bugs that are reported on the PTS. It's like they need players to report the very same issues on the live server before they are given any legitimacy. The reasons for that baffle me.

I think I safely speak for every player on the PTS when I say we enjoy the game tremendously and we are willing to go that extra mile to help make this game successful. We sacrifice our paid time in the actual game in order to serve as volunteer ZOS employees working on the PTS. All we ask for is to have our issues taken seriously and have them addressed as quickly and as efficiently as possible. That never happens and so I am really struggling to find any benefit to devoting time to the PTS.

Elsonso · November 2014

LonePirate wrote: »

I am starting to wonder if ZOS actually wants players to report bugs on the PTS. To me, it seems like the PTS is a focus group tool that allows players to provide positive/negative feedback on the content

Minus the rhetoric that follows (yeah, go look, I will wait .... done? ok.) I think that ZOS accepts bug reports from us, but that is not the main reason why they use PTS and that is not really what they are looking for when we play.

I think they really want players to be players on the PTS. They want the players to just go play the new stuff and give feedback on what they think. They are not looking for ZOS employee wannabees. They are looking for players who, as players, are willing to give feedback on new features. They want play testers, not free QA staff.

If you come across a bug along the way, or what you think is a bug, drop them a /bug report. They will read it and put it into their system.

The feedback you provide on the new features will be immensely more valuable. What you did, what worked well, what did not work well, what they could do to improve it, and why you feel that way. My feeling is that, from their perspective, this is the main reason for you to be there.

This is why 1.5.2 did not go to PTS before Live. They did not need additional feedback prior to release.

So, get on PTS. Play with the new features, areas, and capabilities. Let them know what you think with constructive feedback. If you want them to change something, tell them why you think they should change it. Present your argument as clearly as you can.

You went on a tangent about them ignoring negative feedback. Keep in mind as you play that they have already decided what they want to do and how they want the features rolled out. By the time it gets to PTS it has been through weeks of design, implementation, and testing. .

It is not that they ignore negative feedback, it is that the feedback needs to be constructive and present a compelling argument that makes them rethink what they are doing. They may simply disagree. They may agree but rework would take too long. They may agree and make immediate changes.

With that in mind, remember to give positive feedback, too. Remember that someone else out there dislikes what you like about the game and they are telling ZOS all about how they should change. Let them know what you like and why they got it right.

xaraan · November 2014

They ask why we don't PTS more - nothing I reported was fixed.

Sacadon · November 2014

KhajitFurTrader wrote: »

It basically goes like this:
Bugs get reported by player(s).

Bug reports get entered into the bug tracking database (BTDB), either as new bugs, or as an existing ones, depending on clarity/quality of players' bug reports.

New bugs are marked as unconfirmed and classified into a severity category depending on actual (not perceived) impact on gameplay and frequency of reports. The more testers report it, the better the chances for a bug getting a higher priority, depending on impact.

New, unconfirmed bugs are scheduled for confirmation by QA, depending on their priority. Again, a detailed bug report with a concise list of steps on how to reproduce it helps with prioritization. A detailed description of how the bug can be triggered also helps a ton with locating the bug in the code base.

Confirmed bugs are reassessed into a severity category, mostly depending on severity of gameplay impact and (potential) number of players affected. Also, if bugs affect new content that's going to be pushed out of the door, they'll get more attention.

Fix development is assigned to developers depending on severity classification. New fixes are collected in batches, which comprise a new build, which in turn is then pushed on internal development/test servers, before going to the PTS.

New builds are scheduled for testing by QA in order to verify that the fixes correctly address the issues they've been programmed for. N.B.: due to time constrictions, not each and every possible side-effect might be detected at this time.

Repeat steps 5. - 7. as necessary.

Bugs that are deemed to be squashed get marked as resolved in the BTDB.

New build gets pushed out on live.

Restart at step 1.

Some things to keep in mind:
This is an ongoing process.

It happens all the time, not just when a new major content patch hits live.

At any time, there are virtually millions of entries in the BTDB (most of which the players have no clue about).

It is simple statistics that hundreds of thousands of players on live can put in more hours into playing and thus are more likely to encounter bugs, than a QA team with a few dozen heads.

Some bugs only occur on live, where concurrency goes into the millions and systems are under constant stress. These conditions are hard to simulate on a test server with a team of less than a hundred people.

Most importantly: there are schedules and deadlines to be met. The reason why a build goes live that still has known bugs, is that development on an MMO never, ever, stops. It'll only stop when the servers will be switched off for good.

The above is exactly why... and no user/player on the planet likes it until they see what happens when the above type of process is NOT followed. When all known bugs are fixed and lumped into a single release (massive risk, delay, cost and likely failure).

Software development is one of the most difficult technology professions to get right and even then "right" is a spectrum and highly subjective. This is offset by more frequent and clear communications with user/players which ZOS has sloowwwly improved upon.

We'll never be happy, but as an organization that's the wrong goal, the goal is to tell us what we're getting, when and why (like it or not) and such things should have a market strategy associated with them. I believe they like any organization that wants to be profitable, and feed themselves and their family's are in fact working to be better. These things just do not happen over night.

We should not be lowering our expectations though and we still need better expectation setting.

LonePirate · November 2014

lordrichter wrote: »

This is why 1.5.2 did not go to PTS before Live. They did not need additional feedback prior to release.

So, get on PTS. Play with the new features, areas, and capabilities. Let them know what you think with constructive feedback. If you want them to change something, tell them why you think they should change it. Present your argument as clearly as you can.

You went on a tangent about them ignoring negative feedback. Keep in mind as you play that they have already decided what they want to do and how they want the features rolled out. By the time it gets to PTS it has been through weeks of design, implementation, and testing. .

It is not that they ignore negative feedback, it is that the feedback needs to be constructive and present a compelling argument that makes them rethink what they are doing. They may simply disagree. They may agree but rework would take too long. They may agree and make immediate changes.

With that in mind, remember to give positive feedback, too. Remember that someone else out there dislikes what you like about the game and they are telling ZOS all about how they should change. Let them know what you like and why they got it right.

I fully understand your POV here. Feedback (not bugs but feedback) of all kinds should be supplied so long as the submitter understands their feedback pretty much stands zero chance of impacting or changing the game, not even six months down the road. For instance, I logged some feedback on how much I hate the new gold drop rate rules. I knew at the time of submission that ZOS would not act on it prior to 1.5 and will likely never act on it because of their general resistance to allowing player opinions override their own. I can grumble about that but ultimately it is OK as the game is their property, not mine.

That being said, as a paying customer, I expect this game to work when they release it. So when they turn over new content on the PTS and it doesn't work, I fully expect ZOS to hold up their end of the bargain or contract and provide a functioning product, especially when I divert my paid time from the live server and log bugs on the PTS play server. For whatever reason, ZOS ignores or refuses to fix the vast majority of legitimate player reported bugs on the PTS prior to an update's release on the live server.

We can hypothesize all day why ZOS does not fix more player reported bugs because they will never publicly admit why this bug fix rate is so dramatically low. Whatever the reason, they prefer to fix player bugs only after they are reported on the live server. Maybe they think the bugs will magically fix themselves between the time the update hits the PTS and when it hits the live server.

That makes me question why the PTS is not called the PFS (Public Feedback Server) because ZOS has proven time and time again that they couldn't care less about player testing on that server. If they truly cared about player reported bugs on the PTS, they would fix more of them. Actions speak louder than words, or in ZOS' case, inaction speaks louder than words.

Gyudan · November 2014

I think the content of update 5 far outweighs the small bugs introduced here and there. Thanks ZoS for not waiting too long to add it on live.

Erock25 · November 2014

Gyudan wrote: »

I think the content of update 5 far outweighs the small bugs introduced here and there. Thanks ZoS for not waiting too long to add it on live.

Small bugs?

That is a joke, right? Stealth breaking on weapon swap. Cyrodiil low pop bonus giving ridiculous amounts. We're getting about 25% of the XP we should be getting for Veteran Content. Auto leveled group dungeons are EXTREMELY out of whack. Taunts don't work all the time. Not all crafting writs working.

Lionxoft · November 2014

KhajitFurTrader wrote: »

It basically goes like this:
[list="basic"]
[*] Some bugs only occur on live, where concurrency goes into the millions and systems are under constant stress. These conditions are hard to simulate on a test server with a team of less than a hundred people.
[*] Most importantly: there are schedules and deadlines to be met. The reason why a build goes live that still has known bugs, is that development on an MMO never, ever, stops. It'll only stop when the servers will be switched off for good.
[/list]

Thanks for explaining basic SDLC (partially). Zenimax needs to become familiar with it a bit more. Good job.

Another problem here is that they don't regression test and I highly doubt they have any form of load automation test plans. Probably a lack of funds or staff but a quality product was promised when the subscription model was announced so that's not a valid excuse. Only they know why the QA process is floundering.

With that said the QA team at Zenimax Online Studios could be hog tied (or doesn't exist) but meeting a deadline while releasing a defective version to production is not wise. They were already a week late with the 4-6 week update cycle commitment so it would have been better to delay it just a bit longer and deliver a version that didn't upset folks even more than a late update would.

I was tempted to set up a TFS or QC environment to keep track of the bugs submitted from myself in addition to the forums and using hard data to exhibit the lack of attention they receive but I can't be bothered with doing free work.

There's still an set bonus(es) bug I reported from 1.2 that is pretty humorous albeit severe. Still waiting for them to resolve that one.

Lionxoft · November 2014

Sharee wrote: »

Why do you even have to ask.

If ZOS waited till every single bug ever present on the PTS was fixed before making the patch live, we would get one update every half year, if that. Since content has to be released on a schedule to keep players happy, the patch is going live once it is deemed 'good enough', then more fixes are applied to the live build as they are ready.

Except it wasn't "good enough". It was no where near "good enough". PvP scoring, VP/XP and stealth break should have been enough to delay the patch. Those three affect the gameplay experience, scoring and character progression while greatly impacting the overall experience of the customers.

With that said. It's expected that severe bugs are handled appropriately before a version is out to production while the less severe bugs that have little to no business impact can be patched up at a later date. This version was released with severe bugs.

Maybe Zenimax doesn't prioritize bugs as severe unless it causes the game to cra- wait... Nevermind.

Thechemicals · November 2014

has anyone been to pts lately? *** is empty.

Goibot · November 2014

Thechemicals wrote: »

has anyone been to pts lately? *** is empty.

Geee I wonder why.....

LonePirate · November 2014

Yeah, there is not much need to be on the PTS when it contains the same code as the live server. Experimenting with a limited resource like gold upgrade mats and trying respecs made possible with all the gold available to new top tier characters are the main reasons to visit the PTS now.

Of course, once Update 6 is deployed (probably in a week or two), the PTS will be busy again, especially in one particular location if player suspicions turn out to be true.

reften · November 2014

The real shame is...you all are doing work for these developers...for free. I'm sure there is a reason why these bugs are not being fixed, gotta get the content out, have to give priority to the big bugs, etc.

But, back to my point, the real shame is a lack of communication from Zos to their FREE, volunteer employees. Obviously you guys that do all this testing and reporting are left in the dark.

"Dear Player,

Thank you for the report, we'll do our best to work on these, but some might still get pushed live. We value your work, blah blah."

Just seems weird. Even a reply to this thread. Takes just a few moments.

demonlkojipub19_ESO · November 2014

The last time they spoke on this they said they give bugs a priority level. Bugs considered stop to progression would be looked at first and fixed before live launch. Clearly none of the reported bugs were at that priority.

But what they say wont fool me. They care more about pushing the pve content through despite the bugs. As long as the chance of a bug preventing progress isnt 100% they will push the patch through. There were many bugs they shouldn't let slide. Momwntum, lethal arrow, undauntes pledge problems and crafting writ problems and weapon swap causing stealtg breaking and occasionally game freeze shouldn't have made it live. Their magical priority system isnt the same as ours or at least mine.

What pisses me off the most is if they could have fixed most of thise glaring bugs we reported in a weeks time, they should have just waited a week, or 2, and released something that worked for more than just themselves and saved a lot of us the frustration. But they have kept with their trend of failing do do so and failing us.

Tiruviel · November 2014

Odd that no ZOS representative has commented upon this thread yet, especially considering how much dissatisfaction it contains, posted by players who have spent considerable time playing on the PTS.