How will the AI that monitors our in-game chats effect RP?

Ugrak

SuspensionDispersingAutomaton wrote: »

The humans hired were no more useful than automatic bots, as they banned me for consensual chat among close friends and sent a generic copypasted reply I saw they sent to others affected, with absolutely no consideration for my explanation and reasoning that it was all consensual.

Yeah earlier I mentioned similar experiences with corporate invoice handling systems, using automated matching systems combined with third-party ticket handling services for the anomalies (Capgemini India, I think). People who seemed inundated and pressed for time, with little to no understanding of the context around the invoices, and relying heavily on copy-pasted prompts and replies. Effectively bots (though I think it's more a system fault than with the individuals), and communicating with them felt like trying to write code. Half of it was debugging SAP processes.

I would not be surprised if poor experiences with ESO CS can be largely attributed to the limitations of similar types of communication structures.

TaSheen

Syldras wrote: »

TaSheen wrote: »

"offshore resources reading chats"?

Sounds like the Amazon callcenters operating mainly from India and Pakistan. The employees are usually students and while they are usually very willing to help, language often is a problem.

Well, yes, I know what it means - I just didn't know ZOS used those.... My website reseller host provider's CS agents are in India, for instance, but they told all of their clients when they changed over.

Arunei

@ZOS_Kevin
Thank you for the update, but as others have said, this doesn't really alleviate concerns. People have been getting warnings and suspensions and bans by people who seem to have as little ability to tell context as a bot or AI. That's not reassuring. There shouldn't need to be extensive training for CS to tell what is and isn't harassing/illegal talk.

There needs to be more transparency on this matter. Why was there no announcement that this would be happening? What sorts of terms or phrases are getting people actioned, even in settings they shouldn't be? Why is CS blatantly actioning people when they shouldn't be?

This is not a matter where an agent's personal opinions, biases, or sensibilities should be making decisions. I don't care if a CS agent gets offended by someone's private ERP or two friends bantering and calling each other all sorts of names. That stuff isn't CS' business, making sure no one is being harassed or nothing illegal is being planned is.

brylars

ZOS_Kevin wrote: »

Hi All,

We want to follow up on this thread regarding moderation tools and how this intersects with the role-play community. First, thank you for your feedback and raising your concerns about some recent actions we took due to identified chat-based Terms of Service violations. Since you all raised these concerns, we wanted to provide a bit more insight and context to the tools and process.

As with any online game, our goal is to make sure you all can have fun while making sure bad actors do not have the ability to cause harm. To achieve this, our customer service team uses tools to check for potentially harmful terms and phrases. No action is taken at that point. A human then evaluates the full context of the terms or phrases to ensure nothing harmful or illegal is occurring. A human is always in control of the final call of an action and not an AI system.

That being said, we have been iterating on some processes recently and are still learning and training on the best way to use these tools, so there will be some occasional hiccups. But we want to stress a few core points.

• We are by no means trying to disrupt or limit your role-play experiences or general discourse with friends and guildmates. You should have confidence that your private role-play experiences and conversations are yours and we are not looking to action anyone engaging in consensual conversations with fellow players.
• The tools used are intended to be preventative, and alert us to serious crimes, hate speech, and extreme cases of harm.
• To reiterate, no system is auto-banning players. If an action does occur, it’s because one of our CS agents identified something concerning enough to action on. That can always be appealed through our support ticketing system. And in an instance where you challenge the appeal process, please feel free to flag here on the forum and we can work with you to get to the bottom of the situation.
• As a company we also abide by the Digital Service Act law and all similar laws.

To wrap this up, for those who were actioned, we have reversed most of the small number of temporary suspensions and bans. If you believe you were impacted and the action was not reversed, please issue an appeal and share your ticket number. We will pass it along to our customer service to investigate.

We hope this helps to alleviate any concern around our in-game chat moderation and your role-play experiences. We understand the importance of having safe spaces for a variety of role-play communities and want to continue to foster that in ESO.

We appreciate that ZOS does not "intend" or is not "trying" to disrupt or limit our gameplay, but feeling like something is looming over us does do that. What someone considers "hate speech" is actually subjective. There are few communities that are protected from that in the US. Those who have a religion, for example, are not protected from hate speech, they are protected from discrimination. There is a legal difference.

If there is no auto-banning, then is there auto-suspending? The two are different. Is the human a person who understands American English? I know sometimes I misunderstand and I am an American. English taught outside of this region is very different than what we speak.

Anyway. This sounds like THE response, so we have what we are going to get about it.

We can continue to discuss it to vent frustrations and post instances of continuing occurrences.

dkblight

I don't RP myself, I did in my 20s, but that was some time ago.

Is it possible to create a toggle "check this box" option in the settings of Social, to select that the account RPs?

Like you can select whether or not to harm NPCs, an option to select if you plan to engage in RP in ESO. Which would then segregate those players in a different filtration system less severe than normal chat. Then if chat were to engage with someone who did not choose that setting, a more thorough review can be created.

Selecting the option would change Chat to say RP Say, RP Yell, RP Guild, RP Group, etc.

Players can also toggle it off if they wish to do non-RP related content. Just as I have guildmates toggle off their auto store craftbag when we do farm runs for the guild.

Like I said, I don't RP, but I respect those who do.

Vaqual

Dragonnord wrote: »

From ZOS_Kevin:

"No action is taken at that point. A human then evaluates the full context of the terms or phrases to ensure nothing harmful or illegal is occurring. A human is always in control of the final call of an action and not an AI system.

To reiterate, no system is auto-banning players. If an action does occur, it’s because one of our CS agents identified something concerning enough to action on."

Thank you Kevin. Because several people were blaming ZOS and AI of automatic banning.

As I said, people becoming paranoid with AI.

I hope @StaticWave and @Heren are relieved now.
 

He literally confirmed their system had "hiccups" and actions had to be undone. Whether these humans decided stuff based on snippets with broken context or whatever led to this, the result was still that people were penalized without there being a violated party. This is literally what people were bemoaning, if now AI is behind it, a bot or flawed human action.

The pipeline should be:
offence > report > action
and not:
consensual interaction > action > appeal > 24-96 h customer service processing time > work through bot response 1-4 > pray you get to play again

Are you deliberately trying to paint this in a good light because you picked your opinion before being fully aware of the context in the other thread?

Players being confronted with losing years worth of commitment and money due to a malfunctioning system shouldn't be met with "seems it wasn't AI, so no harm done".

GuuMoonRyoung

I don't know, we humans are so broken right now, even harmless comments are being labeled as "hate speech", "dehumanizing toward protected groups", I don't think AI will do any worse. Maybe it will do better.

Dragonnord

Vaqual wrote: »

Dragonnord wrote: »

From ZOS_Kevin:

"No action is taken at that point. A human then evaluates the full context of the terms or phrases to ensure nothing harmful or illegal is occurring. A human is always in control of the final call of an action and not an AI system.

To reiterate, no system is auto-banning players. If an action does occur, it’s because one of our CS agents identified something concerning enough to action on."

Thank you Kevin. Because several people were blaming ZOS and AI of automatic banning.

As I said, people becoming paranoid with AI.

I hope @StaticWave and @Heren are relieved now.
 

He literally confirmed their system had "hiccups" and actions had to be undone. Whether these humans decided stuff based on snippets with broken context or whatever led to this, the result was still that people were penalized without there being a violated party. This is literally what people were bemoaning, if now AI is behind it, a bot or flawed human action.

The pipeline should be:
offence > report > action
and not:
consensual interaction > action > appeal > 24-96 h customer service processing time > work through bot response 1-4 > pray you get to play again

Are you deliberately trying to paint this in a good light because you picked your opinion before being fully aware of the context in the other thread?

Players being confronted with losing years worth of commitment and money due to a malfunctioning system shouldn't be met with "seems it wasn't AI, so no harm done".

Human errors are everywhere in this life. Thing is AI does not decide anything here, it just takes info and provides it to humans. So, again, people can stop being paranoid with AI.
 

spartaxoxo

If the humans are going to automatically side with the AI, instead of being trained that it's often wrong, then it's not any better than AI doing the moderating.

I hope that this serves as a lesson to the relevant training teams.

1) It is important to understand the context a comment was made and not just what was said. On its own with no context, someone shouting in all caps that they hope someone dies or to kiss off seems like a violation. But, when you realize that nobody flagged it, it was a private instance, and the player was just trash talking Molag Bal before banishing him, suddenly with context you realize that no violation occurred. It's just roleplay.

2) Appeals should be handled by people separate from the mod team. And they should not assume the person appealing is guilty. Mod teams handle a large volume of upsetting speech, it is inevitable that they will make mistakes. Assumptions that the mod team was correct in how they handled it leads to unjust bans sticking around, which have a harmful effect on a player's account.

HatchetHaro

ZOS_Kevin wrote: »

As with any online game, our goal is to make sure you all can have fun while making sure bad actors do not have the ability to cause harm. To achieve this, our customer service team uses tools to check for potentially harmful terms and phrases. No action is taken at that point. A human then evaluates the full context of the terms or phrases to ensure nothing harmful or illegal is occurring. A human is always in control of the final call of an action and not an AI system.

If I might make a suggestion, since the automated checks will always flag these "potentially harmful words or phrases", it means your customer support representatives will have to check through a large number of messages, and therefore also increasing the likelihood of misinterpretation of chat messages and therefore sending out undeserved warnings, suspensions, or bans.

Instead of having automated checks flood the system with false positives, perhaps it'd be better to have the players themselves determine whether a message was harmful. For example, if a player receives a potentially harmful message, they then can determine for themselves whether or not it was harmful to them, and then have the ability to flag those phrases themselves for customer support to review and take action on. I'm thinking they can right click the player's name in the chat for that option. We can call it the "Report Player" button!

spartaxoxo

Also, I hope resources are made available for the mental health of the mod team.

The type of stuff that is NOT flagged by a human but IS very harmful is likely to be highly upsetting e.g criminal communication. The employees need support and training on how to handle it emotionally too.

Syldras

HatchetHaro wrote: »

If I might make a suggestion, since the automated checks will always flag these "potentially harmful words or phrases", it means your customer support representatives will have to check through a large number of messages, and therefore also increasing the likelihood of misinterpretation of chat messages and therefore sending out undeserved warnings, suspensions, or bans.
Instead of having automated checks flood the system with false positives, perhaps it'd be better to have the players themselves determine whether a message was harmful. For example, if a player receives a potentially harmful message, they then can determine for themselves whether or not it was harmful to them, and then have the ability to flag those phrases themselves for customer support to review and take action on. I'm thinking they can right click the player's name in the chat for that option. We can call it the "Report Player" button!

Exactly. Just like it had been before. I'm not sure why it was changed? Is there a legal reason? Youth protection laws or something like that (I'm not sure how it is in the US)?

I'd wager autoflagging leads to a bigger amount of cases mods have to look through than if users have to do the reporting themselves, so it's not exactly less work.

Dragonnord

spartaxoxo wrote: »

If the humans are going to automatically side with the AI, instead of being trained that it's often wrong, then it's not any better than AI doing the moderating.

I hope that this serves as a lesson to the relevant training teams.

Mod teams handle a large volume of upsetting speech, it is inevitable that they will make mistakes.

And how do you (we) know a CS agent made a mistake when actioning a case?

Just because a banned player claims innocence that's enough to say that the CS agent made a mistake?
 

Northwold

Just for clarity, AI = artificial intelligence. Which means computers taking information and processing it to take decisions / act.

That is not what ZOS has described here, which is simply picking up on predetermined words and phrases and flagging them for review by a human being -- a technology that's been around for essentially as long as databases have existed.

So the discussion is certainly one worth having, but the constant references to "AI" are super confusing and not very helpful.

Nharimlur_Finor

FROM:
https://forums.elderscrollsonline.com/en/discussion/665984/so-apparently-people-are-being-banned-in-game-for-using-certain-words-with-their-friends

I'm sure they may have a legal right to do something like this.

What isn't clear is our consumer rights in this situation.

For instance, our right to know some basic information about this system, why it is being implemented and what choices we have with our purchased product.

@ZOS_MattFiror

spartaxoxo

Dragonnord wrote: »

spartaxoxo wrote: »

If the humans are going to automatically side with the AI, instead of being trained that it's often wrong, then it's not any better than AI doing the moderating.

I hope that this serves as a lesson to the relevant training teams.

Mod teams handle a large volume of upsetting speech, it is inevitable that they will make mistakes.

And how do you (we) know a CS agent made a mistake when actioning a case?

Just because a banned player claims innocence that's enough to say that the CS agent made a mistake?
 

Well, Kevin just told us that they overturned the bans. And some of the examples cited were people making risque jokes among friends or trash talking NPCs in private instances. I think Kevin's statement validates those claims considering they just said they overturned most of the unfair bans that happened because the tools are new and training is on going.

Nharimlur_Finor

https://forums.elderscrollsonline.com/en/discussion/comment/8190184/#Comment_8190184

Also, we don't need and shouldn't have to endure another AI chat bot monitor thread, not least because we haven't yet seen any direct evidence of anything beyond automated swear word recognition which we've always known about.

If there's anything new to add to the main thread on this then fine, let's focus on the discussion there, otherwise there'll be no follow-up by ZOS except the locking of the myriad of other secondary threads on the subject.

Who are you to dictate forum behaviour?

Arunei

Dragonnord wrote: »

spartaxoxo wrote: »

If the humans are going to automatically side with the AI, instead of being trained that it's often wrong, then it's not any better than AI doing the moderating.

I hope that this serves as a lesson to the relevant training teams.

Mod teams handle a large volume of upsetting speech, it is inevitable that they will make mistakes.

And how do you (we) know a CS agent made a mistake when actioning a case?

Just because a banned player claims innocence that's enough to say that the CS agent made a mistake?
 

Maybe because Kevin himself outright said they've reversed multiple actions made against accounts? Why would they have done that if they weren't making mistakes or just outright actioning accounts for stuff they didn't like but weren't actually offending the other involved party(ies)?

AvalonRanger

On the other hand....AI ignore mud ball harassment....

Why developer made such a "toy"?

Synapsis123

ZOS_Kevin wrote: »

A human then evaluates the full context of the terms or phrases to ensure nothing harmful or illegal is occurring.

Everything said online is all words so to check for "harmful" things you would have to check to see if anyone was impacted. These bans would handed out and no one was contacted. How do you determine harm without human interaction? If you ban without hearing what people have to say on the matter then you aren't taking the full context into account. Also I know for a fact that my warning wasn't taken in full context because they only looked at the single sentence that got me banned and not the ones before or after it.

Nharimlur_Finor

https://forums.elderscrollsonline.com/en/discussion/comment/8189833/#Comment_8189833

And who told you ZOS is doing anything?

There several post now discussing this issue.
Part of what I said, is that we deserve to know what ZOS is doing.

So I come to the forums saying ZOS is spying us with a satellite and you demand explanation just because me, a random person, said that and I didn't provide any proof?

I can ask ZOS for that proof. How else will the question be answered?

Also, seems you didn't read the part above where I say:

"You don't need AI to monitor that. It's been like that with ZOS for years. There are several certain words that are flagged and can trigger an alert on ZOS side."
I did read it. It wasn't relevant.

Every mmo has that since ever.

That doesn't make is acceptable. Or legal.

IA has nothing to do here.

Transparency is the issue. Paying customers have the right to be able to make informed choices.

It's ZOS' product. If they decide to make changes that impact all players, then all players need to understand the changes and make an informed choice about continuing to use the product, or not.

Elsonso

EnerG wrote: »

This also doesn't answer the question of what kind of program is scrubbing chat to begin with, and how it's affecting latency in the game. Please give a more clear answer about WHAT is sending "concerning" chats to support, and how intensive the system being used is.

Since they have been logging chats for the last 10 years, and the program reviewing the chat logs doesn't even have to be running on the game servers, it is not unreasonable to guess that nothing has changed.

brylars

I think the issue is done.

We asked the queation. We got our official answer.

Nothing will be done because ZOS is denying the accusations and has said it will not change anything.

Since this thread will do nothing more to enact change @ZOS_Kevin please close the thread.

DakhathKilrathi

And that lack of an answer is grounds for dropping subscription for me. "Listen we got it wrong (not that we'll admit that) but appeals totally work" doesn't really cover it, when we all know that appeals do not, in fact, usually work or even happen in an efficient manner.

Syldras

Elsonso wrote: »

Since they have been logging chats for the last 10 years, and the program reviewing the chat logs doesn't even have to be running on the game servers, it is not unreasonable to guess that nothing has changed.

I'd find it interesting to know if something changed or not. Are the many complaints right now just a conincidence? Or was the filter updated and the rules got stricter? An update is possible, after all. Then again I'm not sure whether they'll tell us.

Elsonso

TaSheen wrote: »

"offshore resources reading chats"?

Everyone is "offshore" to someone.

I am not sure where this "offshore" thing is coming from, except from the player above. Does anyone know what they are talking about?

Elsonso

Syldras wrote: »

Elsonso wrote: »

Since they have been logging chats for the last 10 years, and the program reviewing the chat logs doesn't even have to be running on the game servers, it is not unreasonable to guess that nothing has changed.

I'd find it interesting to know if something changed or not. Are the many complaints right now just a conincidence? Or was the filter updated and the rules got stricter? An update is possible, after all. Then again I'm not sure whether they'll tell us.

I meant changed in the "latency" department. Sounds like ZOS has new tools to play with, though. I would not expect that those tools are running on the game servers, but they could be.

Syldras

Elsonso wrote: »

Everyone is "offshore" to someone.
I am not sure where this "offshore" thing is coming from, except from the player above. Does anyone know what they are talking about?

Probably an assumption (one of many here).

TaSheen

Elsonso wrote: »

TaSheen wrote: »

"offshore resources reading chats"?

Everyone is "offshore" to someone.

I am not sure where this "offshore" thing is coming from, except from the player above. Does anyone know what they are talking about?

Syldras wrote: »

Elsonso wrote: »

Everyone is "offshore" to someone.
I am not sure where this "offshore" thing is coming from, except from the player above. Does anyone know what they are talking about?

Probably an assumption (one of many here).

Yeah.... that's what I was thinking myself. I just don't pay much attention as I never chat....

Nharimlur_Finor

Syldras wrote: »

Elsonso wrote: »

Since they have been logging chats for the last 10 years, and the program reviewing the chat logs doesn't even have to be running on the game servers, it is not unreasonable to guess that nothing has changed.

I'd find it interesting to know if something changed or not. Are the many complaints right now just a coincidence? Or was the filter updated and the rules got stricter? An update is possible, after all. Then again I'm not sure whether they'll tell us.

This is the real issue.

Real-world legislation is getting stricter on many IT/data related issues. This may be something in response to that. That doesn't discount the fact that customers need to be advised of the changes and how they could be individually affected.