How will the AI that monitors our in-game chats effect RP?

Syldras

Ugrak wrote: »

I do not know that the chat functions of a MMO qualifies for such legal protection.

Not sure how it is in the whole EU, but in my country, every type of digital communication also falls under that law, including private messages and chats.

Ugrak wrote: »

However it did spark a question in my mind. You (playing on EU; possibly being a citizen of the EU also) previously in this thread mention being surprised to not have been sanctioned.
While I haven't looked into where other players who report being sanctioned are coming from, it seems relevant to mention EU specific legal frameworks such as the EU AI act and GDPR as possible explanations, as several major AI services do not operate in the EU market (or at least postpone deployment) specifically because of this.

This is an interesting aspect. Now, in case someone who was sanctioned wants to disclose whether they're from...? Although I hope that the official statement we'll hopefully get on Monday will clarify things.

SilverBride

Being told that this will be looked into and investigated rather than "No we aren't invading players privacy" speaks volumes.

Skarphedinn

ZOS_Kevin wrote: »

Hi all, just wanted to chime in here. We’re looking into some of the questions in the thread and checking in with the team for feedback. Since it’s pretty late in the day on a Friday, we probably won’t have any feedback until earlier next week. But wanted to acknowledge that we’ve seen this and are investigating.

For now, anyone with ban issues, please make sure to put in an appeal and share your ticket number. Happy to pass those along.

REMOVED THE MOD EDIT FROM THIS POST.

Please report this to the GDPR. They are now removing post calling for people to be held responsible to the proper authorities.

This is something that you should be looking into now, not later. What a completely dystopian thing to do.

Syldras

Skarphedinn wrote: »

[snip]

They wouldn't be shut down. But they'd have to pay a fine.

[edited to remove quote]

spartaxoxo

SilverBride wrote: »

Being told that this will be looked into and investigated rather than "No we aren't invading players privacy" speaks volumes.

You can say that again

AzuraFan

Wow. A bot should not be monitoring and reporting on private conversations. Unless a person reports it, nothing should be done. What's next? Monitoring Discord?

It's hilarious that in a game that has us running around killing people, and that includes the thieves guild, the dark brotherhood, slavery, and necromancy, they are concerned about what's being said in private chat between consenting adults.

Sleepsin

AzuraFan wrote: »

Wow. A bot should not be monitoring and reporting on private conversations. Unless a person reports it, nothing should be done. What's next? Monitoring Discord?

It's hilarious that in a game that has us running around killing people, and that includes the thieves guild, the dark brotherhood, slavery, and necromancy, they are concerned about what's being said in private chat between consenting adults.

Do you really think Discord is not being monitored?

LadyGP

spartaxoxo wrote: »

SilverBride wrote: »

Being told that this will be looked into and investigated rather than "No we aren't invading players privacy" speaks volumes.

You can say that again

Similar to the performance thread thats lasted 4 months.

Ostonoha

CoolBlast3 wrote: »

If this is intentional and/or isn't rolled back entirely I'm kinda done. I like ESO as a game, but a vast majority of my 10000 hours of playtime are RP and my purchase power goes directly to stuff I can use in RP. With this, I can no longer RP without fear of being banned for calling a fellow RPer's character stupid in roleplay. So I'll no longer spend money on the game. Simple as.

I'm not a RPer in the common sense but just by playing this game I do think we are all RPing to a degree. That being said I also have this issue of fearing that one slight comment could be seen either out of context or taken differently and it makes me not want to participate in the game. I personally think ZOS has been a bit heavy handed of late when it comes to their regulations. They seem to have been on the warpath against character names and guild names. even ones that had been around for YEARS. But now all of a sudden many of them are bad? And rather than saying you must change the name outright banning people or suspending without given a chance to correct their actions. Now according to this thread they are looking to be more heavy handed on in-game chat. All it makes me think is of finding some other game or thing to do. We have a profanity filter. If people circumvent the filter I understand a warning or ban. But if I go into options and willingly turn the filter off. I know what I signed up for.
I may be old school coming from games like SWG where the forums, and in game chat essentially were the wild west (and much better forums because of it)
At some point why even use an open forum discussion area or an in game chat. I am even worried this amount of criticism of the game could lead to a suspension or warning.

Ugrak

Before we get too fired up over the automated moderation issue, it might be worth remembering that while the TOS states AI may be in the loop, it is not clear to what extent and what exactly is going on (aside from apparent excessive moderation).

It might as well be an automated flagging system that feeds into the workflow of a third-party human reviewer (the "Associates in India" meaning of AI ). In that case the real issue is either the policies they have been instructed to enforce, or potentially doing it with insufficient care.

Then again, it may be unrealistic to assume this month's new hire at Third-Party Consulting has sufficient familiarity to exercise much judgement beyond the very basics surrounding compliance with TOS. In such a case it may not even be fit for purpose. Though I'm getting far ahead of myself here as I really don't know how ESO actually handles game moderation in terms of infrastructure and such.

Having worked with AI invoice handling systems in multinational corporations, involving outsourcing to several third parties, I have an appreciation of how imperfect such processeses can become. Especially whenever unstandardized data brings a need for contextual judgement into the picture.

As an aside tangent, all this reminded me of the Roman goddess Justitiae, and how her "justice is blind" being an exhortation to juries to remember that the moral responsibility for reining it in rests on them.

Bethgael

If this is a new form of monitoring, then it would explain the recent lag spikes.

ZOS, if your game can't be expanded in, say, housing or PvP because of "technical limitations", it's a bit disingenuous to be using a bot that monitors private conversations.

Skarphedinn

Good to see the mods are taking this seriously and snipping post saying to report it to the GDPR.

Ugrak

Some other food for thought: if there really is gamewide automated monitoring and reporting of all chat streams with no human input, there is still the question whether this is an intentional design, or whether it is a bug. i.e., that a functionality perhaps intended to only assist in the review of reported accounts is running amuck.

AzuraFan

Sleepsin wrote: »

AzuraFan wrote: »

Wow. A bot should not be monitoring and reporting on private conversations. Unless a person reports it, nothing should be done. What's next? Monitoring Discord?

It's hilarious that in a game that has us running around killing people, and that includes the thieves guild, the dark brotherhood, slavery, and necromancy, they are concerned about what's being said in private chat between consenting adults.

Do you really think Discord is not being monitored?

I meant by ZOS.

LadyGP

Bethgael wrote: »

If this is a new form of monitoring, then it would explain the recent lag spikes.

ZOS, if your game can't be expanded in, say, housing or PvP because of "technical limitations", it's a bit disingenuous to be using a bot that monitors private conversations.

yup.

Elsonso

SilverBride wrote: »

@ZOS_Kevin @ZOS_GinaBruno

Are they really monitoring our private whispers, and using AI to do it?

My guess is: Yes to monitoring. No to AI.

I am guessing that they are using pattern matching, which is probably what they have always been doing. AI would probably be a step up, but imagine how much it would cost (time, CPU/GPU, money) to run everything players say through an AI. Eeek.

Syldras

Elsonso wrote: »

I am guessing that they are using pattern matching, which is probably what they have always been doing.

I could understand pattern matching and then autoforwarding the part of the dialogue to a real person to make a decision - when it comes to keywords that might mean an actual threat and planning of severe crimes. Obvious example: The word "bomb", or names of real terrorist organizations (although it's questionable whether people planning such crimes would openly write about it in normal words in a game chat - but that's a different topic).

But bans because of absolutely harmless things like mild swearwords? Or sometimes complete "nonsense" or out of context jokes? Not only that it's clear that there is no real person involved when it comes to such decisions; it generally seems out of scope to even scan text automatically for something as trivial as some stupid cusswords.

Elsonso

Syldras wrote: »

Elsonso wrote: »

I am guessing that they are using pattern matching, which is probably what they have always been doing.

I could understand pattern matching and then autoforwarding the part of the dialogue to a real person to make a decision - when it comes to keywords that might mean an actual threat and planning of severe crimes. Obvious example: The word "bomb", or names of real terrorist organizations (although it's questionable whether people planning such crimes would openly write about it in normal words in a game chat - but that's a different topic).

But bans because of absolutely harmless things like mild swearwords? Or sometimes complete "nonsense" or out of context jokes? Not only that it's clear that there is no real person involved when it comes to such decisions; it generally seems out of scope to even scan text automatically for something as trivial as some stupid cusswords.

That would be the "what they have always been doing" part of my comment. If you look back, you see complaints about them banning or suspending people for all sorts of things that seem automated. Automated does not mean AI. As I said, AI might be an improvement, but it will not come for free.

Syldras

Elsonso wrote: »

That would be the "what they have always been doing" part of my comment. If you look back, you see complaints about them banning or suspending people for all sorts of things that seem automated. Automated does not mean AI. As I said, AI might be an improvement, but it will not come for free.

Indeed, but it seems that this happens increasingly often lately? And I have not heard about people being banned for literally talking to themselves in a private instance - which means that it's absolutely impossible someone reported them - before (or have I just missed the threads?)... It seems like something has changed. Very curious about the upcoming statement.

Sleepsin

AzuraFan wrote: »

Sleepsin wrote: »

AzuraFan wrote: »

Wow. A bot should not be monitoring and reporting on private conversations. Unless a person reports it, nothing should be done. What's next? Monitoring Discord?

It's hilarious that in a game that has us running around killing people, and that includes the thieves guild, the dark brotherhood, slavery, and necromancy, they are concerned about what's being said in private chat between consenting adults.

Do you really think Discord is not being monitored?

I meant by ZOS.

The AI monitoring program wasn't written by ZOS. A 3rd party programmer makes them and sells the use of it to companies like ZOS. A discord channel flagged as an ESO rp guild could very well being monitored by the same program that is watching in-game chat channels.

faradves

Kashya_Vulano wrote: »

jaws343 wrote: »

Monitoring keywords does not equal AI or bots... Setting up a monitor on all in-game coms to flag specific words used in chat literally requires zero AI. It's no different than a basic word sensor on forums or in character creation. A simple query against a database of flagged words that returns a simple result.

Here's a snippet from the TOS. It doesn't have to mean AI, you're right, but this is also in the terms of service.

Also, there is nothing wrong with the utilization of analytic AI to monitor some things, in fact it can be useful. However I believe in this case it is a violation of TOS and should be used in tandem with in-game reports. Lets say we use AI to specifically monitor the conversations of someone who is under scrutiny for violating the community terms (as in, they were reported by players for harassment). If used sparingly it can be helpful, but even then, we are crossing into dangerous territory of privacy violation and should be used sparingly. Community wide monitoring of everyone's chats, including private conversations goes against TOS which clearly states they do not "pre-screen content" and are under no obligation to monitor conversations. There are several contradictions in their TOS and this should be pointed out. Even if the TOS says it can do something, that doesn't make it right.

wolfie1.0.

Pelanora wrote: »

jaws343 wrote: »

Pelanora wrote: »

jaws343 wrote: »

Inari Telvanni wrote: »

TOS says,

" Content Moderation

To the extent that ZeniMax performs any content moderation of UGC to ensure its compatibility with these Terms of Service (including the Code of Conduct or any relevant EULA), such content moderation may be carried out via human review as well as through the use of AI-powered proactive and reactive moderation methods including without limitation, software that uses algorithmic decision making.

ZeniMax's proactive content moderation includes without limitation using tools to block and filter UGC that is illegal and/or incompatible with these Terms of Service.

Reactive content moderation methods include without limitation user reporting features which allow You to inform ZeniMax of the behavior or Content of other users that You have encountered which you believe is illegal and/or incompatible with these Terms of Service (including the Code of Conduct) and any such behaviour or Content can be reported to ZeniMax by contacting ZeniMax Customer Support at help.bethesda.net or help.elderscrollsonline.com. If You are in-Game, You can report an issue using an in-Game help feature where applicable. Where ZeniMax is required to do so by virtue of the Statutory Obligations (as defined in Section 1), ZeniMax shall advise You of remedial steps action taken against another user as a result of Your report including details of what steps ZeniMax has taken to investigate your report, if ZeniMax has removed Content that You have reported or if any other restrictions have been applied to the Content or the other user. "

This is an enormous red flag.

No it is not..

Zos, as a company that can be held liable for conversations that might happen through the use of their platform, have a responsibility to make sure that they can monitor illegal things being done in their game. Even if those things are done in private chats. If you are using their tools to do illegal things, they are legally put in a spot to intervene or face severe legal repurcussions if they do not.

'Illegal things.'

The whole game is about doing things illegal irl. Talking about the game would in that case be a bannable offence. They will need to be more sophisticated than key words.

I mean, yes. Just like any chat or social media service is potentially liable for the planning or conducting of actual illegal activity with their service Zos is also beholden to that. If a group of people use whispers or group chat to plan a real crime, and Zos did not do everything within their power to prevent their service from being used for that manner, then they could be held liable for allowing that conduct. An entire platform recently is under fire for exactly this in Europe.

and people use chat to plan killing a boss or other players in PVP and software can distinguish that from IRL chat how?

and if someone is buying a game licence and logging in here to plan IRL stuff rather than an encyrpted tool accessible on a phone- i mean really you think thats whats happening? because thats just silly

It's actually not, which is sad. Remember that it wasn't that long ago that a dispute in a video game resulted in a Swatting incident resulting in death. Grooming, exploitation, release of classified documents... among other things has all happened either in live service games or ancillary services related to them.

Pelanora

Skarphedinn wrote: »

Good to see the mods are taking this seriously and snipping post saying to report it to the GDPR.

GDPR is a law, not an organisation.. Which only covers the EU server and EU citizens. Would need to talk to the relevant privacy monitor in your country who operates under the GDPR.

However, you agreed to the TOS so likely you've received notice and agreed to what they're doing with your data for the purposes of the GDPR.

Syldras

wolfie1.0. wrote: »

It's actually not, which is sad. Remember that it wasn't that long ago that a dispute in a video game resulted in a Swatting incident resulting in death. Grooming, exploitation, release of classified documents... among other things has all happened either in live service games or ancillary services related to them.

As sad as it is: this is humans. Just the medium they use for these things changed over the centuries.

Before somebody complains: No, not all humans. Not even the majority. But it is the human behind the screen that causes this, not the tech.

Pelanora

wolfie1.0. wrote: »

Pelanora wrote: »

jaws343 wrote: »

Pelanora wrote: »

jaws343 wrote: »

Inari Telvanni wrote: »

TOS says,

" Content Moderation

To the extent that ZeniMax performs any content moderation of UGC to ensure its compatibility with these Terms of Service (including the Code of Conduct or any relevant EULA), such content moderation may be carried out via human review as well as through the use of AI-powered proactive and reactive moderation methods including without limitation, software that uses algorithmic decision making.

ZeniMax's proactive content moderation includes without limitation using tools to block and filter UGC that is illegal and/or incompatible with these Terms of Service.

Reactive content moderation methods include without limitation user reporting features which allow You to inform ZeniMax of the behavior or Content of other users that You have encountered which you believe is illegal and/or incompatible with these Terms of Service (including the Code of Conduct) and any such behaviour or Content can be reported to ZeniMax by contacting ZeniMax Customer Support at help.bethesda.net or help.elderscrollsonline.com. If You are in-Game, You can report an issue using an in-Game help feature where applicable. Where ZeniMax is required to do so by virtue of the Statutory Obligations (as defined in Section 1), ZeniMax shall advise You of remedial steps action taken against another user as a result of Your report including details of what steps ZeniMax has taken to investigate your report, if ZeniMax has removed Content that You have reported or if any other restrictions have been applied to the Content or the other user. "

This is an enormous red flag.

No it is not..

Zos, as a company that can be held liable for conversations that might happen through the use of their platform, have a responsibility to make sure that they can monitor illegal things being done in their game. Even if those things are done in private chats. If you are using their tools to do illegal things, they are legally put in a spot to intervene or face severe legal repurcussions if they do not.

'Illegal things.'

The whole game is about doing things illegal irl. Talking about the game would in that case be a bannable offence. They will need to be more sophisticated than key words.

I mean, yes. Just like any chat or social media service is potentially liable for the planning or conducting of actual illegal activity with their service Zos is also beholden to that. If a group of people use whispers or group chat to plan a real crime, and Zos did not do everything within their power to prevent their service from being used for that manner, then they could be held liable for allowing that conduct. An entire platform recently is under fire for exactly this in Europe.

and people use chat to plan killing a boss or other players in PVP and software can distinguish that from IRL chat how?

and if someone is buying a game licence and logging in here to plan IRL stuff rather than an encyrpted tool accessible on a phone- i mean really you think thats whats happening? because thats just silly

It's actually not, which is sad. Remember that it wasn't that long ago that a dispute in a video game resulted in a Swatting incident resulting in death. Grooming, exploitation, release of classified documents... among other things has all happened either in live service games or ancillary services related to them.

AI will never be able to distinguish RP from any of that. Release of classified info?!!!! There are such better platforms for that.

Any investigation of a crime means intent and context are required.

wolfie1.0.

Syldras wrote: »

wolfie1.0. wrote: »

It's actually not, which is sad. Remember that it wasn't that long ago that a dispute in a video game resulted in a Swatting incident resulting in death. Grooming, exploitation, release of classified documents... among other things has all happened either in live service games or ancillary services related to them.

As sad as it is: this is humans. Just the medium they use for these things changed over the centuries.

Before somebody complains: No, not all humans. Not even the majority. But it is the human behind the screen that causes this, not the tech.

Yep, and as the mediums change so do the steps to try to protect against it.

Syldras

Pelanora wrote: »

AI will never be able to distinguish RP from any of that.

Unless maybe if you write "(OCC: This is roleplay! No harm done!)" between each line.
And then criminals will do the same while discussing real plans...
You're right: AI cannot and will never be able to distinguish that.

Syldras

wolfie1.0. wrote: »

Yep, and as the mediums change so do the steps to try to protect against it.

As I said: This I can understand -

Syldras wrote: »

I could understand pattern matching and then autoforwarding the part of the dialogue to a real person to make a decision - when it comes to keywords that might mean an actual threat and planning of severe crimes.

But letting a machine decide without any human double-checking?
Also, it's not neccessary to use it for some silly swearword filtering, let alone banning people because of swearwords.
For severe, real matters, I understand the necessity. But not for things beyond that.

And yes, AI can't understand roleplay vs real life, that's what the human double-checking is for.

Carcamongus

"I'm sorry Draveth Telvanni, I'm afraid I can't do that."

Pelanora

Zos shouldn't be banning words!!!!!!!!!!!

Whatever tool they use to track it is not the point they shouldn't be banning words or phrases!

Having AI or a lookup or a person read for some list of phrases some puritan has created is awful!

We should have free speech.

Ban behaviours, ban abuse and nastiness and stalking- etc. Which is about intent and context and effect on a person.

Not speech alone.