Bots - The Technical Reasons Behind Them

CrimsonThomas

They're game-destroying. Zenimax should stop flirting with the issue through timers and saying "Just report them!" and take the primary, active role in going after the people ruining The Elder Scrolls Online.

That $15 monthly subscription supposedly paid for better support, let's see our subscription dollars at work.

SirAndy

Repost from another thread:

I'm a programmer and worked in the game industry for many years.

My take:
ZOS made the mistake to have client side movement where the decision about your movement in the world is made locally in your game client and that decision is then sent to the server.

This opens the door WIDE for all sorts of exploits. No game relevant decision should ever be made by the client and then be sent to the server. Ever. All bad.


Instead, every multiplayer game i have ever worked on used some sort of prediction model where the server kept track of your characters state at all times and the client only send the user input to the server.
The server then decides whether that input is within the capabilities of the character, calculates an outcome and then transmits that result to the client.
In the mean time the client has done a similar calculation to "predict" the server answer and renders the client side prediction before even getting an answer from the server.
99% of the time the answer matches the prediction and you get a smooth client side game display with very little lag.
The few times when the prediction fails, you'll see little hickups, but they rarely happen.

Now, the real upside of these prediction implementations is that even if a hacker was able to sent a command to the server saying "i'm here at this node, now teleport me over to this other node 1000 yards away" the server would simply reject this input since it would fall outside of the abilities of your character.
The server would always take your current state (location, speed, buffs, terrain, abilities etc.) into consideration and calculate to see if what you're trying to do (for example move 1000 yards at once) is even possible and reject anything that is impossible.

On top of that, they're using the windows message loop to query your input devices, like the keyboard and mouse.
It is stupendously easy to inject messages into that loop from 3rd party applications. That is why you already see bots that actually run between nodes just like a normal character.

I mentioned this to ZOS the first time 8 months ago. There are low level windows functions to directly talk to input hardware without using the message loop and those functions are inherently more safe to use since injecting false information is a lot harder.

Oh well ...

Publius_Scipio

I am worried because if I am not mistaken all the spamming and the bots mean that players have hacked ESO correct?

Is our personal information safe?

Publius_Scipio

KerinKor wrote: »

Publius_Scipio wrote: »

So the creators behind the bots have HACKED ESO?

LOL, no.

ESO is the same as pretty much every MMO, this is nothing new. Just use Google to search for the obvious and you'll learn very quickly the basis on how these bots work.

Publius_Scipio wrote: »

How can they create these automated characters that are invisible and whatever else?

Tele-bots are nothing new, 'under the ground bots' are nothing new, what I've not seen before are 'above ground' invisi-bots.

They are not inevitable, but it requires ZOS to design systems with anti-bot and anti-spam mechanics in plac, they singularly failed to do so and we are now reaping the rewards of that negligence.

Maybe ZOS needs to hire hackers to combat hackers. Or put some really priority on their system and code designers to write something into the inner workings of the game.

Publius_Scipio

Daggers wrote: »

Thechemicals wrote: »

Its not just bots from gold sellers. A customer purchases 2 copies of eso, a main and a mule, then uses the mule to use macro programs and hacks. Hacks in this game are very easy to do versus say WoW. Blizzard encrypts the data (laymans)from their side while this game does not. You dont even need a hack program, you could just use a standard legal office macro program to bot.
* I do not promote or use these programs.

If you dont like this and feel its an ice breaker for you then you should cancel your subscription to this game and all games youll ever play, mmo,fps, rts, doesnt matter the genre. The only shield is the quality of the security team for the company. Again Blizzard is exceptional at this above all other companies.

I think that's a little genre alarmist. ESO easily has the worst bot infestation I've seen in a launch MMO specifically because the broke the first rule of serious networking - they trust the client. Ironically, I'd compare it to trusting the rest of the internet with our login usernames, but hey we've done that argument already.

The more I discover, the more stupid I think the people responsible for this are, in the truest sense of "where were you when God gave out brains?".

This is schoolboy error stuff, not the hallmark of a multi-million-dollar investment. It's fixable, but the fact we're here now talking about it is quite strange.

There must of been a reason for ZOS building the game the way they did I would imagine. They had a serious budget for this game and one of the minds had to of brought up client side vs. server side. No?

jordan_hoffmanb14_ESO

This thread has me depressed. Wow, what a complete fail....

Thunder_Downunder

GW2 had lots of bot problems initially, but it seems to be under control. The added controls have however reduced the overall satisfaction of the game. I hope ZOS do something about this. Personally, I'm thinking of taking 3 months off to see if they can fix this issue and the bugs.

Gwarok

Publius_Scipio wrote: »

Bots are a big topic right now and I want to understand more about what they are and who is behind them.

There are gold sellers, boss campers, and I am hearing something about invisible bots flying to nodes and collecting them... Is this the same "people in China" thing? Secondly and of more interest to me, how do these bots exist in the first place?

That's just Secret Asian Man & Company

robacooperb16_ESO

Publius_Scipio wrote: »

There are gold sellers, boss campers, and I am hearing something about invisible bots flying to nodes and collecting them. First off, who creates the bots? Is this the same "people in China" thing?

A few google searches tells me that it seems to be mostly Russian sites offering exploit/bot programs.

Secondly and of more interest to me, how do these bots exist in the first place? I know ESO has its API's which allow modding and addons to be created, but I know ZOS has really tightened up which API's are available for use. So the creators behind the bots have HACKED ESO? How can they create these automated characters that are invisible and whatever else?

No. By allowing the client (ie. your computer) to dictate actions your toon makes to the server it's really easy to manipulate the data being sent to the server. Most (if not all) MMOs use server predictive movement and 'card' commands - ie. server is predicting your movement, you on your computer change that movement, server compares your change to what is possible and decides to allow it or ignore if not (that ever so brief delay between action and response in other games), or why abilities are on "timers" so that server can dictate when option to use ability is available.

class101

@Publius_Scipio‌
I believe they are bot like the famous one that existed under WoW, was called mmo glider if you want to search for its wiki history

Basically it doesn't use any of the developers API, it was a third party program that use to monitor the game window, you could set waypoints in it and the program would play instead of you, capable of following waypoints, hitting enemies, etc... etc

They just have the limit to be stupid and easily spottable because it is obvious if you look them playing that they are not human

@SirAndy

Even if ZOS made the mistake of client side movement, mmoglider used to bypass this finger in the nose because this bot just used to automate a real player nothing much and in this case only advanced known program detection like they made in Warden could detect it, but not ultimately, that's why they went it court to completely kill the society making money behind this bot.

SirAndy

class101 wrote: »

Even if ZOS made the mistake of client side movement, mmoglider used to bypass this finger in the nose because this bot just used to automate a real player nothing much and in this case only advanced known program detection like they made in Warden could detect it, but not ultimately, that's why they went it court to completely kill the society making money behind this bot.

@class101
But with mmoglider you're never *better* than any quick real player. Sure, you can leave it run forever and a real player needs sleep every once in a while but other than that the bot just moves around like a normal player.

The big difference is, right now the bots in ESO can teleport from anywhere in to zone to anywhere else in an instant!
Heck, there are already some for PvP that teleport to the scroll temples, take the scrolls and then park themselves *under* the terrain so that no-one can get to them.

None of that is possible with mmoglider ....

Vorpedagel

@Publius_Scipio thank for bringing this to everyone's attention, I've been looking into this a little myself. For them to change over to sever side calculations.. They are going to have to fork out a LOT of money for new hardware.

I'm sure they'll push in a better genuine client verification system at some point but I wouldn't say it priority.. You're right it is a big issue and I can guarantee it hasn't been overlooked. It's not going to be easy for them and it's going to take some time.

Cry_Wolfe

just want to point out that for all its vaunted systems, WoW was still compromised.

Floating gnomes corpses spelling out gold seller site addresses.

Warsong Gulch flag captures in the first 3 seconds of a match.

Underground farm bots.


"admitting to a problem is always the first step"

Publius_Scipio

Honestly, after a certain amount of time and effort what do these people behind the bots want with any MMO to begin with?

robacooperb16_ESO

Publius_Scipio wrote: »

Honestly, after a certain amount of time and effort what do these people behind the bots want with any MMO to begin with?

$$$ The vast majority of these bots support gold sellers as secondary account "harvesters" - they don't spam gold seller messages but just collect resources to generate revenue in game.

Publius_Scipio

I know the phrase "in a perfect world" applies here BUT.... If every player stopped buying game gold for real money the entire business and the bots along with it would collapse. Thing to remember is that there are real people paying these sellers real money for in game gold. The spam and the bots aren't there for the sake of spamming and having bots run around, which is the scary part. And the people paying for gold are most likely the ones that have never visited the forums, and surely not the ones that posted in any of these threads.

GTaichou

Mmmmmm about gold spammers not posting on forums and not being real people... I do know someone who was approached and was offered a job as a gold farmer. Active. Human-input farming. So it's not like it doesn't happen...

cheeser123

Publius_Scipio wrote: »

I know the phrase "in a perfect world" applies here BUT.... If every player stopped buying game gold for real money the entire business and the bots along with it would collapse.

In a fictional world where dealers can be banned and the mechanics can be altered to prevent dealing, why focus on the buyers? People will always buy if the price is right.

Publius_Scipio

GTaichou wrote: »

Mmmmmm about gold spammers not posting on forums and not being real people... I do know someone who was approached and was offered a job as a gold farmer. Active. Human-input farming. So it's not like it doesn't happen...

Well that is probably more on the rare side as far as occurrence.

Publius_Scipio

cheeser123 wrote: »

Publius_Scipio wrote: »

I know the phrase "in a perfect world" applies here BUT.... If every player stopped buying game gold for real money the entire business and the bots along with it would collapse.

In a fictional world where dealers can be banned and the mechanics can be altered to prevent dealing, why focus on the buyers? People will always buy if the price is right.

Because like I said, in a perfect world, everyone right now can stop buying gold. Literally this very instant everyone can stop buying the gold. No tweaks needed for the game. The business will simply wither and die.

In a perfect world that is....

Adlerson

Publius_Scipio wrote: »

I am worried because if I am not mistaken all the spamming and the bots mean that players have hacked ESO correct?

Is our personal information safe?

What you see from this hacking and the bots in the game has nothing to do with your personal information on Zenimax' servers, nor any payment info or anything like that. So even though 'they' have been able to hack certain aspects of it (Movement, perma-dodge etc) they do NOT have access to your information. They do not have access to the servers, rather, they alter and falsify the raw data going in to controlling the character/toon on the screen.

I was standing outside the inn in Fell's Inn the other night, watching these teleport bots come and go, lvl 5 characters in Coldharbour gear in a lvl 30 area with random names, and made a comment about this on /zone. Some guy started /whispering me, explaining what was going on, and pointed me to a forum on a website. I went there and read up on this, and what surprised me first of all was that many of these people did not seem to be Chinese or Russian, but instead were people from the good 'ol US of A, who does this either 'because they can', or some of them even to make money. The second thing that surprised me was that the people who were making these programs and figuring out how to do these hacks all called Zenimax' programming 'lazy', in that they found it easier to hack these things in this game than in any other game they've played before.
Now, personally I don't know if it's because they've been lazy, I think a previous poster was on to something; This has been touted as a game with huge multiplayer battles, with people able to do real time aiming and dodging and what have you, and given that it might be they chose to forego securing the datastream since that would add a significant overhead to the server load. (I think.) One thing is doing server side movement double checking in battles with up to 32 players, but if I'm not mistaken we've been told we'll be able to have hundreds of people on the screen at any one point in this game.

I am sure they'll come up with some kind of tool to take care of these things though, but in the meantime, don't worry about your secure information on their servers. Not saying that can't be hacked, but that is NOT the same kind of hacking that is going on with these bots, cheaters and gold farmers etc.

Cheers,
Adlerson

Aci

In other games there are a lot instanced areas. In this game there is no such for the public dungeons. Maybe the number of bots is not higher than it was in other games. I truely think its that painfull because we actually see them now, there is not much place to hide for them.

Dont get me wrong, I like open world and megaserver. If Z only could find a way through all that...

reaper85b16_ESO

cheeser123 wrote: »

Publius_Scipio wrote: »

I know the phrase "in a perfect world" applies here BUT.... If every player stopped buying game gold for real money the entire business and the bots along with it would collapse.

In a fictional world where dealers can be banned and the mechanics can be altered to prevent dealing, why focus on the buyers? People will always buy if the price is right.

People will NOT always buy, if the Company behind the Game is hard enough in their punishment.
For comparison take EVE-Online. There are nearly no sellers for ingame-currency, because the seller will be banned for life and the buyer will lose everything he had in the game, so nearly no one is willing to buy anything unallowed. That works pretty good for CCP.
Back to the Game here: I think we should just wait and see, what Zenimax will do about it. It's only 2 weeks since release, so keep calm and wait. They have repeatedly postet that they are aware of the problem and search for ways to solve it.

OmniDo

"n00b mistakes"
As we in the w0w community often said.

Another reference that would apply here is: "lrn2dev"
WoW has resolved this and other issues years ago, and the only viable "Bots" are those that are so well designed and so immaculately programmed, that they resemble all the normal actions and responses of a real player.

One might have expected developers of a AAA MMO title to have done a little background research into these and other obstacles, but apparently they did not.
Unfortunately, "n00b" fits the description of these developers quite well.

They'll learn, and adapt, or their game will fail and we'll all move on to something else. I certainly hope the former is the outcome and not the latter, because I thoroughly enjoy the Mundus Universe.

mike.crewsb14_ESO

aeroch wrote: »

@Thechemicals‌ Why wouldn't Zeni encrypt their data? Is there a technical reason?

I'm actually curious about how WoW does it. How can you encrypt communications from the client to the server, without the client knowing the key? If the client knows the key, so does the hacker and the encryption - no matter how powerful - is useless.

GreySix

krix_ost wrote: »

Here is the answer you may not want to hear.

ZeniMax has chosen to use Client Side Trust for many of the work load for the game. This does have advantages because every computer can do their own calculations and send that to the Server for Updating. However, it seems this data is not being encrypted (which adds overhead and complicates network delivery). As hinted above.

This.

Which is why hackers are having a field-day, and you see comments like this on their forums of choice:

Yea, I've done similar in at least 1/2 dozen games over the last ten years.

• When ever a dev is foolish/lazy enough to allow this in the client
• Often times finding ways to get around server side checks/sync when they do have it.

starkerealm

mike.crewsb14_ESO wrote: »

aeroch wrote: »

@Thechemicals‌ Why wouldn't Zeni encrypt their data? Is there a technical reason?

I'm actually curious about how WoW does it. How can you encrypt communications from the client to the server, without the client knowing the key? If the client knows the key, so does the hacker and the encryption - no matter how powerful - is useless.

@SirAndy‌ actually discussed this. The short version is, TESO trusts the client to send legitimate data. This is a major faux pas in client management for, well, really any network system.

The reason is, you can't trust the data coming from the client, ever. If someone's using any number of tools, they can simply modify the information that the client sends, and do... pretty much whatever they want.

What they can actually get the server to send them is an open question.

In contrast, most MMOs will register a command on your system, and then ask the server if it's okay with that, then pass it back to the client and let you do something. In a lot of ways it resembles computer architecture from the 1960s. It's slower, more server intensive, but much more secure.

In retrospect, those bold claims about a lag free launch make a lot of sense now. See, in a normal MMO, there's a massive overhead on the server, it needs to know where every player is at all times, and it has to verify every command issued through the system...

But, with TESO... that's not quite true. The server needs to know where everyone is, but it doesn't have to think about how they got there or what they've been doing. It also doesn't need to know what's happening to their health, stamina, mana, whatever unless it's passing that to another player.

This is also why, when disconnected, you can still run around until the client realizes, "whoops, I'm not online anymore." And, if you've ever had that happen, you'll know it doesn't reset you back to your starting location. The server doesn't care where you are, it accepts the client's story that, "no, really, we're over here now."

This means, for an MMO, the game scales to handle ludicrous numbers of players, but it comes at the cost of virtually no security.

Forztr

People use goldsellers because this game has pretty severe cash sinks and some useful rare mats are really rare and with the fractured market hard to find.

Basically the design of the game is perfect for bots and goldsellers, they've made it easy to bot and created a demand for goldsellers services.

starkerealm

Forztr wrote: »

People use goldsellers because this game has pretty severe cash sinks and some useful rare mats are really rare and with the fractured market hard to find.

Basically the design of the game is perfect for bots and goldsellers, they've made it easy to bot and created a demand for goldsellers services.

Yeah, this too.

There's actually a distinct possibility, because of how gear deterioration is handled, that the bots are actually increasing it by proximity, meaning players who encounter bots are more likely to face egregious repair costs.

j.frank.nicholsb14_ESO

Well, I have seen one programmer (not ZOS programmer) and a lot of people that claim to know how to design MMO's correctly.

First I seriously doubt the problem is as simple as everyone is making it out to be. I also seriously doubt that the server "completely" trusts the client. I am sure there is some trust involved, and I don't doubt that there is an attempt by ZOS to use client side movement and possibly even more. There are a significant number of advantages to doing that. There are certainly a number of risks.

I expect ZOS is working on resolving the risk issues.

To state that the ZOS programmers are stupid is stupid. Typical attacks by people that have never done design and want to make themselves sound important. Designing complex systems is not done by stupid people. ESO works with very few bugs compared to other MMOs when they were 1 month old. I expect ZOS is trying to architectural changes (design) and the result is a swarm of bot programs taking advantage of the new design. I also expect ZOS designers will make the necessary changes to address the problems.

Also, just a note - a large percentage of the posts in this thread are focused on the teleport, subterranean, and flying bots - which while visually interesting have little to do with the bot issue, other than cosmetics and making it a little harder for players to report them. Farming could just as easily be done using normal player mechanics (run over and click). The Ultimate spamming is a more serious problem, allowing power leveling easier and boss loot harvesting easier - note I said easier. Even if they had to user normal player mechanics, they could still do it, it would just be slower.

Whenever I read someones opinion and they result to name calling and denigration I consider the source, and if any of them have ever been involved in a successful multimillion dollar project - I expect the answer is no. And I expect all the "haters" here are just that - "haters" with nothing but hiding behind a screen to support their claims and insults.