Data Base Storage Question and lag

Ranger209

@ZOS_GinaBruno

ZOS_GinaBruno wrote: »

Next Monday with the PC launch of Update 21, we’ll be implementing some code related to ESO’s database that will allow us greater flexibility with updates and maintenance as it continues to grow over time. Due to this change, after downloading the patch on Monday, you are likely to encounter a very long load screen upon first logging into the game. This will only occur the first time you log in after character select, and it will not affect subsequent logins. That said, the length of the load screen could vary anywhere between 5-10 minutes to a couple hours. We understand this is far from ideal and is frustrating on launch day, but it’s a necessary evil just this one time.

Now, this loading is actually directly tied to the total number of bank items, inventory items, and mail attachments everyone has. This means if you’re able to clear out excess items in your inventory and attachments before maintenance begins Monday morning (or even just reduce them), this will greatly help the amount of loading the client will need to do! Additionally, once you’re at the load screen, please just wait until you’ve made it into the game; closing the client will not speed things up.

Thanks for your understanding, everyone, and we look forward to Wrathstone and Update 21 launch in just a few days!

This got me thinking...When I first read it I thought oh great maybe they have found a way to optimize the data base or have developed tools to do so. With over 300,000 items and 1200 slots being used I expected a very long load screen. Surprise, surprise it was just like normal. I start reading the forums and read how the siege bug has cured lag in Cyrodiil. I was unable to play day 1 or was doing something else in game and did not go to Cyrodiil myself. Day 2 in Cyrodiil chat I am hearing that the lag is coming back, but to me it still seems far better than it was pre-patch. now I'm not sure if it degraded the rest of the week or not. I was convincing myself that siege had fixed the lag, and maybe it did, but I am not certain if it was my own wishful thinking with nothing else to credit it with or if it was really the reason.

So I have been thinking about the data base that stores all of our items, and wondering was it the siege, or was it that as more and more people logged in for the first time doing that data base magic that things got more and more congested. It got me thinking when the game performs combat calculations where is it pulling our gear set information from within that data base? Is it client side or server side? If it is server side is it partitioned out from say my 27000 alkahest and my other meaningless odds and ends, or is it all lumped into one big data base. If it is partitioned out in the data base can it be done so further to reduce the amount of data it has to sift through to find the gear set to perform the calculation. Is there already some specific place that only deals with equipped sets, or is there a place that has my account information and all of the data relating to it that the code goes straight to when dealing with me? Probably way off base here but I have never been afraid to ask the dumb question when I don't know the answer to it.

Think I'll get back on the OP siege bandwagon, I liked it!!

JTorus

Ranger209 wrote: »

@ZOS_GinaBruno

ZOS_GinaBruno wrote: »

Next Monday with the PC launch of Update 21, we’ll be implementing some code related to ESO’s database that will allow us greater flexibility with updates and maintenance as it continues to grow over time. Due to this change, after downloading the patch on Monday, you are likely to encounter a very long load screen upon first logging into the game. This will only occur the first time you log in after character select, and it will not affect subsequent logins. That said, the length of the load screen could vary anywhere between 5-10 minutes to a couple hours. We understand this is far from ideal and is frustrating on launch day, but it’s a necessary evil just this one time.

Now, this loading is actually directly tied to the total number of bank items, inventory items, and mail attachments everyone has. This means if you’re able to clear out excess items in your inventory and attachments before maintenance begins Monday morning (or even just reduce them), this will greatly help the amount of loading the client will need to do! Additionally, once you’re at the load screen, please just wait until you’ve made it into the game; closing the client will not speed things up.

Thanks for your understanding, everyone, and we look forward to Wrathstone and Update 21 launch in just a few days!

This got me thinking...When I first read it I thought oh great maybe they have found a way to optimize the data base or have developed tools to do so. With over 300,000 items and 1200 slots being used I expected a very long load screen. Surprise, surprise it was just like normal. I start reading the forums and read how the siege bug has cured lag in Cyrodiil. I was unable to play day 1 or was doing something else in game and did not go to Cyrodiil myself. Day 2 in Cyrodiil chat I am hearing that the lag is coming back, but to me it still seems far better than it was pre-patch. now I'm not sure if it degraded the rest of the week or not. I was convincing myself that siege had fixed the lag, and maybe it did, but I am not certain if it was my own wishful thinking with nothing else to credit it with or if it was really the reason.

So I have been thinking about the data base that stores all of our items, and wondering was it the siege, or was it that as more and more people logged in for the first time doing that data base magic that things got more and more congested. It got me thinking when the game performs combat calculations where is it pulling our gear set information from within that data base? Is it client side or server side? If it is server side is it partitioned out from say my 27000 alkahest and my other meaningless odds and ends, or is it all lumped into one big data base. If it is partitioned out in the data base can it be done so further to reduce the amount of data it has to sift through to find the gear set to perform the calculation. Is there already some specific place that only deals with equipped sets, or is there a place that has my account information and all of the data relating to it that the code goes straight to when dealing with me? Probably way off base here but I have never been afraid to ask the dumb question when I don't know the answer to it.

Think I'll get back on the OP siege bandwagon, I liked it!!

Zones are segmented by areas. You can usually see what area you're in with the help of some addons.
I broke out my crayons, and drew an example that is likely very inaccurate, but for the purposes of this conversation, representation enough..



What players see as lag, is really a bottleneck issue. Connectivity between the user and ZOS's server (latency) is usually fast enough. Sometimes it's easy to be misled when we we see that number climb from say 40 - 50 to 200+. This number seems to be calculated by more than a player's connectivity to ZOS's server, but instead how long it takes ZOS to receive, process and return data to the user. It seems that other calculations are factored in to represent an overall representation of the player's latency while interacting with Tamriel.

Remember, that no matter how sophisticated computers get, basically, you can still only do one calculation at a time. In order to optimize things, it makes sense that ZOS would segment a zone into smaller areas. Rather than one large authoritative process crunching the numbers, these smaller areas would be responsible for what takes place under their jurisdictions, only cross communicating necessary data (zone chat, AP ticks, player's moving through, etc). This way, ZOS can program prioritization, load balancing techniques and so on. All to work towards better fidelity.

Sometimes this compartmentalizes the 'lag' to just one spot (IE: a keep fight) but larger battles, or the sum of multiple medium battles still lag the whole zone (IE: scroll runs), the bottleneck is the result of the server's resources unable to reconcile the queue of information as swiftly as it receives it. This is why sometimes, players time out when they travel to an area with a large battle already in progress. They're so far back in the line, they may as well be somewhere else.

Yes, Duh, right? We all know this more or less, why say it?

It emphasizes the point that lag cannot be fixed. ZOS has overwhelmed their server's capabilities, and they did it intentionally. Some of us remember performance prior to a certain patch. A lot of blame and suspicion was place towards things like particle effects, deer, torchbugs, and so on. Cute.

Prior to that certain patch a few years back, there was quite a lot of wacky behavior. Players (bots) were able to instantly move from one resource node (plants, ore, runes, etc) and harvest them, often below the ground. Not hard to do once you have those things mapped out with XYZ coordinates. Other odd behavior was exhibited too. This whole game relied way too heavily on the honor system. ZOS's servers took what your client sent to it and just crunched it, and moved on. For most people, that's fine, but for those that would exploit this, well, it was open season. A complete lack of foresight on the initial development. Never trust your users to maintain the integrity of your data.

The solution was to implement a check system. Information your client sends back to ZOS (character position, movement speed, abilities, etc) are now double checked. Prior to this patch it would be like playing Dungeons and Dragons without a DM, and you could use any dice (die, whatever) you like! But now, an additional burden falls to the servers to basically double-check everything you do, ensuring it is acceptable.

This is not a new problem, and not just with ZOS. To place all players on one single 'server' is an ambitious goal, especially when you can not predict to what extent it'll be utilized. Millions of connections, all querying a large database, and altering it, repeatedly, in milliseconds; that's a hell of thing. EVE Online sufferers this same issue and to this day, has not solved it. Instead, they implemented 'time dilation'. And their battles make what takes place in Cyrodiil seem like rock-scissors-paper matches on the school yard.

What does any of this have to do with siege? The siege 'bug' addressed a symptom. Much like NyQuil addresses your cold. Siege scared the people who would otherwise slam the server with more data than it could crunch, thereby reducing its load. Players dispersed, their battles were not as prolonged, less people showed up to 'zerg'. Chaos prevailed. Had the siege bug remained, I suspect we likely would have seen the bottleneck issue return in other ways.

TL;DR Lag is the result of an intentionally implemented 'checks and balances' system to reduce blatant exploitation, and cannot be removed. The only solution is to continue to address the symptoms, and 'distract' everyone enough with other objectives *cough*daedric artifacts*cough* in order to maintain a bit of server integrity.

Also, rewind back up to my crayon picture. There's your stuck in combat bug. There's an issue where data doesn't release you from combat as you cross areas (or even zones, I've had it happen after Battleground matches end). I've been able to intentionally re-create the bug sometimes, but not every time. But, I'm also very lazy. Maybe I'll name one of my pets DROP_TABLE and see if that fixes it.

Crown

JTorus wrote: »

TL;DR Lag is the result of an intentionally implemented 'checks and balances' system to reduce blatant exploitation, and cannot be removed. The only solution is to continue to address the symptoms, and 'distract' everyone enough with other objectives *cough*daedric artifacts*cough* in order to maintain a bit of server integrity.

@JTorus There are two relatively simple (to identify - not to implement) solutions if this is indeed the case:

1. Throw hardware at the bottleneck. You're right that one calculation can be computed at a time, though with today's systems you can get MANY cores working simultaneously in distributed models for parallel processing. If each zone or area has its own set of hardware (physical or virtual), and a logical event (such as getting out of combat - if combat actually ended and wasn't bugged) triggered the switch of your character's processing to another set of hardware, then adding processing power and transaction speed (more front ends for database server interactions and better read speeds) should help for each zone. There should also be a standby set of "extra" hardware (likely virtual) that can be pulled into the pool for each zone when big fights happen. Example - when capability for processing reaches 80%, add additional VMs until they're at 60%, then as needs lower, release some of the pool back to the standby set.

2. Optimize code. With every bit of damage or healing, calculations include base damage, gear bonuses, positive effects and buffs, negative effects and debuffs, CP calculations (we can see no-CP is a lot less laggier than CP). Those calculations exist for the numbers going out from a player, then for the numbers hitting the players that it hits. Then there are the application of additional (positive or negative) effects for each tick of a DoT or HoT. The more that those can be standardized and calculations transformed into simpler equations, the less calculations that have to be performed by the client(s) and checked by the server(s).

JTorus

Crown wrote: »

JTorus wrote: »

TL;DR Lag is the result of an intentionally implemented 'checks and balances' system to reduce blatant exploitation, and cannot be removed. The only solution is to continue to address the symptoms, and 'distract' everyone enough with other objectives *cough*daedric artifacts*cough* in order to maintain a bit of server integrity.

@JTorus There are two relatively simple (to identify - not to implement) solutions if this is indeed the case:

1. Throw hardware at the bottleneck. You're right that one calculation can be computed at a time, though with today's systems you can get MANY cores working simultaneously in distributed models for parallel processing. If each zone or area has its own set of hardware (physical or virtual), and a logical event (such as getting out of combat - if combat actually ended and wasn't bugged) triggered the switch of your character's processing to another set of hardware, then adding processing power and transaction speed (more front ends for database server interactions and better read speeds) should help for each zone. There should also be a standby set of "extra" hardware (likely virtual) that can be pulled into the pool for each zone when big fights happen. Example - when capability for processing reaches 80%, add additional VMs until they're at 60%, then as needs lower, release some of the pool back to the standby set.

2. Optimize code. With every bit of damage or healing, calculations include base damage, gear bonuses, positive effects and buffs, negative effects and debuffs, CP calculations (we can see no-CP is a lot less laggier than CP). Those calculations exist for the numbers going out from a player, then for the numbers hitting the players that it hits. Then there are the application of additional (positive or negative) effects for each tick of a DoT or HoT. The more that those can be standardized and calculations transformed into simpler equations, the less calculations that have to be performed by the client(s) and checked by the server(s).

I doubt Option 1 is viable. The MegaServer ™ © is already a server cluster. When you queue up dungeons, trials, etc, it's already spinning up a VM, Container, kubernetes, whatever they use. I suspect they can't offload anything else, because at this point, it'd ruin the user experience (loading screens between farm, and keep, and bridge, etc). Odds are, as well, that most pertinent information is stored in RAM or buffered to some sort of non-platter storage. (Similar to how ZFS works).

Option 2 however... well, some people raise holy hell when a siege engine becomes lethal (*gasp*). Imagine 'nerfing' proc sets, DOTs HOTs etc.

And lets face it, this is only in PvP where these issues generally show up. How many players are allowed in a campaign? per faction? How many campaigns are there? PvP isn't even 1/10th of ESO's population.

I wouldn't want to deal with it either, for as mean as this crowd can be sometimes.

zyk

The solution to lag is exactly the same as the solution for high frame times/framerates. You meticulously design content so your players can't break the game.

When a level designer creates a new area, they have a budget that limits how much they can have drawn in any given area. This is tested and honed through the development process.

In the same way, ESO game designers should have a processing budget enforced. Unfortunately, they do not. All content is designed and profiled for 12 player instances. It is no wonder it breaks in Cyrodiil.

Regarding security, the ESO client has always been and remains fundamentally trusted. This has been proven over and over again via memory hacking. There is a theory that improved security is why Cyrodiil became laggy over the summer 2014, but it is only a theory. ZOS denies it and has provided an alternate explanation. I write this not to argue about it, but because we need to share facts to the greatest degree possible.

Regardless, the main problem is simply that ZOS designed gameplay their technology can't handle. The true solution is a separate set of rules for Cyrodiil with a proper processing budget.