what even *is* sharding and cold storage?

StabbityDoom

Has this ever been explained? I'm curious to know what it means.

PeacefulAnarchy

It means old accounts that haven't been logged into in x months (I'm not sure they've said how many) are moved to a backup cold storage db, rather than the live db. If those people return, their first login will take a bit longer as the data is moved back to live but in the meantime the live db is smaller.

Tommy_The_Gun

I am not sure if this is the same thing, but I remember that some time ago ZOS was saying something about placing inactive accounts to a separate queue or something like this, so the server would not have to deal with excess accounts that are not being used often.

So, if a player does not play the game anymore, their account will be moved to a different pool, to make login faster for player accounts that are used regularly. If a player will return to the game after long break, the 1st login will take longer, as the server will move the account from "storage" to normal pool of accounts.

This is at least how I understood this.

StabbityDoom

That makes sense, then what is sharding?

deleted221106-002999

StabbityDoom wrote: »

Has this ever been explained? I'm curious to know what it means.

Accounts that haven't been logged into for ages are split off from active and stored separately.

The effect for the account holder, if they choose to return, is there will probably be a pretty hefty delay in logging in for the first time as the account data is retrieved from the 'cooler' and brought into the active database. I seem to recall numbers like 5-10minutes being mentioned but you'd have to search for specific estimates from zos.

I think it was announced as part of the performance updates program when we still collectively had confidence and hope that there was sincerity in the claims.

BlueRaven

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

MageCatF4F

StabbityDoom wrote: »

That makes sense, then what is sharding?

I think it is a misspelling. It has that red squiggle under it. They meant "shredding".

sarahthes

MageCatF4F wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

I think it is a misspelling. It has that red squiggle under it. They meant "shredding".

In video games it is sharding. The explanation above is correct.

Have you ever grouped with someone and been unable to see them because they're in a different version of Wayrest even though their icon shows them right next to you? That's an example of sharding, or instancing.

MageCatF4F

/s? What happened to my performance however is shredding.

LostHorizon1933

Thank you @BlueRaven , I think I suddenly understand why I “zone out” in Mournhold (go to a quick loading screen while walking in the city).

sarahthes

MageCatF4F wrote: »

/s? What happened to my performance however is shredding.

Can't argue that LOL.

Varana

BlueRaven wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

But the game has worked like that for... all the time. There are different instances of game zones - every time you group up with someone and don't see them but only their group member arrow, they're in a different instance from you.

I'm not quite sure they ever explained how the new "sharding" differs from the one that has always been part of the game.

Iselin

BlueRaven wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

I thought that was the whole point of the megaserver. Hasn't this been the case since 2014?

BlueRaven

Varana wrote: »

BlueRaven wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

But the game has worked like that for... all the time. There are different instances of game zones - every time you group up with someone and don't see them but only their group member arrow, they're in a different instance from you.

I'm not quite sure they ever explained how the new "sharding" differs from the one that has always been part of the game.

Iselin wrote: »

BlueRaven wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

I thought that was the whole point of the megaserver. Hasn't this been the case since 2014?

I do not know what improvements to sharding they are making, only that it will improve load times, etc.

Maybe they will be adding more shards? Fixing bottleneck issues? Improving the code that determines what shard you are going to? A combination? Something else?

And in any case the question I was answering was “What is sharding?” not what are the improvements/changes going to be.

Gaeliannas

I do not think they are using Sharding in a game sense here as many may be thinking. In game terms sharding is basically the same as instancing, which ESO has done since launch.

They are most likely referring to database sharding:
https://en.wikipedia.org/wiki/Shard_(database_architecture)

PeacefulAnarchy

I'm not sure they've specified, but the way they've posted about it makes me think they're two parts of the same big process. They're sharding (splitting up) the database, and taking one (or several) of those splits from those who haven't logged in into cold storage.

Zone sharding already exists and while they could change the way that works it doesn't seem like something that would be related to cold storage or the db in general.

Gaeliannas

.

Kesstryl

Gaeliannas wrote: »

I do not think they are using Sharding in a game sense here as many may be thinking. In game terms sharding is basically the same as instancing, which ESO has done since launch.

They are most likely referring to database sharding:
https://en.wikipedia.org/wiki/Shard_(database_architecture)

Why couldn't this have been done PRIOR to deleting our character histories? This should have been the first step before taking something away from us. AwA is fine, but deleting our individual character achievements is WRONG! Wrong, wrong, wrong! They should have explored every avenue to preserve what we have before taking anything away.

Vevvev

StabbityDoom wrote: »

That makes sense, then what is sharding?

It's cloud storage in a database like Cassandra. Basically it's a no-SQL database that stores data on shards which are little islands of information.

Instead of querying the database and waiting for it to find the item you instead send out a call and wait. The shards that hear the call search for the item and then all the shards that have it respond to you. At the end of the wait you look at all the returns and work with those.

It's much faster with dealing with millions if not billions of records, but very slow in a very small scale environment. ESO is a massive game and sharding is the best way to do data storage and retrieval.

Casdha

I hope they take into consideration whether or not you have an active ESO+ account.

They should never bump someone to cold storage who gives them an active income regardless of time between log-ins.

LoneStar2911

Casdha wrote: »

I hope they take into consideration whether or not you have an active ESO+ account.

They should never bump someone to cold storage who gives them an active income regardless of time between log-ins.

Why would that even be important or taken into consideration? If you’re not playing, you’re not playing. And if you’re not playing, it has zero effect on you. Except for the first time you log back in again, you might get a loading screen that takes several seconds more.

Celephantsylvius_Bornasfinmo

*Players whom haven't played in a few years*

https://www.youtube.com/watch?v=GmCsJDpoGgU&ab_channel=MidasFury

*Salutes from the atrium above*

'God speed, Veteran 14...' *nods*

LesserCircle

Youyouz06 wrote: »

*Players whom haven't played in a few years*

https://www.youtube.com/watch?v=GmCsJDpoGgU&ab_channel=MidasFury

*Salutes from the atrium above*

'God speed, Veteran 14...' *nods*

vR14 oh no, I forgot about that

Dark_Lord_Kuro

Kesstryl wrote: »

Gaeliannas wrote: »

I do not think they are using Sharding in a game sense here as many may be thinking. In game terms sharding is basically the same as instancing, which ESO has done since launch.

They are most likely referring to database sharding:
https://en.wikipedia.org/wiki/Shard_(database_architecture)

Why couldn't this have been done PRIOR to deleting our character histories? This should have been the first step before taking something away from us. AwA is fine, but deleting our individual character achievements is WRONG! Wrong, wrong, wrong! They should have explored every avenue to preserve what we have before taking anything away.

I was done prior tough, on console. Pc is the last one to receive cold storage

kargen27

BlueRaven wrote: »

Varana wrote: »

BlueRaven wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

But the game has worked like that for... all the time. There are different instances of game zones - every time you group up with someone and don't see them but only their group member arrow, they're in a different instance from you.

I'm not quite sure they ever explained how the new "sharding" differs from the one that has always been part of the game.

Iselin wrote: »

BlueRaven wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It’s automatically splitting up the player base in a particular zone into different but essentially identical shards of it.

In other words you might be porting to grahtwood, but you may be actually porting to grahtwood - a, or grahtwood - b, etc. it’s a way off splitting up the player base in a particular area so the weight of players dissipates lag as they are distributed amongst different servers.

If you ever played wow, you may remember the player base being distributed into different servers? (Burning Blade, etc)
Same thing, but here the servers overlap/switches more. And a lot of it is done “behind the scenes” so to speak.

I thought that was the whole point of the megaserver. Hasn't this been the case since 2014?

I do not know what improvements to sharding they are making, only that it will improve load times, etc.

Maybe they will be adding more shards? Fixing bottleneck issues? Improving the code that determines what shard you are going to? A combination? Something else?

And in any case the question I was answering was “What is sharding?” not what are the improvements/changes going to be.

I keep seeing an emphasis on loading times. Most the time though loading isn't a problem. I would gladly take a few extra seconds of load screens if the performance one I am in game was improved. Bar swap failing and needing to mash the button several times to get an ultimate to finally pop is much more annoying than a long load screen.
If these changes are simply to improve load times I think they are prioritizing the wrong thing.

RawBrightSilver

One problem that users sometimes encounter after having sharded a database is that the shards eventually become unbalanced. If that occurs, any benefits of sharding the database are canceled out by slowdowns and crashes. The database would likely need to be repaired and resharded to allow for a more even data distribution.

Have we been experiencing anything like that lately?

Crismac

Sharded means I just.... no not that. Cold Storage is when someone died... no not that either never mind. I don't know.

coletas

Vevvev wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

It's cloud storage in a database like Cassandra. Basically it's a no-SQL database that stores data on shards which are little islands of information.

Instead of querying the database and waiting for it to find the item you instead send out a call and wait. The shards that hear the call search for the item and then all the shards that have it respond to you. At the end of the wait you look at all the returns and work with those.

It's much faster with dealing with millions if not billions of records, but very slow in a very small scale environment. ESO is a massive game and sharding is the best way to do data storage and retrieval.

Is faster if you can split querys by a intelligent key. Using a key like "player didnt connected in years, months or hours" or is constantly connected makes no difference on concurrency so there is not going to be any improvement except if the current database architecture is terrible indexed where yeah, having less data would improve a really little the time of the query. For example would be more intelligent key (not the best) to have a shard for accounts beginning with a,b,c characters, other with d,e,f and so on so you can move/scale those querys to different servers. Having a shard with innactive players and a live one with recent players is just pffff is not going to make it scalable in any way... And doing the data move when a user login is another "great" idea lol add to that a bad clustered index and you have a perfect storm lol

Lysette

As far as I got the time to get into cold storage is a year - and if someone is not playing for a year, it is reasonable to put his account into cold storage. I guess it won't bother him as well, if he has to wait for a few minutes to get reactivated.

This said, that this is considered at all sounds to me like a proprietary data base and not a commercial one - it shouldn't make much of a difference if you handle 10 million or a 100 million records - that is going from 2^24 to 2^27 - just 12.5% more effort to grab the record. And in the other direction if you go from 10 million to just 1 million, that is going from 2^24 to 2^20, you save as well just 16.7%, not a big gain.

_Zathras_

MageCatF4F wrote: »

StabbityDoom wrote: »

That makes sense, then what is sharding?

I think it is a misspelling. It has that red squiggle under it. They meant "shredding".

That explains everything.

PeacefulAnarchy

RawBrightSilver wrote: »

One problem that users sometimes encounter after having sharded a database is that the shards eventually become unbalanced. If that occurs, any benefits of sharding the database are canceled out by slowdowns and crashes. The database would likely need to be repaired and resharded to allow for a more even data distribution.

Have we been experiencing anything like that lately?

I doubt it. From what I understand there's only one live shard. Pulling things into the live shard is done by single users, so whatever performance cost would probably be borne by that user in terms of load times. As far as writing to the db it shouldn't be any more intensive, or generally frequent, than a new character or account being created. As for moving things off the live shard I would hope that's something done biweekly during the maintenance.

If the performance issues are due to sharding then it might be that they did some other db change besides that at the same time, but the sharding itself, in their implementation, shouldn't cause those issues.