For those PTS affected accounts, why both locked out while going thru account and a rollback?

Maxamoo · April 16

First, I will admit that I do not understand game design and data storage management, so there may be a very simple answer to this that I do not understand. However, I was wondering why I need to be both locked out of my account while they go over it AND then to have it rolled back?

To prevent the use of assets which we were not supposed to have access to or were not properly earned on the live server, I completely understand, although am not overjoyed, by the immediate reaction to shut down the accounts. It seems if they rollback the affected accounts to before the PTS release, we would not have access to those assets. The accounts could be unlocked and we would not be further inconvenienced by a problem not of our making. ZOS could still go through your account with a fine toothed comb to see if you did anything inappropriate with those assets while you had them, and take appropriate action if discovered. When the Outfit system went a bit bonkers a few weeks ago, they were able to look at who was affected and adjust their accounts without keeping you locked out of the game.

Does anyone have any constructive insight to this?

Maxamoo · April 16

Conversely, if ZOS feels they need to go over each account individually to remove the offending assets, then why the need to roll back the accounts?

NoSoup · April 16

I imagine it's done in this order to see what, if any, assets were distributed to other accounts.

fizl101 · April 16

They have to roll back as some people made changes like deleting charaCters on pts to create test characters and itcdeleted them on live, peoples houses changed because they changed them on pts and so on

sarahthes · April 16

Maxamoo wrote: »

Conversely, if ZOS feels they need to go over each account individually to remove the offending assets, then why the need to roll back the accounts?

Probably to take away the extra champion points they gained, in addition to the things others have mentioned.

Maxamoo · April 16

Again, I understand the need for the roll back...but if that is done, why do we need to continue to be locked out of our accounts while they finish their investigation to see if assets were used inappropriately. If the assets are still on the affected account, the roll back would remove them (or restore them.) If the assets were already sent to unaffected accounts, they are currently in game, so continuing the lock out seems excessive. They will still need to go through the accounts, bit by bit, but they could do that while we have access to our accounts.

Just looking for an explanation as to why both are needed to be done simultaneously.

gamergirldk · April 16

mate there have been an influx of billions of gold into the server pp, have been on a rampart spending spree

jaws343 · April 16

Maxamoo wrote: »

Again, I understand the need for the roll back...but if that is done, why do we need to continue to be locked out of our accounts while they finish their investigation to see if assets were used inappropriately. If the assets are still on the affected account, the roll back would remove them (or restore them.) If the assets were already sent to unaffected accounts, they are currently in game, so continuing the lock out seems excessive. They will still need to go through the accounts, bit by bit, but they could do that while we have access to our accounts.

Just looking for an explanation as to why both are needed to be done simultaneously.

I imagine they can't just hit a button and roll back the accounts automatically. They probably have to do them individually, and since that is already likely to take a while, they are probably reviewing for impropriety at the same time.

It'd likely be a different story if they reset the entire server all at once, but picking and choosing is probably where this gets messy.

virtus753 · April 16

Maxamoo wrote: »

Again, I understand the need for the roll back...but if that is done, why do we need to continue to be locked out of our accounts while they finish their investigation to see if assets were used inappropriately. If the assets are still on the affected account, the roll back would remove them (or restore them.) If the assets were already sent to unaffected accounts, they are currently in game, so continuing the lock out seems excessive. They will still need to go through the accounts, bit by bit, but they could do that while we have access to our accounts.

Just looking for an explanation as to why both are needed to be done simultaneously.

I would guess the rollbacks aren't finished yet. They are having to do them individually, so it may take the couple of days Gina cited.

Maxamoo · April 16

gamergirldk wrote: »

mate there have been an influx of billions of gold into the server pp, have been on a rampart spending spree

Unfortunately keeping everyone locked out of their accounts doesn't change that.

Just for clarification, why do both the roll back AND lock out need to happen at the same time?

Maxamoo · April 16

virtus753 wrote: »

Maxamoo wrote: »

Again, I understand the need for the roll back...but if that is done, why do we need to continue to be locked out of our accounts while they finish their investigation to see if assets were used inappropriately. If the assets are still on the affected account, the roll back would remove them (or restore them.) If the assets were already sent to unaffected accounts, they are currently in game, so continuing the lock out seems excessive. They will still need to go through the accounts, bit by bit, but they could do that while we have access to our accounts.

Just looking for an explanation as to why both are needed to be done simultaneously.

I would guess the rollbacks aren't finished yet. They are having to do them individually, so it may take the couple of days Gina cited.

Thank you virtus!! That makes a bit of sense to me!

Zoye · April 16

Maxamoo wrote: »

Again, I understand the need for the roll back...but if that is done, why do we need to continue to be locked out of our accounts while they finish their investigation to see if assets were used inappropriately. If the assets are still on the affected account, the roll back would remove them (or restore them.) If the assets were already sent to unaffected accounts, they are currently in game, so continuing the lock out seems excessive. They will still need to go through the accounts, bit by bit, but they could do that while we have access to our accounts.

Just looking for an explanation as to why both are needed to be done simultaneously.

I can't imagine the amount of work and data verification that they need to do to ensure that everyone gets put back to the roll back point and done with out further issues.

According to what Gina said here you should get an email letting you know when that is done.

We have temporarily locked the affected accounts until we can restore them to a good state and will be notifying those players via email, and will be rolling back progress on PC NA from those accounts to April 15th at 9:30am EDT. This will take several days to complete and we will alert any affected players when their account is accessible again.

MoonPile · April 16

Maybe cleaner to figure it all out in one sweep, rather than letting the bad actors loose again, to potentially cause further chaos with those items they released to begin with, and having to re-ban.

DoofusMax · April 16

This will be kind of long-winded, but It's not really a cut-and-dried question. As I understand it, the first part of the problem comes back to how PTS is populated with accounts. What normally happens is that the PC NA player data is copied over to PTS on the first day of testing. A couple of weeks later, that player data gets wiped and the PC EU player data is copied over. When I logged in to PTS yesterday, I found it properly populated with my PC NA toons, properly equipped and sitting right where I had parked them when I last logged off. So far so good.

The second part of the problem is that PTS exists in its own server space. Think of it as "what happens on PTS stays on PTS" and you won't be too far off the mark. It's a great place to go try out builds or decorating ideas or check out new content or an upcoming event or whatever because what you do on PTS doesn't propagate beyond PTS beyond any story spoilers you might have hit. That includes Crown Store purchases (everything costs 1 crown and your crowns replenish daily). You can start a new character at Level 50, CP3600, with the sweetest gear in the game. If you are at the limit on your character slots and need to delete one or more characters to make room for one or more behemoths (or a new character if you want to go low-level), you can do that. Think of all the things you can do with infinite purchasing power, virtually infinite materials, instant access to all the gear in the game, and the most meta of meta characters. That's what's possible on PTS.

The key part of the problem is that changes on PTS are not supposed to propagate beyond PTS. Yesterday, they did. People who deleted characters on PTS to make room for that behemoth (or just start over) found their character(s) deleted on the live server, and the behemoth (and all its goodies) occupying a character slot. Crown Store purchases also went to the live server. The few Runestones that I harvested on my way to Skingrad (one must keep Ember happy) probably ended up in my live-server craft bag. For reasons which have not yet been revealed, player data on PTS and the PC NA live server were not isolated from each other.

Now put yourself into the shoes of the developer facing this problem. You could determine a point in time at which you know that the player data on the live server is absolutely correct, take down the live server, and restore everyone on the live server to that point in time. This would catch all the PTS players and their data, but it would also include the people who were going about their normal playing on the live server in that few-hours window between "known good" and "oops!" So you'd be rolling back whatever achievements they proc'd, whatever goodies they acquired, whatever they purchased from the Crown Store, whatever outfit changes they made, whatever respecs they did, and all the rest. You'd basically be saying "it was all a dream" (showing my age with that one). This would not make for a server full of happy campers.

Alternatively, you could restrict your rollback to only the players who logged in to PTS since they are the accounts with the potential problems. But since you are going to be selective about the accounts you roll back, what do you do about their live-server characters while you are doing that? The most direct method of handling those is to simply lock them in place while you do those rollbacks. Would I like to be able to log on and play for a while? You bet. But I now know that my account is going to be rolled back to its state at 9:30am EDT on April 15th (I was making morning coffee and wasn't even online at that time, so no loss), but anything that I do between now and the rollback will get wiped when the rollback goes in. Less frustrating for everyone to simply keep my account locked until the rollback because I'd be highly upset to pop a few Aetherial Dust out of some reward coffers only to have it wiped away by the rollback.

blktauna · April 16

good version control systems will help in matters like this. I used to be able to pinpoint changes on dev, qa and live servers easily by checking the deploymeys comments.

Maxamoo · April 16

Thank you DoofusMax! I understood what happened, but not why they choose the route to fix it that they did. Your explanation helps and I really appreciate the time you took to explain it all.

Maxamoo · April 16

and yes, we did get an email that said our accounts were frozen and a rough time frame for return. But it lacked any details about the "why we are doing it this way" and there has been no communication (via email) since. I appreciate everyones efforts in trying to help me understand it all.

Elrond87 · April 16

why did it effect live server accounts pts is totally separate?

virtus753 · April 16

Elrond87 wrote: »

why did it effect live server accounts pts is totally separate?

It wasn't totally separate, though.

The PTS went live a little too literally yesterday. Anyone logged into the PTS yesterday was actually creating/deleting characters on live.

That meant any template character boosted the account to 3600 CP and potentially introduced a ton of gold and mats and sets into the game. That's why they had to shut both servers and decide how to clean it up.