SilverBride wrote: »

Yormula wrote: »

SilverBride wrote: »

... The story just doesn't add up.

Thank you for the insistent and continuous vote of confidence. As you may have noticed, other threads with issues based purely on disciplinary actions got that fact mentioned very quickly (and closed).

I am not trying to discredit you, but think there are too many unknowns to know exactly what is going on. Everyone is so quick to jump on the "Let's bash ZoS" bandwagon but there were still unanswered questions.

I'd like to add that ZoS did contact you on 6/14/21 as you've shown, and aren't just ignoring this issue as so many have accused them of.

Any kind of discussion of specific disciplinary actions would get a thread locked, even if the OP didn't mention that it was a disciplinary action, so it is safe to assume that is not the case here.

I think what instead is going on is something of a perfect storm of a bunch of issues. 9 months ago the world was very much in the midst of a raging pandemic, ZOS was working from home. This change in environment is partly to explain why all updates since Greymoor have been less smooth. Then you have an issue where a data-set is corrupted, which is not an easy fix especially for standard support. This might be something that requires a programmer's expertise.

If that is the case, it isn't so simple as asking a Data Systems Programmer to look at the corrupted file and <hAcKeR tYpE> on the keyboard in five minutes to fix it. They might also be working on future updates, such as Waking Flame and Deadlands, as well as 2022's year of content, potential new systems, etc., so peeling off from all of that work to help 1 customer is a fairly big ask.

With that said, this thread has gotten quite the attention, and it would be in the community's best interest if Yormula is helped to a satisfactory resolution. The solution might not be simple, but if one can be found it should be implemented.

I would say if there was a cold storage back-up or older version of Yormula's character somewhere, that it could be used to rebuild his save file. If I had something go this terribly wrong, I'd rather start over from last November instead of 2014 (although, let's be honest, I'd much rather nothing go terribly wrong so I'm going to go ahead and knock on wood until my knuckles are red).

Good lord still no resolution to this? I wish I could say I'm surprised, but I'm sadly not. I hope they eventually get this fixed for you because this seriously sucks.

Personally I would not mind if the ESO+ sub would go up in order to cover the financial burden of having readily available database backups and the protocols in place to perform a safe restore in case something goes awry.

Not just for ESO+ subscribers mind you, for players in general.

Of course this is not without its flaws; some may argue that is an exploitable avenue, others that shouldering the financial burden is unfair, however the bottom line is that running without a recovery solution as we can see with the case at hand poses a significant risk.

I know that if I were in OP's shoes no amount of dismissive 'is just a game, get a life' or 'you should not have gotten invested then' hand waving would diminish my mental anguish and frustration in any way.

Why has there been no word on this from zos yet?!

Just got caught up on the thread, and I remain:

1. Depressingly unsurprised this happened.
2. Impressed by OPs level of calm and rationality throughout the thread.
3. Astonished by ZOS's unwillingness to just come into this thread and either assure us that this can't happen to us or at least reassure us that they're working on making sure it can't happen to us.

SilverBride wrote: »

ZoS contacted the OP on 6/14, and he posted a screenshot of the message. They also are not allowed to discuss another player's specific issues publicly.

They are allowed. This is not a disciplinary action nor is it about personally identifiable information such as his real name.

My point is that they have contacted the OP. They are not just ignoring him. And ZoS cannot reassure that nothing will ever happen to anyone ever again because unforseen things sometimes just happen.

That being said, they can put out a better effort to make amends.

SilverBride wrote: »

And ZoS cannot reassure that nothing will ever happen to anyone ever again because unforseen things sometimes just happen.

Based on what can be seen in this thread yes ZOS can reassure that this would not happen again. The issue appears to be ZOS attempting to correct the OPs problem and instead breaking the character beyond repair. There are multiple references to the broken part being a save file. If it is truly a file then standard procedures in any basically competent IT shop would be to back up the file and also make a copy. Any modifications would be made to the copy which when complete and verified would be used to replace the broken original. If things went to pieces the backup could be restored leaving everything in its original state. If the save file is really an entry/entries or row/rows in a database best practices would be to make a copy of the tables in a test database, make the changes and verify them. If they were ok then a save point in the live database and the update made. In the event something went wrong updating the live database a rollback could readily be performed because of the previous save point. The key is standard procedures always allow the data being updated to be restored to its original state in the event something goes wrong. Based on the information we have available it is the lack of backup and ability to restore the character to its original state that are the issue not that the attempt to fix the problem broke things further.

Galadan wrote: »

What kind of database are not editable...

A surprising number, actually.

At least, not, user editable. This is especially true when you consider the perspective. The end user can't edit their database entry, obviously, but even the Customer Support team has limited access to the database, and limited opportunity to manipulate it, and this entire incident should illustrate why you restrict database access.

A customer support rep did get in there and modify the database, and the end result was that they destroyed the character.

Galadan wrote: »

...what kind of IT system has no logs and backups, and dev support cannot restore missing data for one single customer with this logs?

Who keeps comprehensive logs going back over 9 months on every record update? Keeping in mind that there are millions of records, and each active record receives thousands to tens of thousands of updates each day?

Where do you store these logs?

Do you even bother to log everything? (And, the answer to this one, in this case, is, "no.")

If the user's account was corrupted entirely, it's likely they could restore the entire account from a backup (or at least clone a new one from a backup), but it would mean wiping away the last 9 months of progress on the account. Something few people would be willing to accept.

And, no, it's not really that weird that the character records can't be easily edited separately from the account record. You do see stuff like this in databases, where it's not as simple as simply plugging in a new sub-record and you're good to go. Especially given that there have been issues in the past with players seeing another player's characters on their account during login. I don't have any real insight to the database architecture, but it wouldn't surprise me if you can't simply revise a character, without having to tweak a lot of other fields along the way.

Galadan wrote: »

I understand it's not a trivial task and could took a weeks of work, but data management is one of f fundamentals of every large IT infrastructure. If we lost a client data in our company, the whole IT department would be handcuffed to a computers till find a solution or manually restore it.

I remember while working for FedEx, I pulled a package off a truck. The thing weighed nearly 200lbs. The belt manager was annoyed. I was annoyed. Ground doesn't move anything heavier than 150. This thing should have been sent freight. I suggested, "maybe we should just call freight, and have them send someone over to pick it up." My superior's response was, "but, who's going to pay for that?"

Your company has a valuable contract.

@Yormula is paying 15 bucks a month (at most.) Your average coder at ZOS is making roughly 120k a year. So, if you took a programmer from the database team, told them to reconstruct this account, and it took them two weeks, that's a cost of $2500. Now, it's possible Yormula has spent more than that on ESO, but it's extremely likely that, even over seven years, they have not. Meaning reconstruction would be more expensive than @Yormula is worth as a customer. (No offense, man.)

If you're looking at a multi-million dollar contract, tying up a team for a couple weeks is still worth it, because you're throwing 20-30k at saving a contract worth 100 times that. But, that's not relevant here, because one individual customer isn't worth that much.

Galadan wrote: »

I don't beleive such strong and large team as ZOS could not do anything about this. I rather suppose it's a question of their priorities and motivation, and it's a sad thing.

ZOS, guys, please, don't leave your loyal customer, investigate such incident and show that you care.

@Yormula i truly hope that this situation will be solved for you, man. As many here, ill be wathing this thread and wait.

@Yormula is paying 15 bucks a month (at most.) Your average coder at ZOS is making roughly 120k a year. So, if you took a programmer from the database team, told them to reconstruct this account, and it took them two weeks, that's a cost of $2500. Now, it's possible Yormula has spent more than that on ESO, but it's extremely likely that, even over seven years, they have not. Meaning reconstruction would be more expensive than @Yormula is worth as a customer. (No offense, man.)

I see what you are saying & it's true that any company must do a cost analysis for any job to see if it's worth doing. However, at this stage, it's not just Yormula's character that is at stake. It is now a pretty serious PR issue if the player base loses confidence in the company's ability to have data integrity. Perhaps that customer confidence is worth justifying the work required... I hope so for all player's sakes.

amm7sb14_ESO wrote: »

Rukia541 wrote: »

This has to be the biggest gaming yikes of all time, RIP bro. I wonder if this would be fixed if they had just made achievements account wide, as it seems some timey wimey crap with the character specifically or they just don't back up data properly. I know in WoW you can straight up delete a character and get it back again, doesn't matter how long afterwards.

They can here too. When I came back in December 2019, I deleted what was my original launch character, because I wanted to start over (I was only level 25). A couple days later, I changed my mind and decided I wanted to keep my original character, so I petitioned ZOS, and they gave him back to me. I didn't have any gear upon getting him back, but they were able to restore my character.

And, that's a potential option, ask customer service to roll that character back more than 9 months, to before the original error occurred. There is a sticky problem here where we don't know what the options @Yormula has been offered. We're also not sure what options are really available.

So, this is speculation, but we can reasonably guess that the character's record was corrupted when customer support deleted the achievement. From personal experience with databases, a corrupted record can spread to other parts of the database. It is possible, though we don't know for sure, that deleting the character could cause that corruption to "migrate," to the next character record. Meaning, you'd have a (slow) cascade, as you deleted and restored characters. Given some elements of the corruption affected the account as a whole, it's possible that the account data itself is partially corrupted, meaning the entire account would need to be salvaged.

We don't know how frequent database backups are (especially older ones) as it's now been 10 months, and we don't know how having everyone working from home may have contributed to this.

Like, I get that it's a frustrating situation, but, we don't know what solutions are viable and which aren't. Again, if @Yormula has the option, a rollback may be the best they can hope for. If that is, in fact, an option.