Arizona_Steve wrote: »I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.
Ideally the live DB from which they are copying should be a read replica, not the actual writable database.
(Software engineer for > 30 years)
DewiMorgan wrote: »Arizona_Steve wrote: »I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.
Ideally the live DB from which they are copying should be a read replica, not the actual writable database.
(Software engineer for > 30 years)
I'd expect it to be on-demand copies, since they switch from one server to another as the source of truth on a regular basis.
So rather than copy all the millions of accounts over to PTS every time they bring it up, they'd just copy over the dozens of people who log into PTS, on demand, as they first log in. Way less data transfer, so way less downtime for data migration.
DewiMorgan wrote: »Arizona_Steve wrote: »I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.
Ideally the live DB from which they are copying should be a read replica, not the actual writable database.
(Software engineer for > 30 years)
I'd expect it to be on-demand copies, since they switch from one server to another as the source of truth on a regular basis.
So rather than copy all the millions of accounts over to PTS every time they bring it up, they'd just copy over the dozens of people who log into PTS, on demand, as they first log in. Way less data transfer, so way less downtime for data migration.
At that point it becomes a simple config thing, listing one server as the source, one as the destination: so simple, even the intern can do it. Until they get two servernames the wrong way round, and pow: they have this bug.
And since it's such an uncommon task (once every few months) and such a trivial edit (couple of lines), and everyone editing them should know what they're doing, at development time it would not have seemed worth spending the time to put any safeties in place to prevent putting the live servers as destinations instead of sources.
That's just a wild guess, though. There are quite a few other ways I could think of to cause the same issues, by messing up routing, message queues, caches, etc.
DewiMorgan wrote: »I've a very vague memory of them adding an additional day or two to ESO Plus in the past, to cover an extended period of flakiness, though I can't remember why, and may be misremembering. I don't think it matters, but could be a nice goodwill move if technically possible.
Personally, I'm hoping they extend the event by a day: they do that quite often, too.
And yes: ZoS aren't the people I'd be angry at over this.
At least in my experience, fixing the issue isn't usually what takes the time. Even cleaning up the first-order problem doesn't take too much time.
What takes time is cleaning up after all the jerks who exploited the bugs.
Not just the time it takes to detect and ban them, that's easy: but tracing down the items they transferred to other people. Hard enough with things like attunable stations, but for stackable stuff like gold, it's a *nightmare*! Alice uses the exploit to get 2 million gold, and gives Bob 1M, but he already had some money, so now he's at 1.5M. Bob trades Claire 0.5M for Kuta, and Dave 0.5M for an undetectable trade. Who, now, do you remove the gold from?
Rinse, repeat, for every single person exploiting, every trade they made, every trade that was made on from that... some of the attunable stations will have got attuned and merged with the all-in-one stations, so that needs to be backed out... other stuff bought or traded was consumable and has been consumed... all just a gigantic mess that some poor engineers are having to work several hours of unpaid overtime to manually trace through and pick out all the threads and fix.
And they have to do it before the server comes up. Because if they bring it up and then work on unpicking those threads while the server's up and people are able to log in, then people will work frantically to launder their ill-gotten gains, or scam people with their imaginary gold, so the job will become a hundred times harder and make a hundred times more players get ripped off.
At least, that was my experience as an MMO dev.
If we see an exploitable bug, and we exploit it, we're screwing all of us by extending the downtime. We're not going to get to keep the stuff anyway, and we might get a permanent ban for doing it, losing everything. It's not worth it.
So we should all try not to be that dude: instead, /bug and report it.
And grouch at the damn exploiters for causing this downtime. Bugs happen, but I'm betting it'd all be back up by now if not for their greedy selfishness.
The thing is though, the copies of our accounts we get on the test server are usually from a few days ago- they are never copies of our current, live accounts. There's usually a 3-5 day difference between our live account, and the copy on the test server. So I don't think they're making on-demand copies.
How far back would a rollback go? I haven't played since last night...should I log on later expecting lost items? I've got my account maxed out with toons, have done writs on all of them daily since the event, and spent ridiculously long hours grinding for those style pages...will all the time and goods I've earned all be gone?
How far back would a rollback go? I haven't played since last night...should I log on later expecting lost items? I've got my account maxed out with toons, have done writs on all of them daily since the event, and spent ridiculously long hours grinding for those style pages...will all the time and goods I've earned all be gone?
We're not going to know until they fix the problem and tell us how they fixed it.
Until then, we're all just guessing, I wouldn't panic...yet.
StackonClown wrote: »Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?
StackonClown wrote: »Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?
Is this common on ESO? I just bought this game and spent a lot (a LOT) on crowns and other things to get me going. Once I realized there was only one US server, my spider senses started to tingle... that's not a lot of eggs in one basket, that's literally all of them in one basket. Makes me a bit nervous considering how much I spent.
WolfStar07 wrote: »StackonClown wrote: »Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?
Not sure if you're being sarcastic, but yes. The most recent rollback I'm aware of only affected Cyrodiil since that's on a different server from PVE, and that was earlier this month.
StackonClown wrote: »WolfStar07 wrote: »StackonClown wrote: »Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?
Not sure if you're being sarcastic, but yes. The most recent rollback I'm aware of only affected Cyrodiil since that's on a different server from PvE, and that was earlier this month.
Hehe, i was being srs!
but then how many hours /days did they roll it back??
I didnt really notice anything and I log in most days
WolfStar07 wrote: »I'm pretty sure it was only hours, but I didn't PvP that day (got the info from my PvP guild), so I didn't pay too much attention to the specifics. I am hoping that with this anticipated server wide rollback that it's also only a couple of hours to coincide with when PTS went live.
What is the most extensive rollback you've ever experienced in your gaming life?
(My character will be gone if it's a week or so)
I feel bad for ESO.
I feel bad for the players.
I feel bad for the 'bad actors' that thought it was ok to take what they wanted without a thought for the ramifications.
I hope it wasn't ransomware...someone mentioned that in another post.
At the end of the day, I think sending good thoughts to the ESO community can't hurt and might help. We'll get through this and have a wild tale to tell.
Cheers.