SERVER DOWN?

Elldarian

ZOS_GinaBruno: (11:18 EDT)

"Just checking in to let everyone know investigations are continuing, and both PC NA and PTS will remain under maintenance until further notice. We'll provide another update in an hour or two regardless of the status."

DewiMorgan

Arizona_Steve wrote: »

I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.

Ideally the live DB from which they are copying should be a read replica, not the actual writable database.

(Software engineer for > 30 years)

I'd expect it to be on-demand copies, since they switch from one server to another as the source of truth on a regular basis.

So rather than copy all the millions of accounts over to PTS every time they bring it up, they'd just copy over the dozens of people who log into PTS, on demand, as they first log in. Way less data transfer, so way less downtime for data migration.

At that point it becomes a simple config thing, listing one server as the source, one as the destination: so simple, even the intern can do it. Until they get two servernames the wrong way round, and pow: they have this bug.

And since it's such an uncommon task (once every few months) and such a trivial edit (couple of lines), and everyone editing them should know what they're doing, at development time it would not have seemed worth spending the time to put any safeties in place to prevent putting the live servers as destinations instead of sources.

That's just a wild guess, though. There are quite a few other ways I could think of to cause the same issues, by messing up routing, message queues, caches, etc.

ragnarok6644b14_ESO

It could even be a login server bug - I've noticed some weird login server errors lately (lots of 103s and 307s for me, more than usual).

Imagine the databases are correctly configured and everything is ready to go, but a malfunctioning login server draws from both databases because it's error catching has been told to look for the database, and if it doesn't find it, look again, and keep looking until you find it -

- and, ofc, in a list of all databases that normally includes one, having a second after it in the load order could be excitingly confusing.

Of course this is entirely speculation, I usually work with air-gapped LANs, but it sounds worse than just a config correction.

Arizona_Steve

DewiMorgan wrote: »

Arizona_Steve wrote: »

I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.

Ideally the live DB from which they are copying should be a read replica, not the actual writable database.

(Software engineer for > 30 years)

I'd expect it to be on-demand copies, since they switch from one server to another as the source of truth on a regular basis.

So rather than copy all the millions of accounts over to PTS every time they bring it up, they'd just copy over the dozens of people who log into PTS, on demand, as they first log in. Way less data transfer, so way less downtime for data migration.

Makes a lot of sense, given the ratio of accounts using the PTS vs the number of accounts on the live server.

ArchangelIsraphel

.

DewiMorgan wrote: »

Arizona_Steve wrote: »

I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.

Ideally the live DB from which they are copying should be a read replica, not the actual writable database.

(Software engineer for > 30 years)

I'd expect it to be on-demand copies, since they switch from one server to another as the source of truth on a regular basis.

So rather than copy all the millions of accounts over to PTS every time they bring it up, they'd just copy over the dozens of people who log into PTS, on demand, as they first log in. Way less data transfer, so way less downtime for data migration.

At that point it becomes a simple config thing, listing one server as the source, one as the destination: so simple, even the intern can do it. Until they get two servernames the wrong way round, and pow: they have this bug.

And since it's such an uncommon task (once every few months) and such a trivial edit (couple of lines), and everyone editing them should know what they're doing, at development time it would not have seemed worth spending the time to put any safeties in place to prevent putting the live servers as destinations instead of sources.

That's just a wild guess, though. There are quite a few other ways I could think of to cause the same issues, by messing up routing, message queues, caches, etc.

The thing is though, the copies of our accounts we get on the test server are usually from a few days ago- they are never copies of our current, live accounts. There's usually a 3-5 day difference between our live account, and the copy on the test server. So I don't think they're making on-demand copies.

TaSheen

Just glad I was too ill to log into game at all today (first time I've ever been happy being sick for sure)!

fireshine6623

DewiMorgan wrote: »

I've a very vague memory of them adding an additional day or two to ESO Plus in the past, to cover an extended period of flakiness, though I can't remember why, and may be misremembering. I don't think it matters, but could be a nice goodwill move if technically possible.

Personally, I'm hoping they extend the event by a day: they do that quite often, too.

And yes: ZoS aren't the people I'd be angry at over this.

At least in my experience, fixing the issue isn't usually what takes the time. Even cleaning up the first-order problem doesn't take too much time.

What takes time is cleaning up after all the jerks who exploited the bugs.

Not just the time it takes to detect and ban them, that's easy: but tracing down the items they transferred to other people. Hard enough with things like attunable stations, but for stackable stuff like gold, it's a *nightmare*! Alice uses the exploit to get 2 million gold, and gives Bob 1M, but he already had some money, so now he's at 1.5M. Bob trades Claire 0.5M for Kuta, and Dave 0.5M for an undetectable trade. Who, now, do you remove the gold from?

Rinse, repeat, for every single person exploiting, every trade they made, every trade that was made on from that... some of the attunable stations will have got attuned and merged with the all-in-one stations, so that needs to be backed out... other stuff bought or traded was consumable and has been consumed... all just a gigantic mess that some poor engineers are having to work several hours of unpaid overtime to manually trace through and pick out all the threads and fix.

And they have to do it before the server comes up. Because if they bring it up and then work on unpicking those threads while the server's up and people are able to log in, then people will work frantically to launder their ill-gotten gains, or scam people with their imaginary gold, so the job will become a hundred times harder and make a hundred times more players get ripped off.

At least, that was my experience as an MMO dev.

If we see an exploitable bug, and we exploit it, we're screwing all of us by extending the downtime. We're not going to get to keep the stuff anyway, and we might get a permanent ban for doing it, losing everything. It's not worth it.

So we should all try not to be that dude: instead, /bug and report it.

And grouch at the damn exploiters for causing this downtime. Bugs happen, but I'm betting it'd all be back up by now if not for their greedy selfishness.

I've got something in a trunk in game somewhere called the "blanket treatment" for exploiters. Let's treat them as a triple XP dolmen and see how many people show up. Exploiters are unethical and irresponsible, and have no place in this community. As you know already Dew. Thanks for speaking out against them, you have my 100% full support.

stephanierain

When Gina says "regardless of status" that's shorthand for "strap in folks, this is going to be a long one".

See you next monday

fireshine6623

@ZOS_GinaBruno @staff just a TLC ping, you've got all of our love here. Take your time, make sure everyone gets to sleep, eat dinner, don't worry about us. We're all with you. Lots of support here (and if not I have an Orc tank that will be asking the questions, and he is not friendly about negative nellies....) Godspeed, take care of yourselves amid the chaos.

DewiMorgan

Sakiri wrote: »

The thing is though, the copies of our accounts we get on the test server are usually from a few days ago- they are never copies of our current, live accounts. There's usually a 3-5 day difference between our live account, and the copy on the test server. So I don't think they're making on-demand copies.

Oh, good argument! And that then makes the current case even weirder, then, since the copies from PTS to PC-NA were not several days delayed. Well, at least so far as I've heard, I've no good sources though.

Ooh! Maybe they were testing out a new on-demand copy system, and had it configured backwards?

krayphysh

How far back would a rollback go? I haven't played since last night...should I log on later expecting lost items? I've got my account maxed out with toons, have done writs on all of them daily since the event, and spent ridiculously long hours grinding for those style pages...will all the time and goods I've earned all be gone?

hiyde

krayphysh wrote: »

How far back would a rollback go? I haven't played since last night...should I log on later expecting lost items? I've got my account maxed out with toons, have done writs on all of them daily since the event, and spent ridiculously long hours grinding for those style pages...will all the time and goods I've earned all be gone?

We're not going to know until they fix the problem and tell us how they fixed it.

Until then, we're all just guessing, I wouldn't panic...yet.

StackonClown

Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?

VaxtinTheWolf

Oh boy.. It's getting late here and I just sat down to open up the game before reset in 6 hours to do the usual stuff before sleeping and apparently this has been going on. Just.. Lovely.

Well, I hope things magically get resolved before then. Maybe a hamster was taking an extended break. Maybe quietly replace it with a more motivated one yes?

majulook

hiyde wrote: »

krayphysh wrote: »

How far back would a rollback go? I haven't played since last night...should I log on later expecting lost items? I've got my account maxed out with toons, have done writs on all of them daily since the event, and spent ridiculously long hours grinding for those style pages...will all the time and goods I've earned all be gone?

We're not going to know until they fix the problem and tell us how they fixed it.

Until then, we're all just guessing, I wouldn't panic...yet.

I would be surprised if they give much more than "We restored / recovered / fixed the issue, all accounts are up as of xxxx time prior to taking the servers down. we have extended the event buy xxxx amount of time.

LyricsEcho

Well... Six hours downtime it must be serious...i was about to buy the new chapter today too.

EchoesofThunder

StackonClown wrote: »

Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?

If you do a search, you will find there were quite a few in the early days, and even some in 2022, but they were very short rollbacks (2 to 10 minutes or so), to counter crashes.

WolfStar07

StackonClown wrote: »

Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?

Not sure if you're being sarcastic, but yes. The most recent rollback I'm aware of only affected Cyrodiil since that's on a different server from PVE, and that was earlier this month.

L3giôn

Is this common on ESO? I just bought this game and spent a lot (a LOT) on crowns and other things to get me going. Once I realized there was only one US server, my spider senses started to tingle... that's not a lot of eggs in one basket, that's literally all of them in one basket. Makes me a bit nervous considering how much I spent.

Pyr0xyrecuprotite

There was a bug causing random small rollbacks in PvE as well a few years ago (PC anyway, not sure if that hit the other platforms). It was very weird at the time. And I think there have been one or two intentional server-wide rollbacks other than Cyro too. Quite a while ago tho.

Sakiri

L3giôn wrote: »

Is this common on ESO? I just bought this game and spent a lot (a LOT) on crowns and other things to get me going. Once I realized there was only one US server, my spider senses started to tingle... that's not a lot of eggs in one basket, that's literally all of them in one basket. Makes me a bit nervous considering how much I spent.

Not common.

Giraffon

Still, how can we celebrate 10 years of ESO without a little server down time?

StackonClown

WolfStar07 wrote: »

StackonClown wrote: »

Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?

Not sure if you're being sarcastic, but yes. The most recent rollback I'm aware of only affected Cyrodiil since that's on a different server from PVE, and that was earlier this month.

Hehe, i was being srs!
but then how many hours /days did they roll it back??
I didnt really notice anything and I log in most days

WolfStar07

StackonClown wrote: »

WolfStar07 wrote: »

StackonClown wrote: »

Has there ever been a Live server rollback in the history of ESO??
I cant remember that this ever happened?

Not sure if you're being sarcastic, but yes. The most recent rollback I'm aware of only affected Cyrodiil since that's on a different server from PvE, and that was earlier this month.

Hehe, i was being srs!
but then how many hours /days did they roll it back??
I didnt really notice anything and I log in most days

I'm pretty sure it was only hours, but I didn't PvP that day (got the info from my PvP guild), so I didn't pay too much attention to the specifics. I am hoping that with this anticipated server wide rollback that it's also only a couple of hours to coincide with when PTS went live.

Ravenson

WolfStar07 wrote: »

I'm pretty sure it was only hours, but I didn't PvP that day (got the info from my PvP guild), so I didn't pay too much attention to the specifics. I am hoping that with this anticipated server wide rollback that it's also only a couple of hours to coincide with when PTS went live.

I hope you're right.

Darquer

Over two hours have passed since the last update @ZOS_GinaBruno... Anything?

Runesong

What is the most extensive rollback you've ever experienced in your gaming life?
(My character will be gone if it's a week or so)

I feel bad for ESO.
I feel bad for the players.
I feel bad for the 'bad actors' that thought it was ok to take what they wanted without a thought for the ramifications.

I hope it wasn't ransomware...someone mentioned that in another post.

At the end of the day, I think sending good thoughts to the ESO community can't hurt and might help. We'll get through this and have a wild tale to tell.

Cheers.

stephanierain

Runesong wrote: »

What is the most extensive rollback you've ever experienced in your gaming life?
(My character will be gone if it's a week or so)

I feel bad for ESO.
I feel bad for the players.
I feel bad for the 'bad actors' that thought it was ok to take what they wanted without a thought for the ramifications.

I hope it wasn't ransomware...someone mentioned that in another post.

At the end of the day, I think sending good thoughts to the ESO community can't hurt and might help. We'll get through this and have a wild tale to tell.

Cheers.

I'll tell you what would be best.

If everytime this happened, the forums would also go down. That way, everyone would be shitting themselves at a much higher level.

Only good things could come from higher level shitting.

lydra

@ZOS_GinaBruno
Any chance for an update? It's been almost 3 hours since the last one.

I know if it was good news you'd already have told us, but even just a status update would be appreciated!

StackonClown

Honestly, I didn't grind for any of the event goodies so cant say I will 'lose' anything, but I can sympathize with those that may lose hours and hours of going in circles - hopefully there might be some small compensation for those affected..

Though if someone is in the middle of a vet trial for example not sure what would be reasonable here though?
Lets hope only few more hours and not days