PTS Down?

ganzaeso · April 15

Does this mean I have just lost the last 8 hours of playtime on Live?

TheMasterMeme · April 15

Same issue here, pts template character also copied to live server...

Lauranae · April 15

are you sure you did not shut down the NA too ?? server crashed at the same that you closed PTS

HeavyMetalHippie420 · April 15

It's not just pts that's down. PC NA got taken offline. They literally broke the whole game.

Gorø · April 15

Lauranae wrote: »

are you sure you did not shut down the NA too ?? server crashed at the same that you closed PTS

They have to close NA and roll back. People were exploiting the pts bug and some others literally lost their chars

licenturion · April 15

I constantly got a 'loading data error' when I rode 30 seconds with my horse before. So I thought there was something really wrong.

GlassHalfFull · April 15

Yes, last world boss in Murkmire, the easy one, is to go go, he is almost down, and suddenly ESO is down.

I know server or network trauma happens occasionally however, this is rather a toasty bummer for this achievement. This means the next world boss in Murkmire for me with be the Thunder, and hard to find help.

Ugh.

Dagoth_Rac · April 15

Hi ZOS. I just got all 5 of the Anniversary style pages after dinner. What a lucky streak. If you could just take my word for it and manually place those pages on my character after you finish rollback, that would be great. Thanks!

HeavyMetalHippie420 · April 15

Gorø wrote: »

Lauranae wrote: »

are you sure you did not shut down the NA too ?? server crashed at the same that you closed PTS

They have to close NA and roll back. People were exploiting the pts bug and some others literally lost their chars

I hope I didn't lose my character on PC NA.

Gorø · April 15

HeavyMetalHippie420 wrote: »

Gorø wrote: »

Lauranae wrote: »

are you sure you did not shut down the NA too ?? server crashed at the same that you closed PTS

They have to close NA and roll back. People were exploiting the pts bug and some others literally lost their chars

I hope I didn't lose my character on PC NA.

If you didnt delete that char on PTS, you're good

IncultaWolf · April 15

FabresFour wrote: »

There are people taking PTS gold and trying to pass it on to unsuspecting players. They just tried to do this to me, if I hadn't known about the bug I probably would have accepted it.

Wow that's crazy

bloodandiamonds · April 15

Starting to wonder if they can get anything right

virtus753 · April 15

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

eveboufa · April 15

Hey now, someone at zos is having a real bad day 🤣, and likely owes his team beers after work for the next while.

MoonPile · April 15

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

Dagoth_Rac · April 15

MoonPile wrote: »

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

They don't necessarily have to rollback to when live characters were copied to PTS. They have to rollback to right before the first PTS character was copied to Live. That was actually a pretty small window of time between PTS chars showing up on Live and taking down Live server. But it depends on their backup schedule. Do they take continual incremental backups and can roll back to any second in time? Or do they take one image per day at 5am and have to roll back to that? ¯\_(ツ)_/¯

MoonPile · April 15

Dagoth_Rac wrote: »

MoonPile wrote: »

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

They don't necessarily have to rollback to when live characters were copied to PTS. They have to rollback to right before the first PTS character was copied to Live. That was actually a pretty small window of time between PTS chars showing up on Live and taking down Live server. But it depends on their backup schedule. Do they take continual incremental backups and can roll back to any second in time? Or do they take one image per day at 5am and have to roll back to that? ¯\_(ツ)_/¯

I see. I don't even understand how this happens... in any online work I've done, testing environments are always totally separate and changes copied over essentially manually, in one sweep and with great care. You could not just randomly have things mirror on Live, like deleted characters. But Idk what their back-end looks like of course.

EDIT: *NVM I get it, in theory. PTS was still pointed at Live and not desynced, essentially. Though still Idk how that happens without multiple checkpoints, for such a major launch during a major event.

Truly, what the heck?

Also... if anyone from ZOS is reading, can we please have some crowns or crates or something better than XP scrolls for comp?

hamgatan · April 15

Dagoth_Rac wrote: »

MoonPile wrote: »

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

They don't necessarily have to rollback to when live characters were copied to PTS. They have to rollback to right before the first PTS character was copied to Live. That was actually a pretty small window of time between PTS chars showing up on Live and taking down Live server. But it depends on their backup schedule. Do they take continual incremental backups and can roll back to any second in time? Or do they take one image per day at 5am and have to roll back to that? ¯\_(ツ)_/¯

Nobody, is running incrementals on that kind of scale. I don't even think Azure DC's runs inc's that fast.. 10 min maybe at most.. but we're talking big data here so yeah.. I'd say at least a 15min RPO

licenturion · April 15

They have a very long workday lol

Reginald_leBlem · April 15

Christ on a bike what a mess

Dagoth_Rac · April 15

hamgatan wrote: »

Dagoth_Rac wrote: »

MoonPile wrote: »

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

They don't necessarily have to rollback to when live characters were copied to PTS. They have to rollback to right before the first PTS character was copied to Live. That was actually a pretty small window of time between PTS chars showing up on Live and taking down Live server. But it depends on their backup schedule. Do they take continual incremental backups and can roll back to any second in time? Or do they take one image per day at 5am and have to roll back to that? ¯\_(ツ)_/¯

Nobody, is running incrementals on that kind of scale. I don't even think Azure DC's runs inc's that fast.. 10 min maybe at most.. but we're talking big data here so yeah.. I'd say at least a 15min RPO

The equity trading system I work on at my job can rollback to any second of any day for the last 7 years. But it is not that we have 220 million backups to account for every second of the last 7 years. It is that every database transaction is logged. So you can recreate the database at any moment in time through a combo of daily backups (the starting point) and parsing the logs to reapply all transactions from end of backup to desired time. But whether ZOS have something like that available and whether it is feasible (they likely have a much heavier transaction load than we do)? I have no idea.

hamgatan · April 15

Dagoth_Rac wrote: »

hamgatan wrote: »

Dagoth_Rac wrote: »

MoonPile wrote: »

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

They don't necessarily have to rollback to when live characters were copied to PTS. They have to rollback to right before the first PTS character was copied to Live. That was actually a pretty small window of time between PTS chars showing up on Live and taking down Live server. But it depends on their backup schedule. Do they take continual incremental backups and can roll back to any second in time? Or do they take one image per day at 5am and have to roll back to that? ¯\_(ツ)_/¯

Nobody, is running incrementals on that kind of scale. I don't even think Azure DC's runs inc's that fast.. 10 min maybe at most.. but we're talking big data here so yeah.. I'd say at least a 15min RPO

The equity trading system I work on at my job can rollback to any second of any day for the last 7 years. But it is not that we have 220 million backups to account for every second of the last 7 years. It is that every database transaction is logged. So you can recreate the database at any moment in time through a combo of daily backups (the starting point) and parsing the logs to reapply all transactions from end of backup to desired time. But whether ZOS have something like that available and whether it is feasible (they likely have a much heavier transaction load than we do)? I have no idea.

the more likely scenario from a Chg Mgmt perspective is SAN snapshot prior to PTS Update, then a second prior to switching it on. That would be the ideal rollback scenario.

hamgatan · April 15

MoonPile wrote: »

Dagoth_Rac wrote: »

MoonPile wrote: »

virtus753 wrote: »

ganzaeso wrote: »

Does this mean I have just lost the last 8 hours of playtime on Live?

When did the PTS become available, and when was the last backup on live before that?

I was on PTS and got booted; had stuff in my mail there that was at least from this morning or possibly yesterday. I forget exactly, but hopefully that means "only" a day at most?

They don't necessarily have to rollback to when live characters were copied to PTS. They have to rollback to right before the first PTS character was copied to Live. That was actually a pretty small window of time between PTS chars showing up on Live and taking down Live server. But it depends on their backup schedule. Do they take continual incremental backups and can roll back to any second in time? Or do they take one image per day at 5am and have to roll back to that? ¯\_(ツ)_/¯

I see. I don't even understand how this happens... in any online work I've done, testing environments are always totally separate and changes copied over essentially manually, in one sweep and with great care. You could not just randomly have things mirror on Live, like deleted characters. But Idk what their back-end looks like of course.

EDIT: *NVM I get it, in theory. PTS was still pointed at Live and not desynced, essentially. Though still Idk how that happens without multiple checkpoints, for such a major launch during a major event.

Truly, what the heck?

Also... if anyone from ZOS is reading, can we please have some crowns or crates or something better than XP scrolls for comp?

id settle for better RNG on these stupid bound style pages..

ItsNotLiving · April 16

Hailan wrote: »

There going to have to roll the server back to a state before pts went live.

All those people who JUST pulled those styles pages, this one is gonna be a riot. Do they not know the fashion chasers are the most toxic players in the game????

MoonPile · April 16

https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

Grateful for the updates but what a bummer for everyone including/especially the devs – this kind of thing cannot lead to a good day or fun future meetings, and a bunch of you are definitely working late, if not all night. It's past midnight, EST.

It's a game, we'll be ok and I'm sure it'll all get sorted. But yeah I agree with everyone saying I hope the real people involved will be ok, if it is human error.

MoonPile · April 16

Update: in case anyone's wondering, here's what the email looks like, re: this latest update – (https://forums.elderscrollsonline.com/en/discussion/comment/8089804/#Comment_8089804)

We have temporarily locked this account due to an issue with the PTS affecting your Live PC NA account, and will be rolling back progress from this account to April 15th at 9:30am EDT. You can expect this to take 2-3 days to complete and we will alert you when your account is accessible again.

Guess how I know. I just wanted to see the new furniture, ffs.

Literally was walking around Zhan Khaj Crest before getting booted, not knowing what was happening. ...IMO that house would be a suitable comp for being locked out for DAYS

🙏

Dovahkiin02191973 · April 16

I certainly didn't lose any sleep over this as I took the attitude that it is just a game. Speculation, ranting and raving would have only driven up my Blood Pressure. Now that some of the dust has cleared I will say that locking out live accounts that were affected for 2 to 3 days is not the answer when in the end everyone is just going to get rolled back. Why not just go ahead and roll us back and be done with it. I resurrected an old account so I can at least play and I upgraded it to the Gold Road which means I paid for it again. A decision that was mine to make. As far as compensation for this goes not everyone will be happy with it should compensation be granted because you can't please the majority in this game. So we are all just sitting here spinning our wheels.

Moving on. Why is the PTS still offline with no word from the devs since about 2AM EST last night? Maybe it is because the issue still exists on the PTS and if so it needs to be fixed before you guys unlock it but a status update would be very nice right now even if it is not something we want to hear it would let us know where things stand.

Look, all we want in the end is for this to be made right. Not everyone will be content with the end results but I think we can all deal with a happy medium.