SERVER DOWN?

BetweenMidgets · April 15

If some people DONT get banned for this, its going to be such a bad look for ZOS.

There is no way to NOT KNOW that a PTS character, with all the CP and gold on the LIVE server isn't a bug. If they don't come down hard on those bad actors, it gives them an even worse look than they already have.

This update has been a real [snip]

[edited for profanity bypass]

Sakiri · April 15

Bluepetunia wrote: »

No need to panic, relax, get up out of your chair........

Take a shower
Eat something besides microwaved food
Go brush your teeth
Walk the dog
Spend time with spouse, kids, Mom, Dad (at least let them know you are still among the living)

Maybe put on some sunglasses and venture outside. I promise no dragons, werewolves, harpy's (well maybe the wife........j/k), vampires will get you'

Dude.

It's almost 2 am. I'm not doing any of that, since I've either done it already, don't need to, or can't.

And my bff's due on in two hours and we were gonna farm me some tank gear tonight.

KKolly · April 15

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

BetweenMidgets · April 15

KKolly wrote: »

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

Not in the budget.

Darquer · April 15

Help. Help, I need ESO now! LOL, Seriously though, why? What might I lose? I've been doing this for 10 years and now 14 characters, so it gets messy,

Loralai_907 · April 15

Well....I can't say I've seen this be an issue before....

ragnarok6644b14_ESO · April 15

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

baratron · April 15

kyle.wilson wrote: »

I'm curious as to why the pts server going live affected the live server.
They should be completely separate with no intermingling.

"Should" is the word, and indeed explains what the issue is.

I have no idea how this could have happened and I feel very bad for the person who made the mistake, as well as everyone who has to try to clean it up.

ragnarok6644b14_ESO · April 15

It's a database dickup, it happens all the time - that's why backups exist in the first place. Databases are almost never flawless.

Now how specifically this database dickup happened is not something I can describe, but as someone who works for large entities with lots of data, it's distressingly common how often someone explodes some critical database or something.

I got locked out of my own office by the electronic keypad because two separate professionals exploded two separate databases (that they were used to working with every day) with bad inputs within 15 seconds of each other. It's just humans and data failing to co-exist.

KKolly · April 15

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

shadyjane62 · April 15

Just tried playing on EU.

Not worth my time.

virtus753 · April 15

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

A PTS for the PTS!

PTSception.

ragnarok6644b14_ESO · April 15

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

Jimbru · April 15

I remember many years ago in the early days of EverQuest 1, there was a bug after a patch that glitched loot tables throughout the game. Suddenly every mob in the game, all the way down to the level 1 rats in the starting areas, began dropping level 50 end game boss loot, and of course people went on a feeding frenzy. When Sony realized what was happening, they literally pulled the plugs on the servers and instantly downed the game, and it stayed down for a long time while they fixed the bug and rolled back everything to get rid of all the ill-gotten loot. This error in ESO tonight strikes me as falling into the same category of "Big Trouble Needs Big Fix" in more ways than one, and I wouldn't bet on us getting to play any more in US PM time tonight. So it goes, I guess.

fireshine6623 · April 15

kyle.wilson wrote: »

I'm curious as to why the pts server going live affected the live server.
They should be completely separate with no intermingling.

This must be the doing of Morlag Bal, trying to pull the worlds together lol. Either way I'm rooting for all the staff, and @ZOS_GinaBruno thank you for keeping us posted.

I recommended eso to a few potential new players, and I got a text over this past weekend that they got the base game and LOVED it. It truly is an excellent game, but things happen. I'm not disturbed by it in the least, I have every confidence in our beloved ZOS guardian angels of the servers. After all, they preserve the world we love to play in.

Immortal_Dark410 · April 16

Dam R.I.P NA

SilverBride wrote: »

Bluepetunia wrote: »

No need to panic, relax, get up out of your chair........

Take a shower
Eat something besides microwaved food
Go brush your teeth
Walk the dog
Spend time with spouse, kids, Mom, Dad (at least let them know you are still among the living)

Maybe put on some sunglasses and venture outside. I promise no dragons, werewolves, harpy's (well maybe the wife........j/k), vampires will get you'

That is assuming that we spend every second of our day at our computers and never interact with other people, or take care of our hygiene, or get fresh air, or eat healthy. That is not a fair assessment.

ClowdyAllDay · April 16

This one wonders how many people will be perma banned from this screw up that they caused.

Oh hey look a template for gold gear woo! let me play with that on pc na so i can get perma banned!

shadyjane62 · April 16

Why was the PTS even up now? During the so-called event of the year? They should have been fixing the abysmal drop rate on the style pages instead.

HeavyMetalHippie420 · April 16

BetweenMidgets wrote: »

KKolly wrote: »

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

Not in the budget.

But they can afford to travel the world and host large events.

Glorious_Platypus · April 16

Y'all got any more of that (scratches neck)... ESO?

ragnarok6644b14_ESO · April 16

shadyjane62 wrote: »

Why was the PTS even up now? During the so-called event of the year? They should have been fixing the abysmal drop rate on the style pages instead.

I agree, they should fix something that isn't broken instead of testing the next major update to the game for bugs or issues. I mean really.

(/s)

Alaztor91 · April 16

fireshine6623 wrote: »

This must be the doing of Morlag Bal, trying to pull the worlds together lol.

It took a decade, but the Planemeld is now complete. PTSharbour and Liveriel are now one.

KKolly · April 16

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

JoeCapricorn · April 16

The entire Elder Scrolls series is a fever dream of a Soulshriven living in the post Planemeld world.

dcrush · April 16

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Obviously this is a bigger bug, but it would be nice if ZOS afforded a bit more urgency to issues like the Cyrodiil rollbacks to at least give the impression of caring about their customers. There is too big of a difference between the approach to both issues.

Path · April 16

Really? No one has mentioned the hamsters?

Feed them!

HeavyMetalHippie420 · April 16

ZOS_GinaBruno wrote: »

Everyone, just popping in to let you know we are working on things and will update as soon as we can.

Is there an estimate on how long this will take? Also, will the rollbacks impact players who don't use pts?

virtus753 · April 16

Path wrote: »

Really? No one has mentioned the hamsters?

Feed them!

They fed the hamsters, all right. They gave the PTS hamsters Live food.

ragnarok6644b14_ESO · April 16

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

How do you test if a live server is attached to the wrong database before attaching it to the wrong database?

It's like asking to test if a screw is screwed in tightly before you screw it in...

And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.

kyle.wilson · April 16

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

How do you test if a live server is attached to the wrong database before attaching it to the wrong database?

It's like asking to test if a screw is screwed in tightly before you screw it in...

And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.

by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.