SERVER DOWN?

fireshine6623 · April 16

Path wrote: »

Really? No one has mentioned the hamsters?

Feed them!

So very true!!! I want to feed them cookies

We have been remiss on the hamster jokes.

For the people spreading the negativity about this issue, there are indeed other games. Feel free to try them... Complaining won't help the situation, it just spreads a bad vibe about a great game.

Sakiri · April 16

JoeCapricorn wrote: »

The entire Elder Scrolls series is a fever dream of a Soulshriven living in the post Planemeld world.

You have no idea how accurate this is.

ArchangelIsraphel · April 16

virtus753 wrote: »

Path wrote: »

Really? No one has mentioned the hamsters?

Feed them!

They fed the hamsters, all right. They gave the PTS hamsters Live food.

The PTS hamsters willingly embrace Bosmer culinary practices I see.

ZOS_GinaBruno · April 16

Hi all, no update as of yet. We'll continue to check back in periodically.

Immortal_Dark410 · April 16

kyle.wilson wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

How do you test if a live server is attached to the wrong database before attaching it to the wrong database?

It's like asking to test if a screw is screwed in tightly before you screw it in...

And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.

by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.

Yup have to agree with this, they need to be separate from the live servers to ensure going forward this doesn't occur at all again. I feel bad for anyone who got gear or items during this time and they do a roll back. I was in doing my writs opening boxes and getting items when I got kicked, so I'm assuming everything I learned on my character will be rolled back, which I have no issue with if they do, but I think they need to make sure it doesn't happen going forward and the best thing to do is just separate the 2 servers, PTS and live.

Sakiri · April 16

The databases are connected to a degree in order to facilitate character copies. I just wish it'd been EU's turn for the PTS this go around so THEY got kicked offline instead of me. -_-

M33tr3 · April 16

ZOS_GinaBruno wrote: »

Everyone, just popping in to let you know we are working on things and will update as soon as we can.

Its been 2 hours...ANY update?

ragnarok6644b14_ESO · April 16

kyle.wilson wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

ragnarok6644b14_ESO wrote: »

KKolly wrote: »

dcrush wrote: »

Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

Players gaining things because of a server error - panic button

Typical.

I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

The real question is:

WHERE IS ZOS' QUALITY AND ASSURANCE

This is becoming downright embarrassing on their end.

Right here, fixing the problem less than an hour after it was detected?

What, do you want them to minority-report fix things before they happen?

How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release

Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

How do you test if a live server is attached to the wrong database before attaching it to the wrong database?

It's like asking to test if a screw is screwed in tightly before you screw it in...

And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.

by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.

But it's a database, not a server.

If you have two independent circuits that never touch each other, but each have the same shaped data bus, the only thing preventing one person from plugging one circuit into the other circuit is... well, that person and their capacity to make errors*

*yes, the analogy is strained because you could put the physical circuits several meters apart... but it's software.

A better analogy is this:
You have a LAN with FILE SYSTEM A on it, and another LAN with FILE SYSTEM B on it. The only way to prevent similar-looking files from crossing the streams is to have them totally air-gapped....

... and yet, if the person working with the material is allowed to thumb-drive between the two systems, you will STILL have moments where some of the files on LAN A are identical to the files on LAN B by mistake. Almost exactly that scenario is why you can't plug thumb drives into most private organization's proprietary LANs. Because people literally make this mistake all the time. And once you ban plugging in a thumb drive, people find ever-more creative ways to make this kind of mistake.

Herschel77 · April 16

M33tr3 wrote: »

ZOS_GinaBruno wrote: »

Everyone, just popping in to let you know we are working on things and will update as soon as we can.

Its been 2 hours...ANY update?

ZOS_GinaBruno wrote: »

Hi all, no update as of yet. We'll continue to check back in periodically.

furiouslog · April 16

I completely reorganized my inventory today, did about 200 master writs, spent gold, spent vouchers, attuned crafting stations and added them to newly purchased grandmaster stations. I was all set up, clean inventory, furniture placed, etc etc. It's all gonna get rolled back. This is a super bummer. Having to go do it all over again is going to make me angry, sad, and frustrated. If only we could roll back our lives like they roll back the server.

On the other hand, it's an exciting time to be in ESO, so there's that.

fireshine6623 · April 16

furiouslog wrote: »

I completely reorganized my inventory today, did about 200 master writs, spent gold, spent vouchers, attuned crafting stations and added them to newly purchased grandmaster stations. I was all set up, clean inventory, furniture placed, etc etc. It's all gonna get rolled back. This is a super bummer. Having to go do it all over again is going to make me angry, sad, and frustrated. If only we could roll back our lives like they roll back the server.

On the other hand, it's an exciting time to be in ESO, so there's that.

It is an exciting time to be in eso!

And thank you @ZOS_GinaBruno for the update! We appreciate your efforts!

kyle.wilson · April 16

ZOS_GinaBruno wrote: »

Hi all, no update as of yet. We'll continue to check back in periodically.

Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin

ziggy29 · April 16

I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production. That *totally* defeats the purpose of having a separate test environment before rolling out changes to production.

jcaceresw · April 16

Someone wrote on another thread (now closed):

This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.

What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?

I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?

Not intending to speak ill against the company but something like that even happen on small busineses.

Calm_Fury · April 16

HeavyMetalHippie420 wrote: »

ZOS_GinaBruno wrote: »

Everyone, just popping in to let you know we are working on things and will update as soon as we can.

Nevermind

ragnarok6644b14_ESO · April 16

ziggy29 wrote: »

I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.

Someone shipped the database intended to facilitate testing on said test server to the live server instead... a thing I've seen before.

You generate a database to feed new code the data...

.... and the next time you are using the stable code, you find out you accidentally plugged it into the database you built to test the new code with. The old code runs fine - after all, the code runs on the current database, so the current database plus a bit doesn't throw any errors or raise any flags immediately...

It's especially common in environments where people have to COPY the old database and just add to it for the testing thing - that kind of back-and-forth swapping of files opens the door to confusion.

manwoodsal · April 16

I think the problem come about when they was to be copying the accounts of PC-NA over to the PTS. SO the PC-NA can play on the PTS. But some how the database it was to be looking up wasn't the PTS account info.. But the Live Server Account info.

jcaceresw · April 16

ziggy29 wrote: »

I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.

Either they copied the web.config or messed up with the connection string.

connection_string=server=xxxx;database=pcna;userid=admin;password=admin

instead of:

connection_string=server=yyyy;database=pcna_pts;userid=admin;password=admin

where xxxx and yyyy are different servers and pcna_pts is a copy of the live one at a certain point

Dagoth_Rac · April 16

ziggy29 wrote: »

I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.

There has to be some kind of connection between Live and PTS to allow for NA live characters to be available on PTS. My guess is they temporarily turn on connection to allow NA chars to get copied to PTS and then are supposed to turn off connection before bringing up PTS. And someone, or some process, failed to break the connection. Oops!

Glorious_Platypus · April 16

Yo this is taking a pretty long time, [snip]

[edited for petition/protest]

ragnarok6644b14_ESO · April 16

I think the only appropriate response is to have someone better at photoshop than me put Gina's (? probably whomever is trying to fix this right now) face on this gif:

ZOS_GinaBruno · April 16

kyle.wilson wrote: »

ZOS_GinaBruno wrote: »

Hi all, no update as of yet. We'll continue to check back in periodically.

Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin

Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

Ariordin · April 16

I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?

Calm_Fury · April 16

ziggy29 wrote: »

I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production. That *totally* defeats the purpose of having a separate test environment before rolling out changes to production.

You probably know how it is.

Someone probably either forgot to update DB_HOST and the other database variables to point to the test one or copied the wrong ones in the PTS build.

The best practices for cloud today all recommend having completely different accounts for each environment for that specific reason, but as far as we know ZOS have their own servers (by the announcements of hardware upgrades).

The PTS machines and infrastructure are probably on the same network as the production one, so a simple mistake like pointing to the wrong DB host causes this.

Still very surprising it could happen with a company this size and that has existed for this long.

TKo_ROUSE · April 16

jcaceresw wrote: »

Someone wrote on another thread (now closed):

This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.

What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?

I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?

Not intending to speak ill against the company but something like that even happen on small busineses.

To be fair I wouldn’t worry too much about that. If the backups were all bad up until a year ago. They would turn the server back on and just commit to manually making corrections. No way they would restore a year old backup. The data is still usable as you could see we were playing. It’s not like it got hit by ransomware and has no backups.

Source: disaster recovery planning is part of my job.

purple-magicb16_ESO · April 16

Glorious_Platypus wrote: »

Yo this is taking a pretty long time, [snip]

yep, I'm starting to get the sense that the issue is not as simple as people on this forum are saying

[edited to remove quote]

dcam86b14_ESO · April 16

hopefully we can expect an extension of both events bc of this delay

ragnarok6644b14_ESO · April 16

Ariordin wrote: »

I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?

Finding new mistakes is far, far better than repeating old mistakes.

It means they're capable of learning....

kyle.wilson · April 16

ZOS_GinaBruno wrote: »

kyle.wilson wrote: »

ZOS_GinaBruno wrote: »

Hi all, no update as of yet. We'll continue to check back in periodically.

Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin

Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

Thanks

purple-magicb16_ESO · April 16

kyle.wilson wrote: »

ZOS_GinaBruno wrote: »

kyle.wilson wrote: »

ZOS_GinaBruno wrote: »

Hi all, no update as of yet. We'll continue to check back in periodically.

Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin

Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

Thanks

yeah, I thought so. Not an issue that can be solved with replacing a config file or 4 lines of code. Have a good nite guys!