Maintenance for the week of December 2:
• PC/Mac: NA and EU megaservers for patch maintenance – December 2, 4:00AM EST (9:00 UTC) - 9:00AM EST (14:00 UTC)
• Xbox: NA and EU megaservers for patch maintenance – December 4, 6:00AM EST (11:00 UTC) - 12:00PM EST (17:00 UTC)
• PlayStation®: NA and EU megaservers for patch maintenance – December 4, 6:00AM EST (11:00 UTC) - 12:00PM EST (17:00 UTC)
• PC/Mac: NA and EU megaservers for patch maintenance – December 2, 4:00AM EST (9:00 UTC) - 9:00AM EST (14:00 UTC)
• Xbox: NA and EU megaservers for patch maintenance – December 4, 6:00AM EST (11:00 UTC) - 12:00PM EST (17:00 UTC)
• PlayStation®: NA and EU megaservers for patch maintenance – December 4, 6:00AM EST (11:00 UTC) - 12:00PM EST (17:00 UTC)
SERVER DOWN?
-
fireshine6623✭✭✭Really? No one has mentioned the hamsters?
Feed them!
So very true!!! I want to feed them cookies
We have been remiss on the hamster jokes.
For the people spreading the negativity about this issue, there are indeed other games. Feel free to try them... Complaining won't help the situation, it just spreads a bad vibe about a great game.2 -
Sakiri✭✭✭✭✭
✭✭JoeCapricorn wrote: »The entire Elder Scrolls series is a fever dream of a Soulshriven living in the post Planemeld world.
You have no idea how accurate this is.2 -
ArchangelIsraphel✭✭✭✭✭
✭✭✭✭
The PTS hamsters willingly embrace Bosmer culinary practices I see.
Legends never die
They're written down in eternity
But you'll never see the price it costs
The scars collected all their lives
When everything's lost, they pick up their hearts and avenge defeat
Before it all starts, they suffer through harm just to touch a dream
Oh, pick yourself up, 'cause
Legends never die1 -
ZOS_GinaBrunoCommunity ManagerHi all, no update as of yet. We'll continue to check back in periodically.6 -
Immortal_Dark410✭✭✭✭kyle.wilson wrote: »ragnarok6644b14_ESO wrote: »ragnarok6644b14_ESO wrote: »ragnarok6644b14_ESO wrote: »Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers
Players gaining things because of a server error - panic button
Typical.
I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.
The real question is:
WHERE IS ZOS' QUALITY AND ASSURANCE
This is becoming downright embarrassing on their end.
Right here, fixing the problem less than an hour after it was detected?
What, do you want them to minority-report fix things before they happen?
How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?
I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release
Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?
Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.
I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.
Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.
Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.
That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.
Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.
How do you test if a live server is attached to the wrong database before attaching it to the wrong database?
It's like asking to test if a screw is screwed in tightly before you screw it in...
And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.
by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.
Yup have to agree with this, they need to be separate from the live servers to ensure going forward this doesn't occur at all again. I feel bad for anyone who got gear or items during this time and they do a roll back. I was in doing my writs opening boxes and getting items when I got kicked, so I'm assuming everything I learned on my character will be rolled back, which I have no issue with if they do, but I think they need to make sure it doesn't happen going forward and the best thing to do is just separate the 2 servers, PTS and live.Retired Guild Leader Of AMAZING DEALS OF TAMRIEL
Retired Guild Leader Of AMAZING DEALS OF NIRN
(ALL FACTION TRADE GUILDS)
PC NA SERVER
DC - DARKDROGO | ORC | STAM DK | LVL 50 | CP 1285
AD - DARK-GEARLT-OF-RIVIA | ALTMER | MAG SORC | LVL 50 | CP 1285
EP - DARK-ABYSS | DUNMER | MAG TEMP | LVL 50 | CP 1285
DC - REAPERS-CLOACK-OF-YASSASEEN | BRETON | MAG NB | LVL 50 | CP 1285
AD - DARK-SLADE-WILSON | KHAJIIT | STAM NB | LVL 50 | CP 585
AD - THE-LAST-DRUID-OF-PARANOR | ALTMER | MAG DK | LVL 50 | CP 1285
AD - GALADRIEL-LADY OF LIGHT | IMPERIAL | STAM SORC | LVL 50 | CP 1285
CONSOLE PS5 NA SERVER AS WELL0 -
Sakiri✭✭✭✭✭
✭✭The databases are connected to a degree in order to facilitate character copies. I just wish it'd been EU's turn for the PTS this go around so THEY got kicked offline instead of me. -_-0 -
M33tr3✭✭✭ZOS_GinaBruno wrote: »Everyone, just popping in to let you know we are working on things and will update as soon as we can.
Its been 2 hours...ANY update?'We agree we haven't been communicating as much as we should on this matter (among others) and for that, we apologize. There really is no excuse.' ~ @ZOS_GinaBruno , nov 20180 -
kyle.wilson wrote: »ragnarok6644b14_ESO wrote: »ragnarok6644b14_ESO wrote: »ragnarok6644b14_ESO wrote: »Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers
Players gaining things because of a server error - panic button
Typical.
I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.
The real question is:
WHERE IS ZOS' QUALITY AND ASSURANCE
This is becoming downright embarrassing on their end.
Right here, fixing the problem less than an hour after it was detected?
What, do you want them to minority-report fix things before they happen?
How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?
I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release
Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?
Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.
I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.
Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.
Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.
That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.
Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.
How do you test if a live server is attached to the wrong database before attaching it to the wrong database?
It's like asking to test if a screw is screwed in tightly before you screw it in...
And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.
by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.
But it's a database, not a server.
If you have two independent circuits that never touch each other, but each have the same shaped data bus, the only thing preventing one person from plugging one circuit into the other circuit is... well, that person and their capacity to make errors*
*yes, the analogy is strained because you could put the physical circuits several meters apart... but it's software.
A better analogy is this:
You have a LAN with FILE SYSTEM A on it, and another LAN with FILE SYSTEM B on it. The only way to prevent similar-looking files from crossing the streams is to have them totally air-gapped....
... and yet, if the person working with the material is allowed to thumb-drive between the two systems, you will STILL have moments where some of the files on LAN A are identical to the files on LAN B by mistake. Almost exactly that scenario is why you can't plug thumb drives into most private organization's proprietary LANs. Because people literally make this mistake all the time. And once you ban plugging in a thumb drive, people find ever-more creative ways to make this kind of mistake.Edited by ragnarok6644b14_ESO on 16 April 2024 00:261 -
Herschel77✭✭ZOS_GinaBruno wrote: »Everyone, just popping in to let you know we are working on things and will update as soon as we can.
Its been 2 hours...ANY update?ZOS_GinaBruno wrote: »Hi all, no update as of yet. We'll continue to check back in periodically.0 -
furiouslog✭✭✭✭✭
✭✭I completely reorganized my inventory today, did about 200 master writs, spent gold, spent vouchers, attuned crafting stations and added them to newly purchased grandmaster stations. I was all set up, clean inventory, furniture placed, etc etc. It's all gonna get rolled back. This is a super bummer. Having to go do it all over again is going to make me angry, sad, and frustrated. If only we could roll back our lives like they roll back the server.
On the other hand, it's an exciting time to be in ESO, so there's that. 3 -
fireshine6623✭✭✭furiouslog wrote: »I completely reorganized my inventory today, did about 200 master writs, spent gold, spent vouchers, attuned crafting stations and added them to newly purchased grandmaster stations. I was all set up, clean inventory, furniture placed, etc etc. It's all gonna get rolled back. This is a super bummer. Having to go do it all over again is going to make me angry, sad, and frustrated. If only we could roll back our lives like they roll back the server.
On the other hand, it's an exciting time to be in ESO, so there's that.
It is an exciting time to be in eso!
And thank you @ZOS_GinaBruno for the update! We appreciate your efforts!1 -
kyle.wilson✭✭✭✭✭
✭ZOS_GinaBruno wrote: »Hi all, no update as of yet. We'll continue to check back in periodically.
Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin1 -
ziggy29✭✭✭I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.
That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production. That *totally* defeats the purpose of having a separate test environment before rolling out changes to production.Edited by ziggy29 on 16 April 2024 00:504 -
jcaceresw✭✭✭✭✭Someone wrote on another thread (now closed):
This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.
What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?
I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?
Not intending to speak ill against the company but something like that even happen on small busineses.Edited by jcaceresw on 16 April 2024 00:491 -
Calm_Fury✭✭✭✭✭HeavyMetalHippie420 wrote: »ZOS_GinaBruno wrote: »Everyone, just popping in to let you know we are working on things and will update as soon as we can.
NevermindEdited by Calm_Fury on 18 April 2024 14:121 -
I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.
That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.
Someone shipped the database intended to facilitate testing on said test server to the live server instead... a thing I've seen before.
You generate a database to feed new code the data...
.... and the next time you are using the stable code, you find out you accidentally plugged it into the database you built to test the new code with. The old code runs fine - after all, the code runs on the current database, so the current database plus a bit doesn't throw any errors or raise any flags immediately...
It's especially common in environments where people have to COPY the old database and just add to it for the testing thing - that kind of back-and-forth swapping of files opens the door to confusion.Edited by ragnarok6644b14_ESO on 16 April 2024 00:512 -
manwoodsal✭✭I think the problem come about when they was to be copying the accounts of PC-NA over to the PTS. SO the PC-NA can play on the PTS. But some how the database it was to be looking up wasn't the PTS account info.. But the Live Server Account info.1
-
jcaceresw✭✭✭✭✭I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.
That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.
Either they copied the web.config or messed up with the connection string.
connection_string=server=xxxx;database=pcna;userid=admin;password=admin
instead of:
connection_string=server=yyyy;database=pcna_pts;userid=admin;password=admin
where xxxx and yyyy are different servers and pcna_pts is a copy of the live one at a certain point5 -
Dagoth_Rac✭✭✭✭✭
✭✭✭✭✭I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.
That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.
There has to be some kind of connection between Live and PTS to allow for NA live characters to be available on PTS. My guess is they temporarily turn on connection to allow NA chars to get copied to PTS and then are supposed to turn off connection before bringing up PTS. And someone, or some process, failed to break the connection. Oops!
3 -
Yo this is taking a pretty long time, [snip]
[edited for petition/protest]Edited by ZOS_Icy on 16 April 2024 11:561 -
I think the only appropriate response is to have someone better at photoshop than me put Gina's (? probably whomever is trying to fix this right now) face on this gif:
3 -
ZOS_GinaBrunoCommunity Managerkyle.wilson wrote: »ZOS_GinaBruno wrote: »Hi all, no update as of yet. We'll continue to check back in periodically.
Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin
Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest2 -
Ariordin✭✭✭✭I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?0 -
Calm_Fury✭✭✭✭✭I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.
That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production. That *totally* defeats the purpose of having a separate test environment before rolling out changes to production.
You probably know how it is.
Someone probably either forgot to update DB_HOST and the other database variables to point to the test one or copied the wrong ones in the PTS build.
The best practices for cloud today all recommend having completely different accounts for each environment for that specific reason, but as far as we know ZOS have their own servers (by the announcements of hardware upgrades).
The PTS machines and infrastructure are probably on the same network as the production one, so a simple mistake like pointing to the wrong DB host causes this.
Still very surprising it could happen with a company this size and that has existed for this long.
6 -
TKo_ROUSE✭✭✭✭Someone wrote on another thread (now closed):
This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.
What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?
I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?
Not intending to speak ill against the company but something like that even happen on small busineses.
To be fair I wouldn’t worry too much about that. If the backups were all bad up until a year ago. They would turn the server back on and just commit to manually making corrections. No way they would restore a year old backup. The data is still usable as you could see we were playing. It’s not like it got hit by ransomware and has no backups.
Source: disaster recovery planning is part of my job.3 -
purple-magicb16_ESO✭✭✭✭✭Glorious_Platypus wrote: »Yo this is taking a pretty long time, [snip]
yep, I'm starting to get the sense that the issue is not as simple as people on this forum are saying
[edited to remove quote]Edited by ZOS_Icy on 16 April 2024 11:57I don't comment here often but when I do, I get [snip]3 -
2
-
kyle.wilson✭✭✭✭✭
✭ZOS_GinaBruno wrote: »kyle.wilson wrote: »ZOS_GinaBruno wrote: »Hi all, no update as of yet. We'll continue to check back in periodically.
Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin
Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest
Thanks0 -
purple-magicb16_ESO✭✭✭✭✭kyle.wilson wrote: »ZOS_GinaBruno wrote: »kyle.wilson wrote: »ZOS_GinaBruno wrote: »Hi all, no update as of yet. We'll continue to check back in periodically.
Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin
Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest
Thanks
yeah, I thought so. Not an issue that can be solved with replacing a config file or 4 lines of code. Have a good nite guys! I don't comment here often but when I do, I get [snip]0
