Maintenance for the week of November 11:
• PC/Mac: NA and EU megaservers for patch maintenance – November 11, 4:00AM EST (9:00 UTC) - 9:00AM EST (14:00 UTC)
• Xbox: NA and EU megaservers for patch maintenance – November 13, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
• PlayStation®: NA and EU megaservers for patch maintenance – November 13, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)

SERVER DOWN?

  • fireshine6623
    fireshine6623
    ✭✭✭
    Path wrote: »
    Really? No one has mentioned the hamsters?

    Feed them!

    So very true!!! I want to feed them cookies :smile: We have been remiss on the hamster jokes.

    For the people spreading the negativity about this issue, there are indeed other games. Feel free to try them... Complaining won't help the situation, it just spreads a bad vibe about a great game.
  • Sakiri
    Sakiri
    ✭✭✭✭✭
    ✭✭
    The entire Elder Scrolls series is a fever dream of a Soulshriven living in the post Planemeld world.

    You have no idea how accurate this is.
  • ArchangelIsraphel
    ArchangelIsraphel
    ✭✭✭✭✭
    ✭✭✭✭
    virtus753 wrote: »
    Path wrote: »
    Really? No one has mentioned the hamsters?

    Feed them!

    They fed the hamsters, all right. They gave the PTS hamsters Live food.

    The PTS hamsters willingly embrace Bosmer culinary practices I see.
    Legends never die
    They're written down in eternity
    But you'll never see the price it costs
    The scars collected all their lives
    When everything's lost, they pick up their hearts and avenge defeat
    Before it all starts, they suffer through harm just to touch a dream
    Oh, pick yourself up, 'cause
    Legends never die
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi all, no update as of yet. We'll continue to check back in periodically.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • Immortal_Dark410
    Immortal_Dark410
    ✭✭✭✭
    KKolly wrote: »
    KKolly wrote: »
    KKolly wrote: »
    dcrush wrote: »
    Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

    Players gaining things because of a server error - panic button

    Typical.

    I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

    The real question is:

    WHERE IS ZOS' QUALITY AND ASSURANCE

    This is becoming downright embarrassing on their end.

    Right here, fixing the problem less than an hour after it was detected?

    What, do you want them to minority-report fix things before they happen?

    How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

    I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release :)

    Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

    Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

    I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

    Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

    Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

    That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

    Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

    How do you test if a live server is attached to the wrong database before attaching it to the wrong database?

    It's like asking to test if a screw is screwed in tightly before you screw it in...

    And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.

    by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.

    Yup have to agree with this, they need to be separate from the live servers to ensure going forward this doesn't occur at all again. I feel bad for anyone who got gear or items during this time and they do a roll back. I was in doing my writs opening boxes and getting items when I got kicked, so I'm assuming everything I learned on my character will be rolled back, which I have no issue with if they do, but I think they need to make sure it doesn't happen going forward and the best thing to do is just separate the 2 servers, PTS and live.
    Retired Guild Leader Of AMAZING DEALS OF TAMRIEL
    Retired Guild Leader Of AMAZING DEALS OF NIRN
    (ALL FACTION TRADE GUILDS)

    PC NA SERVER
    DC - DARKDROGO | ORC | STAM DK | LVL 50 | CP 1285
    AD - DARK-GEARLT-OF-RIVIA | ALTMER | MAG SORC | LVL 50 | CP 1285
    EP - DARK-ABYSS | DUNMER | MAG TEMP | LVL 50 | CP 1285
    DC - REAPERS-CLOACK-OF-YASSASEEN | BRETON | MAG NB | LVL 50 | CP 1285
    AD - DARK-SLADE-WILSON | KHAJIIT | STAM NB | LVL 50 | CP 585
    AD - THE-LAST-DRUID-OF-PARANOR | ALTMER | MAG DK | LVL 50 | CP 1285
    AD - GALADRIEL-LADY OF LIGHT | IMPERIAL | STAM SORC | LVL 50 | CP 1285

    CONSOLE PS5 NA SERVER AS WELL
  • Sakiri
    Sakiri
    ✭✭✭✭✭
    ✭✭
    The databases are connected to a degree in order to facilitate character copies. I just wish it'd been EU's turn for the PTS this go around so THEY got kicked offline instead of me. -_-
  • M33tr3
    M33tr3
    ✭✭✭
    Everyone, just popping in to let you know we are working on things and will update as soon as we can.

    Its been 2 hours...ANY update?
    'We agree we haven't been communicating as much as we should on this matter (among others) and for that, we apologize. There really is no excuse.' ~ @ZOS_GinaBruno , nov 2018
  • ragnarok6644b14_ESO
    KKolly wrote: »
    KKolly wrote: »
    KKolly wrote: »
    dcrush wrote: »
    Rollbacks in Cyrodiil and players losing items - 2 business days to shut down the servers

    Players gaining things because of a server error - panic button

    Typical.

    I exclusively PvP and this is a **MUCH** bigger issue than the rollback bug. In this case servers needed to be taken down ASAP to avoid further exploit.

    The real question is:

    WHERE IS ZOS' QUALITY AND ASSURANCE

    This is becoming downright embarrassing on their end.

    Right here, fixing the problem less than an hour after it was detected?

    What, do you want them to minority-report fix things before they happen?

    How exactly would you catch this error in advance, considering the error was caused by doing the thing that permits testing for issues in advance in the first place?

    I would like the testing to remain on the testing server. I don't pay a monthly subscription to test their mistakes on a stable release :)

    Seriously though, this isn't a bug in code that slipped into a stable release. This was catastrophic databasing error. Its unacceptable. How can I trust them with my private information if this is how they handle their databases?

    Presumably the people that make the PTS go and the people handling your private information have different standards for productivity and safety; I doubt they are trying to move and shake your private information as much as they are the character databases for the server.

    I will say, though, that if you believe no one has ever made a similar database error in some other database this very week, boy have I got a deal on a bridge in Brooklyn for you.

    Hopelessly fragging your database at some point is like a coming-of-age moment for every database manager.

    Databases get ***. Making them interact *** them even more. Its in their nature. But this is why you test something before pushing it out. Its clear they did not.

    That speaks to bad structural design in how they do their work, something that is only made more obvious by the nearly constant slew of gameplay altering bugs. They have no Q&A it seems.

    Dating to last monday, we've had to cancel 3 of our last 4 raids because the game simply wasn't playable. Thats not considering the week before where PvP was broken with tarnished nightmare inheriting other skills modifiers (like proxy). That was unfortunate..but acceptable. It happens. But what happened today should not.

    How do you test if a live server is attached to the wrong database before attaching it to the wrong database?

    It's like asking to test if a screw is screwed in tightly before you screw it in...

    And that's interesting, my groups have been doing their scheduled raids just fine; the only cancellations in the last month or so has been because folks were unavailable.

    by ensuring that none of the test servers have any connection to the live servers. The PTS servers should be 100% segregated from the other servers.

    But it's a database, not a server.

    If you have two independent circuits that never touch each other, but each have the same shaped data bus, the only thing preventing one person from plugging one circuit into the other circuit is... well, that person and their capacity to make errors*

    *yes, the analogy is strained because you could put the physical circuits several meters apart... but it's software.

    A better analogy is this:
    You have a LAN with FILE SYSTEM A on it, and another LAN with FILE SYSTEM B on it. The only way to prevent similar-looking files from crossing the streams is to have them totally air-gapped....

    ... and yet, if the person working with the material is allowed to thumb-drive between the two systems, you will STILL have moments where some of the files on LAN A are identical to the files on LAN B by mistake. Almost exactly that scenario is why you can't plug thumb drives into most private organization's proprietary LANs. Because people literally make this mistake all the time. And once you ban plugging in a thumb drive, people find ever-more creative ways to make this kind of mistake.
    Edited by ragnarok6644b14_ESO on April 16, 2024 12:26AM
  • Herschel77
    Herschel77
    ✭✭
    M33tr3 wrote: »
    Everyone, just popping in to let you know we are working on things and will update as soon as we can.

    Its been 2 hours...ANY update?
    Hi all, no update as of yet. We'll continue to check back in periodically.
  • furiouslog
    furiouslog
    ✭✭✭✭✭
    ✭✭
    I completely reorganized my inventory today, did about 200 master writs, spent gold, spent vouchers, attuned crafting stations and added them to newly purchased grandmaster stations. I was all set up, clean inventory, furniture placed, etc etc. It's all gonna get rolled back. This is a super bummer. Having to go do it all over again is going to make me angry, sad, and frustrated. If only we could roll back our lives like they roll back the server.

    On the other hand, it's an exciting time to be in ESO, so there's that. :smile:
  • fireshine6623
    fireshine6623
    ✭✭✭
    furiouslog wrote: »
    I completely reorganized my inventory today, did about 200 master writs, spent gold, spent vouchers, attuned crafting stations and added them to newly purchased grandmaster stations. I was all set up, clean inventory, furniture placed, etc etc. It's all gonna get rolled back. This is a super bummer. Having to go do it all over again is going to make me angry, sad, and frustrated. If only we could roll back our lives like they roll back the server.

    On the other hand, it's an exciting time to be in ESO, so there's that. :smile:

    It is an exciting time to be in eso!

    And thank you @ZOS_GinaBruno for the update! We appreciate your efforts!
  • kyle.wilson
    kyle.wilson
    ✭✭✭✭✭
    Hi all, no update as of yet. We'll continue to check back in periodically.

    Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
    @ZOS_Kevin
  • ziggy29
    ziggy29
    ✭✭✭
    I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

    That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production. That *totally* defeats the purpose of having a separate test environment before rolling out changes to production.
    Edited by ziggy29 on April 16, 2024 12:50AM
  • jcaceresw
    jcaceresw
    ✭✭✭✭✭
    Someone wrote on another thread (now closed):

    This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.

    What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?

    I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?

    Not intending to speak ill against the company but something like that even happen on small busineses.
    Edited by jcaceresw on April 16, 2024 12:49AM
  • Calm_Fury
    Calm_Fury
    ✭✭✭✭✭
    Everyone, just popping in to let you know we are working on things and will update as soon as we can.

    Nevermind
    Edited by Calm_Fury on April 18, 2024 2:12PM
  • ragnarok6644b14_ESO
    ziggy29 wrote: »
    I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

    That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.

    Someone shipped the database intended to facilitate testing on said test server to the live server instead... a thing I've seen before.

    You generate a database to feed new code the data...

    .... and the next time you are using the stable code, you find out you accidentally plugged it into the database you built to test the new code with. The old code runs fine - after all, the code runs on the current database, so the current database plus a bit doesn't throw any errors or raise any flags immediately...

    It's especially common in environments where people have to COPY the old database and just add to it for the testing thing - that kind of back-and-forth swapping of files opens the door to confusion.
    Edited by ragnarok6644b14_ESO on April 16, 2024 12:51AM
  • manwoodsal
    manwoodsal
    ✭✭
    I think the problem come about when they was to be copying the accounts of PC-NA over to the PTS. SO the PC-NA can play on the PTS. But some how the database it was to be looking up wasn't the PTS account info.. But the Live Server Account info.
  • jcaceresw
    jcaceresw
    ✭✭✭✭✭
    ziggy29 wrote: »
    I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

    That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.

    Either they copied the web.config or messed up with the connection string.

    connection_string=server=xxxx;database=pcna;userid=admin;password=admin

    instead of:

    connection_string=server=yyyy;database=pcna_pts;userid=admin;password=admin

    where xxxx and yyyy are different servers and pcna_pts is a copy of the live one at a certain point
  • Dagoth_Rac
    Dagoth_Rac
    ✭✭✭✭✭
    ✭✭✭✭✭
    ziggy29 wrote: »
    I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

    That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production.

    There has to be some kind of connection between Live and PTS to allow for NA live characters to be available on PTS. My guess is they temporarily turn on connection to allow NA chars to get copied to PTS and then are supposed to turn off connection before bringing up PTS. And someone, or some process, failed to break the connection. Oops! ;)
  • Glorious_Platypus
    Yo this is taking a pretty long time, [snip]

    [edited for petition/protest]
    Edited by ZOS_Icy on April 16, 2024 11:56AM
  • ragnarok6644b14_ESO
    I think the only appropriate response is to have someone better at photoshop than me put Gina's (? probably whomever is trying to fix this right now) face on this gif:
    x37pm54y1efm.gif
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi all, no update as of yet. We'll continue to check back in periodically.

    Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
    @ZOS_Kevin

    Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • Ariordin
    Ariordin
    ✭✭✭✭
    I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?
  • Calm_Fury
    Calm_Fury
    ✭✭✭✭✭
    ziggy29 wrote: »
    I'm trying to figure this one out. I'm a retired programmer, with over 20 years in the field. What I am struggling to understand is how a change made to the TEST environment impacted production. It seems to go against the reason for having a separate test system in the first place.

    That said, I'm not going to sit here and bash this. But I am curious how a change to the PTS -- a test server -- caused problems in production. That *totally* defeats the purpose of having a separate test environment before rolling out changes to production.

    You probably know how it is.

    Someone probably either forgot to update DB_HOST and the other database variables to point to the test one or copied the wrong ones in the PTS build.

    The best practices for cloud today all recommend having completely different accounts for each environment for that specific reason, but as far as we know ZOS have their own servers (by the announcements of hardware upgrades).

    The PTS machines and infrastructure are probably on the same network as the production one, so a simple mistake like pointing to the wrong DB host causes this.

    Still very surprising it could happen with a company this size and that has existed for this long.
  • TKo_ROUSE
    TKo_ROUSE
    ✭✭✭✭
    jcaceresw wrote: »
    Someone wrote on another thread (now closed):

    This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.

    What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?

    I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?

    Not intending to speak ill against the company but something like that even happen on small busineses.

    To be fair I wouldn’t worry too much about that. If the backups were all bad up until a year ago. They would turn the server back on and just commit to manually making corrections. No way they would restore a year old backup. The data is still usable as you could see we were playing. It’s not like it got hit by ransomware and has no backups.

    Source: disaster recovery planning is part of my job.
  • purple-magicb16_ESO
    purple-magicb16_ESO
    ✭✭✭✭✭
    Yo this is taking a pretty long time, [snip]

    yep, I'm starting to get the sense that the issue is not as simple as people on this forum are saying

    [edited to remove quote]
    Edited by ZOS_Icy on April 16, 2024 11:57AM
    I don't comment here often but when I do, I get [snip]
  • dcam86b14_ESO
    dcam86b14_ESO
    ✭✭✭✭✭
    hopefully we can expect an extension of both events bc of this delay
  • ragnarok6644b14_ESO
    Ariordin wrote: »
    I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?

    Finding new mistakes is far, far better than repeating old mistakes.

    It means they're capable of learning.... :open_mouth:
  • kyle.wilson
    kyle.wilson
    ✭✭✭✭✭
    Hi all, no update as of yet. We'll continue to check back in periodically.

    Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
    @ZOS_Kevin

    Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

    Thanks
  • purple-magicb16_ESO
    purple-magicb16_ESO
    ✭✭✭✭✭
    Hi all, no update as of yet. We'll continue to check back in periodically.

    Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
    @ZOS_Kevin

    Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

    Thanks

    yeah, I thought so. Not an issue that can be solved with replacing a config file or 4 lines of code. Have a good nite guys! :)
    I don't comment here often but when I do, I get [snip]
Sign In or Register to comment.