Temporarily taking down the PTS & the PC NA Live Server

ZOS_Kevin
ZOS_Kevin
Community Manager
Hi all, we currently have taken down access to PTS. We encountered an issue with PTS once the server went live. This has also caused an issue with the PC NA live server, so we will be temporarily taking down the PC NA live server as well. Once we have more information, we will update everyone. Thanks for your patience as we begin investigations.
Edited by ZOS_Kevin on May 17, 2024 12:33PM
Community Manager for ZeniMax Online Studio and Elder Scrolls OnlineDev Tracker | Service Alerts | ESO Twitter
Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi everyone, just wanted to let you know investigations are ongoing. To set expectations as we head into the evening, we're looking at several hours of downtime. We'll let you know when we have an update to share.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Just checking in to let everyone know investigations are continuing, and both PC NA and PTS will remain under maintenance until further notice. We'll provide another update in an hour or two regardless of the status.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi everyone,

    Thanks for your patience while we investigated the issues affecting PC NA and the PTS. As some of you discovered, we ran into a problem where character saves on the PTS were affecting PC NA, and we locked both PTS and PC NA immediately. We have temporarily locked the affected accounts until we can restore them to a good state and will be notifying those players via email, and will be rolling back progress on PC NA from those accounts to April 15th at 9:30am EDT. This will take several days to complete and we will alert any affected players when their account is accessible again.

    To be clear, the rollback will primarily affect accounts that logged onto the PTS after maintenance on April 15th.

    Please note if your account is affected, when you attempt to log into your account on any realm you will see a message saying your account was suspended/banned. Rest assured this is not the case and it is simply how our system identifies locked accounts.

    The current plan is to bring PC NA back online within the hour, followed by the PTS later in the day.

    We will follow up in the coming days with more details summarizing what happened and will share any additional information at that time. We sincerely apologize for this inconvenience and thank you again for your patience during this matter.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi everyone,

    We’d like to start off by saying we know this situation is upsetting and we recognize many of you are angry. To say we’re disappointed this happened is an understatement and we empathize with those that are still locked out of their accounts. Just know we are working around the clock on this and will share more information about what happened and what we plan to do to make things right later this week.

    Today, the first priority is getting the PTS back online which should be finished in the next few hours. We will then focus on going through all the affected accounts to get everyone their accounts back as quickly as possible. We also want to clarify that any ill-gotten gains from this issue, such as items and all currency that may have been distributed, will be removed from the economy.

    Thanks for sticking with us and we’ll let you know when the PTS is back online.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Just wanted to let everyone know as of 5:10pm EDT, the PTS is back online. We are continuing to work on all affected accounts that are currently locked or need to be rolled back, and we will provide the next update in the morning (eastern time). Thanks for the ongoing patience and understanding while we work through this issue.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Good morning everyone,

    As promised, we’d like to share an update to the PC NA server downtime we had Monday night.

    First, to acknowledge that many of you lost playtime in the middle of an event, we will extend the Anniversary Jubilee event by two days on PC NA and EU to allow you a little extra time to obtain any items or event tickets. This puts the new end date for PC realms on April 25 at 9:59am EDT. Additionally, all PC NA players will be granted an extra 200 Seals of Endeavor.

    For those who are still blocked from logging in, we are continuing to work on resolving the locked accounts and a small number of accounts will be unlocked later today. To set expectations, unlocking the remaining accounts and performing the necessary rollbacks will require server downtime and at that point, all remaining accounts will be unlocked at the same time. The soonest this will occur is by the end of the week but we will keep you posted on the timeline.

    To that end, for anyone whose accounts have been affected by this issue and are currently locked or being rolled back, we will be emailing you an ESO game code so you can play ESO (either on the Live servers or the PTS) while you wait for your main account. Please note that this is not in any way intended to replace your current account, and we are continuing to work around the clock to restore all affected accounts. We are also discussing some additional in-game items we’d like to send to those that have their accounts locked, which we will share very soon.

    Thanks again for your continued patience and understanding.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Just want to make sure this line was seen:
    We are also discussing some additional in-game items we’d like to send to those that have their accounts locked, which we will share very soon.

    The game code and event extension is not the only thing we plan to do to make things right for those with locked accounts. We hope to share these details in the next day or so.
    Edited by ZOS_GinaBruno on April 17, 2024 3:56PM
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_Kevin
    ZOS_Kevin
    Community Manager
    Hi All, we just wanted to clarify that everyone identified with a locked account as a result of the PTS issues - with the exception of a handful of players who seriously exploited - will receive future communication about additional in-game items.

    Community Manager for ZeniMax Online Studio and Elder Scrolls OnlineDev Tracker | Service Alerts | ESO Twitter
    Staff Post
  • ZOS_JessicaFolsom
    ZOS_JessicaFolsom
    Community Manager
    For those who are still blocked from logging in, we are continuing to work on resolving the locked accounts and a small number of accounts will be unlocked later today.

    Hi, everyone. We have completed the restoration work for the small number of accounts mentioned in the quote above. Those players have received an email notifying them that their accounts are now restored and available. Work on the rest is ongoing, and we will have another update for you tomorrow. Thank you for your continued patience.
    Jessica Folsom
    Associate Director of Community - ZeniMax Online Studios
    Facebook | Twitter | Twitch | Tumblr | Instagram | YouTube | Support
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    As mentioned in our post yesterday, we plan to provide some additional in-game items for those that had their accounts locked due to this issue. This is in addition to the Anniversary Jubilee extension for all PC players until April 25 at 9:59am EDT, and the 200 Seals of Endeavor for PC NA players that were affected from the sudden downtime Monday evening.

    The items we plan to grant to the accounts that were locked are as follows:
    • 16,000 Seals of Endeavor
    • 5x Glorious Anniversary Jubilee Gift Boxes
    • Trueflame Replica Style Page
    • Staff of Worms Replica Style Page
    • Barbas Helmet Replica Style Page
    • Sunna'rah Replica Style Page
    • Ul'Vor Staff Replica Style Page
    • 25x Jubilee Confetti Packs
    We do hope this helps lessen the inconvenience this situation has caused. We’ll follow up when we know when these items will be delivered to you.

    We’d also like to clarify that some locked accounts with an active ESO Plus membership through our account system did have their subscription renewal automatically cancelled; any time left from a current subscription is still active. Please note you will need to manually re-subscribe once you regain access to your account, should you decide to do so. If your subscription was through another storefront including Steam, Xbox or PlayStation, your subscription and renewal are still running. To make up for any lost subscription time, we will be granting one month of ESO Plus (including the crowns) to active ESO Plus subscribers whose accounts were locked as a result of this issue.

    In the meantime, work continues on these locked accounts and the associated rollbacks. We hope to have an update on this by the end of the day and will keep you posted.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_JessicaFolsom
    ZOS_JessicaFolsom
    Community Manager
    As an addendum to the above post that details the items we plan to grant to the accounts that were locked, we want to clarify that we will be granting these on both EU and NA for anyone who lost access to their account.
    Jessica Folsom
    Associate Director of Community - ZeniMax Online Studios
    Facebook | Twitter | Twitch | Tumblr | Instagram | YouTube | Support
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi everyone,

    As we head towards the weekend, we’d like to give you an update on where we stand with unlocking the remaining accounts. Work is still ongoing for any necessary data restoration and ensuring we remove all items and currency that may have made their way from the PTS to the PC NA server. This is a long and complicated process, and we want to ensure we get it right before moving onto the next step of unlocking these accounts.

    On Monday, April 22, we have patches scheduled for both PTS and PC NA/EU. Once those maintenances are complete, we will discuss the best and soonest time to schedule the downtime needed on PC NA to perform the actual rollbacks and subsequently, unlock all remaining accounts. We’ll let you know as soon as possible when plans are solidified but expect it to occur early next week. Additionally, the items we plan to grant to these accounts will be delivered at a later date, which we’ll announce as soon as we know.

    Once again, thank you for your ongoing patience and we’ll follow up with everyone on Monday.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hello everyone,

    We completed the scheduled PC maintenance earlier today but maintenance on the PTS is ongoing. Additionally, we are continuing to perform extensive testing to ensure everything is correct once we perform the necessary rollbacks. Due to all this, in addition to already having console maintenance scheduled on Wednesday, currently the earliest we’ll be able to schedule the needed maintenance on PC NA for the rollbacks and to unlock accounts is Thursday this week. We’ll keep you posted as soon as we know the exact date and time.

    In the meantime, given the length of time these accounts have been locked, we will also plan on granting the Cumberland Anniversary Attire costume to affected accounts, currently on day 21 in this month’s daily login rewards.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi everyone,

    Now that all scheduled maintenance has concluded on the PC, PTS and console megaservers, we’d like to outline next steps and what to expect in the coming days:
    • Tomorrow, April 25 beginning at 6am EDT, we’ll be taking the PC NA megaserver offline for maintenance. During this time, we’ll be restoring affected PC NA accounts back to their state from Monday, April 15 at 9:30am EDT.
    • Once PC NA maintenance concludes by 2pm EDT, anyone currently with a locked account due to the PTS issue last week, on any server, will once again have access to their accounts. You will also receive an email once your account is unlocked.
    • Affected console players will have the items outlined in this post upon logging into their accounts through in-game mail.
    • On Monday, May 6, we will be granting the items outlined in this post to affected PC NA and EU players, during the scheduled maintenance. Again, these items will be sent and claimed through in-game mail.
    • Also on Monday, May 6, we will be granting the one month of ESO Plus with 1650 crowns to those whose accounts have been locked and have an active ESO Plus subscription through the ESO website.
      • Note that this time will automatically be added to whatever remaining time you already have.
      • For those that subscribe through another storefront including Steam, Xbox or PlayStation, we will keep you posted with any updates.
    We’ll let everyone know when maintenance is wrapping up tomorrow. Thanks, once again, for your ongoing patience.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Hi all, just to give you a quick update, maintenance is going well and we expect to open soon. We'll post here once the servers are open and accounts have been unlocked. Thanks!
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    PC NA is opening now and all previously locked accounts are now available. Please let us know if you run into any unexpected issues with your accounts, otherwise we will follow up as we approach the next update on May 6.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Everyone, we need to take PC NA offline to investigate an issue that just surfaced. Please stand by for further updates.
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
  • ZOS_GinaBruno
    ZOS_GinaBruno
    Community Manager
    Thanks for everyone's patience with the sudden downtime; we needed to restore the data from a small number of accounts and have successfully done so. We should be re-opening the PC NA megaserver very soon!
    Gina Bruno
    Senior Creator Engagement Manager
    Dev Tracker | Service Alerts | ESO Twitter | My Twitter
    Staff Post
This discussion has been closed.