Maintenance for the week of December 23:
· [COMPLETE] NA megaservers for maintenance – December 23, 4:00AM EST (9:00 UTC) - 9:00AM EST (14:00 UTC)
· [COMPLETE] EU megaservers for maintenance – December 23, 9:00 UTC (4:00AM EST) - 14:00 UTC (9:00AM EST)
· [COMPLETE] NA megaservers for maintenance – December 23, 4:00AM EST (9:00 UTC) - 9:00AM EST (14:00 UTC)
· [COMPLETE] EU megaservers for maintenance – December 23, 9:00 UTC (4:00AM EST) - 14:00 UTC (9:00AM EST)
There needs to be more communication and acknowledging that they are looking into issues.
baconaura
✭✭✭
Servers are just mass booting people, and preventing people from logging back in after getting booted. there are atleast 5 threads bringing this up, and no communications at all from ZOS. Is there even an oncall team working on this? Can we get some communication that you guys are aware this is an issue and its being investigated? Most companies send out a notification or alert if there is an outage, letting customers know it is being worked on, and updating progress and letting the customer know it is resolved.
It just feels like another day where servers are burning down with noone at the helm of the ship, and forum mods being fed to lions trying to sort everything out when they come into work the next day.
It just feels like another day where servers are burning down with noone at the helm of the ship, and forum mods being fed to lions trying to sort everything out when they come into work the next day.
5
-
sarahthes✭✭✭✭✭
✭✭Servers are just mass booting people, and preventing people from logging back in after getting booted. there are atleast 5 threads bringing this up, and no communications at all from ZOS. Is there even an oncall team working on this? Can we get some communication that you guys are aware this is an issue and its being investigated? Most companies send out a notification or alert if there is an outage, letting customers know it is being worked on, and updating progress and letting the customer know it is resolved.
It just feels like another day where servers are burning down with noone at the helm of the ship, and forum mods being fed to lions trying to sort everything out when they come into work the next day.
The people who would likely post are almost certainly sleeping since it's doubtful they slept last night.
Give the zos folks who are manning things right now (likely a skeleton crew) some time to phone and wake up the comms people.8 -
baconaura✭✭✭They have that section at the top of this forum as the only way to communicate these kind of things to their customers.
If they had a dedicated server status page like most other companies, that would be great. But for now this is the only way they are using to communicate with us, and it is definitely a bottleneck getting information out.
The oncall engineer should be able to post something at the top of this forum letting us know they are aware of the issue, and its currently being looked at.
2 -
sarahthes✭✭✭✭✭
✭✭They have that section at the top of this forum as the only way to communicate these kind of things to their customers.
If they had a dedicated server status page like most other companies, that would be great. But for now this is the only way they are using to communicate with us, and it is definitely a bottleneck getting information out.
The oncall engineer should be able to post something at the top of this forum letting us know they are aware of the issue, and its currently being looked at.
After last night it's probably an intern with a list of phone #s watching things.1 -
MasterSpatula✭✭✭✭✭
✭✭✭✭✭
The people who would likely post are almost certainly sleeping since it's doubtful they slept last night.
Give the zos folks who are manning things right now (likely a skeleton crew) some time to phone and wake up the comms people.
I think OP's point is that if you run a 24/7 service, you need 24/7 support staff, including communications. And OP is not wrong.
Yeah, yesterday sucked for them. But two days in a row of server nonsense is a terrible look, and not even having someone on to communicate right now is thoroughly avoidable."A probable impossibility is preferable to an improbable possibility." - Aristotle10 -
baconaura✭✭✭They have that section at the top of this forum as the only way to communicate these kind of things to their customers.
If they had a dedicated server status page like most other companies, that would be great. But for now this is the only way they are using to communicate with us, and it is definitely a bottleneck getting information out.
The oncall engineer should be able to post something at the top of this forum letting us know they are aware of the issue, and its currently being looked at.
After last night it's probably an intern with a list of phone #s watching things.
Unlikely an intern would be on the oncall rotation. If that was the case, that would be a really bad decision by management.
Look i get it, its been a rough week and the datacenter stuff was unexpected, but they got communications out, and eventually got it back up today. When stuff gets messy and hits the fan, they all gotta be all hands on deck.
It's sorta their job to do this.
I work in tech, and every few weeks im oncall for my rotation. If there is an outage in production, im definitely working on it, and and not sleeping until we've resolved it, or have a clear plan and timeline to tackle it. My manager is kept up to date on the situation and managing the comms with stakeholders to send to their clients, and contacting other teammates to assist if needed.
Its our job and responsibility. We cant just ignore our customers and deal with it in the morning. We owe it to our customers to let them know we are taking it seriously and working on it, and provide an estimated timeline if possible.
I feel like expecting some communication and acknowledgement on widespread server outage is atleast the bare minimum.
Edited by baconaura on December 14, 2024 5:45AM6 -
sarahthes✭✭✭✭✭
✭✭They have that section at the top of this forum as the only way to communicate these kind of things to their customers.
If they had a dedicated server status page like most other companies, that would be great. But for now this is the only way they are using to communicate with us, and it is definitely a bottleneck getting information out.
The oncall engineer should be able to post something at the top of this forum letting us know they are aware of the issue, and its currently being looked at.
After last night it's probably an intern with a list of phone #s watching things.
Unlikely an intern would be on the oncall rotation. If that was the case, that would be a really bad decision by management.
Look i get it, its been a rough week and the datacenter stuff was unexpected, but they got communications out, and eventually got it back up today. When stuff gets messy and hits the fan, they all gotta be all hands on deck.
It's sorta their job to do this.
I work in tech, and every few weeks im oncall for my rotation. If there is an outage in production, im definitely working on it, and and not sleeping until we've resolved it, or have a clear plan and timeline to tackle it. My manager is kept up to date on the situation and managing the comms with stakeholders to send to their clients, and contacting other teammates if needed.
Its our job and responsibility. We cant just ignore our customers and deal with it in the morning. We owe it to our customers to let them know we are taking it seriously and working on it, and provide an estimated timeline if possible.
Sure, but if the majority of key stakeholders are sleeping after dealing with yesterday's crisis, it's gonna take them a little time to respond.
Every indication is that this is an opportunistic DDoS designed to make ZOS look bad, and I would expect these to ramp in intensity over the next few days until the perpetrators get bored. If you've been thru enough DDoS attacks you kind of recognize what they feel like and that's what this is.
Today being the day after yesterday's crisis, they're probably at their most sluggish. Some folks may be heading out for Christmas break also. I'm just inclined to extend them a little grace because of that. It doesn't cost me anything, and it makes me less frustrated at the same time.7 -
Desiato✭✭✭✭✭
✭It could be a ddos, but it could also be related to the data center issues yesterday.
I noticed they don't seem to be using akamai anymore...spending a year dead for tax reasons1 -
OtarTheMad✭✭✭✭✭
✭✭https://forums.elderscrollsonline.com/en/discussion/670288/eso-na-eu-megaservers-offline-dec-12#latest
They communicated constantly. They need to sleep ya know. I think the EU part of the forums continued to update when NA was asleep.0 -
Destai✭✭✭✭✭
✭✭✭I’m usually pretty direct when I feel like they’re not communicating well enough and I will say they were on point through this whole situation. It was a big crisis and they pulled through.4 -
MasterSpatula✭✭✭✭✭
✭✭✭✭✭OtarTheMad wrote: »https://forums.elderscrollsonline.com/en/discussion/670288/eso-na-eu-megaservers-offline-dec-12#latest
They communicated constantly. They need to sleep ya know. I think the EU part of the forums continued to update when NA was asleep.
This isn't so much about yesterday as about what's literally happening right now."A probable impossibility is preferable to an improbable possibility." - Aristotle5 -
Destai✭✭✭✭✭
✭✭✭I would say give it time. Realistically it’s probably the same resources who are recovering. We’ll get through this guys.2 -
OtarTheMad✭✭✭✭✭
✭✭I was replying and then saw the red notice at the top of the forums so... hopefully answers soon for you.1 -
MasterSpatula wrote: »
The people who would likely post are almost certainly sleeping since it's doubtful they slept last night.
Give the zos folks who are manning things right now (likely a skeleton crew) some time to phone and wake up the comms people.
I think OP's point is that if you run a 24/7 service, you need 24/7 support staff, including communications. And OP is not wrong.
Yeah, yesterday sucked for them. But two days in a row of server nonsense is a terrible look, and not even having someone on to communicate right now is thoroughly avoidable.
Absolutely agree!!! Not communicating when crap like this is going on for two days, to millions of players all over the world, is utter incompetence on ZOS part.
We are the people making you millions of dollars! We deserve to be advised when your hardware is broken and given a true estimate of how long it is going to take to fix the situation.
Coz that’s lost time that we have paid for.
You owe us at least an explanation and the daily rewards we lost coz you were down!!!0 -
OtarTheMad wrote: »https://forums.elderscrollsonline.com/en/discussion/670288/eso-na-eu-megaservers-offline-dec-12#latest
They communicated constantly. They need to sleep ya know. I think the EU part of the forums continued to update when NA was asleep.
You may not have heard of shift work 🤷🏽♀️3 -
TheMajority✭✭✭✭✭you really want a sleep deprived person working on this? you got any idea how unfocused a sleep deprived person become?
let them sleep, go watch a movie until it blows over or somethingTime flies like an arrow- but fruit flies like a banana.
Sorry for my English, I do not always have a translation tool available. Thank you for your patience with our conversation and working towards our mutual understanding of the topic.2 -
Orbital78✭✭✭✭Ping statistics for 198.20.200.1:
Packets: Sent = 35, Received = 0, Lost = 35 (100% loss),
it seems things have got worse, but at least there is a message about it being investigated now.2 -
baconaura✭✭✭TheMajority wrote: »you really want a sleep deprived person working on this? you got any idea how unfocused a sleep deprived person become?
let them sleep, go watch a movie until it blows over or something
You really think its 1 overworked dude keeping it all together? There is actually a team responsible for this.
Maybe your okay with not having your time respected and thats cool, you do you and enjoy that movie.
But i and others set aside time on a friday night to do content as a group like trials and we get outages for a few hours and no comms, and no idea if we should call it or wait it out. Like some of us declined plans in real life because we made commitments to our team for regular scheduled raids, but in the end we cant do that content because of server issues and because we said no to plans, we wasted a night.
Thats the part that is frustrating when you tell me to let them sleep, and take the night off. This isnt a 9-5 game, its a 24/7 live service that needs to be supported. There should be people on rotation that can investigate this and give an update.
But it is good to see the update in the announcements that they are aware of the issue.4