Studio Director Update – Recent Service Interruption for ESO on PlayStation

hands0medevil

OutLaw_Nynx wrote: »

hands0medevil wrote: »

TwinLamps wrote: »

I aint from this platform but this response is very fair.
I wish any potential future issues such as this one was would have similar response as well.

Yeah, I wish pvp and cyrodiil's issues had this much attention and response

You literally couldn’t even play the game. Goodness.

For few days, two I guess. And for how long we struggle with main pvp mode in this game? Few years.

Hotdog_23

Thank you for the quick response. Found it to be honest measured and appropriate compensation for time lost. Feel as if I have been paid back made whole for the loss.

Please now fix the green tree mess and you will have hit a homerun out of the park

Stay safe and enjoy the journey 😊

Mayrael

VaranisArano wrote: »

I remember when the Anniversary Event was extended for everyone for a week due to issues with PC/EU. So I wasn't surprised to see this one extended.

It's very nice to see that your team has since considered how to address platform-specific issues like this. I appreciate that not only is the event extended for everyone, but that you also specifically compensated Playstation players for the lost experience and their concerns over lost daily login rewards.

Yeah, shame they don't do this on PC/EU... But then they would have to give use like 1 month of free eso+ every year.

Gythral

With the event extened is the Impresarios inventory gonna be expaned, so we can spend these tickets?

Xuhora

Gythral wrote: »

With the event extened is the Impresarios inventory gonna be expaned, so we can spend these tickets?

you seem to miss the point. this is meant as a compensation for the fellows who could not log in. its not meant as a reward for the other consoles systems.
just bank up the tickets for the next event and enjoy the additional jubilee boxes.

Sarannah

Thank you ZOS, and thank you to all the PS players for your suffering!

LalMirchi

I don't play on PS4 but it was sad to see the downtime. Thanks for excellent response and compensation.<3

BisDasBlutGefriert

[/quote]
how they worked into the wee hours of the night to fix this error, I am thoroughly impressed. I feel they went above and beyond. My faith in this game has been restored.[/quote]

Exactly This^^^ Gina's earlier post was at like 2:40am their time...that's dedication!! And admitting it was entirely their fault to how many thousands and thousands of players....that's definitely a +8 from me!!!

Excellent work team!

🥳🥳🥳

Alpharos7

Thank-you ZOS for working so hard to get the PlayStation servers back up and running, and for the compensation you have given us. Not everyone can stand up and say they messed up, but it is great to see that you are able to do this. Apologies for all the abuse I gave to you the other day.

It's also great to see communication like this, which we need more of. If we had been notified earlier that the PlayStation server was experiencing major issues than it may have alleviated much of the thunder.

Looking forward to seeing what the future holds for ESO.

jaws343

Diminish wrote: »

In the end, I feel as if this is all that many of the people that were vocal about this issue wanted, transparency! Please, no more of this pushing back maintenance windows multiple times, and giving updates that lack any real context. Transparency is a good thing no matter what caused the underlying issue. People just want to know what is going on, and when to expect services to resume. If you ask me, most of this post should have been included in your first update after missing the initial ETR; it would have alleviated the masses. All-in-all though, great job getting things resolved quicker than expected, and actually being informative to the community on the nature of the issue; it's really all most of us ask for, the compensation is icing on the cake. Well done, ZOS!

I think the problem is they really can't disclose most of what was in this post in advance. Likely because the underlying causes weren't fully known at the time and because they still need to work within their best interest. And their best interest isn't taking on all of the blame for the issue before knowing that is the case. That is why the initial outreach is always vague. Any more specific and they open themselves to even more criticism and liability that may otherwise not be necessary.

Phanex

I agree, they owned up to their mistakes and did a decent job of keeping everyone informed (after the fact).

However, now is good time to re-evaluate certain aspects of the game (which I'm sure someone is gonna say my examples don't affect them, which is okay).

For example:

-Daily login rewards. Why time gate it, why not let it build up and let us go in and collect them when we have the chance to get on the game?

Especially during maintenance, updates, and if the player goes on vacation for a few days. What are we suppose to do, take our ps4/xbox with us? I know some of the rewards are lackluster, but still, why punish us if we can't get on the game? It could be an extra ESO+ bonus.

-It would also be nice if they release the updates in small patches here and there for folks who don't have the best internet in the world (or for those who have data caps). I think the last major update it took about 2 days for me to get it all downloaded.

Its not ESO's fault, I'm not blaming them, but it would be nice.

-Also, another thing I'm seeing: PC folks get their updates a few weeks before xbox/ps4, okay cool. The problem I'm seeing is that people will start reporting bugs and issues after the PC roll out and ESO works on them and says okay we know the issues we'll get it fixed next patch.

Okay, my question is, With ESO knowing what bugs were in PC land, and knowing how to resolve it, why not go ahead and fix it for the xbox/ps4 before they roll out the update? We just get an "we are getting messages about this issue, we know how to fix and will patch in future update". Like..you already knew you were sending out a defect product, why not fix it?

I'm sure there are reasons like, its too late, they have to get permission, its easier to fix it afterwards, it takes time, etc. etc. But, I'm also seeing these "bugs/issues" reported on the PTS servers too. Yet these bugs still make it to the final product.

Just weird.

Destai

ZOS_MattFiror wrote: »

Now that the dust has settled and the ESO PlayStation service is back up, here's a quick summary of what happened and what we'll do to make up the day of playtime that our PS4/5 community lost. The extended downtime was particularly unfortunate timing as yesterday was a holiday for many of you, and we know you were looking forward to spending your time off adventuring in Tamriel. We understand how frustrating this was.

First, the issue itself was caused by a service problem between ESO and the PSN platform and was entirely our fault. Because it was a problem between the ESO game and the PSN service, our solution was to rebuild the ESO patch, test it, and submit it for certification. This process takes time, which is where the initial 24-48 hour estimate came from.

However, with critical help from our PSN partners, we figured out a much faster solution - to modify the service itself. With that, it only took a service update and a restart, and after testing and verification, we were back in business very late last night east coast USA time and as I type this everything is running normally.

We know this wasn't fun. We don't like extended downtime either. I apologize for the situation and for the inconvenience and frustration it has caused. Thanks for being such a great community and bearing with us through this.

To alleviate some of the lost playtime from ESO PlayStation players, here's what we're going to do:

• We are extending the Anniversary Jubilee event by a few days for ALL players – it will now run until Thursday April 15 at 10am EDT for all platforms
• On PS4/5 only: The day 7 daily login reward will be modified to give five 150% experience boost scrolls (to make up for lost time)
• On PS4/5 only: The day 8 daily login reward will be modified to give players this month's pet (the Ayleid Wolf Pup, previously awarded on day 21) to make sure no one misses the reward due to disruption in their play schedule
• On PS4/5 only: The normal day 7 and 8 rewards will be moved later in the month, so you won't miss out on them either

Thanks again, and see you all in Tamriel.

Matt

@ZOS_MattFiror Thanks for the elegant response and understanding that some kind of compensation is required here. I hope you guys reconsider your deployment strategies, particularly when you plan maintenance and address the skill vacuums that apparently exist in your organization. While I don't know any specifics, sounds like there's some room for improvement in your devops as these maintenances rarely go as planned. Lastly, this is the level of candor and detail we'd like on a more regular basis, particularly from the CM team.

YandereGirlfriend

Credit where it's due, this is a well thought-out compromise!

Madhojo

Lol I love everyone who isnt even a PlayStation user saying this is "fair compensation."

How?

ZOS moved the daily rewards around... Oh yeah wow what compensation! Oh man, I get a reward 2 weeks earlier than planned wow!!! Mind blown.

No way xp scrolls?! Wow! Man talk about a drop in a bucket, i only need about a million of those to get to 3600, wow so amazing. Woo wow!

I mean its not like say I pay for ESO Plus and have lost over 48 hours thanks to extended maintenance on PS4. Nothing like spending real money for a premium service, losing 2 days of it, and getting completely ignored by ZOS. Oh yeah but the jubilee event got extended 2 days, woo wow such a great event too. Amazing!

Just wow!

method__01

when you going to fix banking issue?this month,this year or this decade?

EvilAutoTech

method__01 wrote: »

when you going to fix banking issue?this month,this year or this decade?

D. None of the above.

ArchMikem

Let this also be a lesson to the community. Hold your insults when issues happen. ZOS won't give an official response until things are resolved, not before. All these thank yous and kind words ring a little hollow after some of the threads I saw.

Rudrani

This is a nice OP.
I just have one honest question for ZOS...

do you have a test environment for PS servers?

It seems like you don't
but that is nearly impossible to imagine.

I say it seems like you don't because you only realized you had a problem with PSN after you put the patch on your live servers. You said you tested things, on the live servers and discovered "hitching, making the game almost unplayable."

Why on earth would you test something after you put it on your live servers?
Unless you don't have any other server connected to PSN?

But I find that reaaaaaaly hard to imagine.

OrdoHermetica

While this is fair compensation, I really, really wish this was the first time something like this had happened, rather than being one of about a dozen event-related issues like this in the last half-dozen years. I'm just... tired of this, you know? I wish I couldn't consistently count on something like this happening with regularity, and yet, I remember so many threads from ZOS like this one...

Ah, well.

jaws343

Rudrani wrote: »

This is a nice OP.
I just have one honest question for ZOS...

do you have a test environment for PS servers?

It seems like you don't
but that is nearly impossible to imagine.

I say it seems like you don't because you only realized you had a problem with PSN after you put the patch on your live servers. You said you tested things, on the live servers and discovered "hitching, making the game almost unplayable."

Why on earth would you test something after you put it on your live servers?
Unless you don't have any other server connected to PSN?

But I find that reaaaaaaly hard to imagine.

You test things on internal setups and live. Because often bugs that don't exist internally will appear in live due to the different environment. It's just the reality of testing software.