PC-EU Server Update - April 11

Verbal_Earthworm

lordrichter wrote: »

UrQuan wrote: »

I get that you're upset (and understandably so), but your posts prove that there's literally nothing ZOS can do that will satisfy you, so why should they do anything other than ignore you?

I am reasonably sure they are not ignoring him (or her).

Like you, however, I don't expect an update just a couple days later, and on a weekend, when they are talking about something that is 2 weeks out. I expect an update on the next ESO Live, if it has not already happened.

To be perfectly honest, my ESO experience since the queue was initiated 3 days ago has been extremely poor to say the least.

Being told to wait patiently like a good little customer for 2 more weeks of this queue is tantamount to trolling at the moment.

You would not like this to happen to you.

As it stands, I am looking at two weeks of waiting to login for my tickets just to crash and get put into a queue again before quitting and going to do something else with my time.

I'd just like to imagine things will be sorted faster.

WhiskeyDad6

First let me say that I'm fairly new to the game and I rarely post comments about any game unless very compelled to do so. I'm also not a big MMO player (aside from WOW and a couple others back in the early 00's), but I've been very impressed with what Zenimax has done with the game and thoroughly enjoyed my time so far.

I was going to leave a comment a few days ago as i didn't see anyone linking this article but it took me a couple days to get a forum invite. I'm glad to see Mr. Firor include the information from that article in this post now. However, based on Mr. Firor's comments this seems like more of a reactive response rather than a proactive one based on how long they have known about this, which worries me (see tldr below).

I'm not sixteen anymore so i won't be cancelling my membership immediately or making dramatic claims about doing so, but i wanted to share my concern based on personal and professional experience. I definitely will continue to play and sincerely hope they do have a plan in place and are able to add/upgrade the server hardware in the next 2-4 weeks. If it goes beyond that and certainly into the Elswyr release i would most likely cancel my membership, for a while at least, and recommend the same to any current or potential users. Fingers crossed...

TLDR:

I managed the data forensics/processing department with over 40 employees at a mid level DM company for many years with our own in house data center that processed Terabytes of data weekly/daily. While i'm not an IT expert i worked very closely at the time with our IT department on software and hardware upgrades to our servers. While that's probably small compared to what Zenimax is dealing with in terms of scope, i do have an idea of what they are facing and can sympathize.

That being said, as others have noted Mr. Firor's comments seem to imply they started having issues in January and saw them progress over the last couple months. This coupled with the free weekends/ESO plus trials (from which i joined in January) and the upcoming dlcs it seems they should have been able to predict a significant increase in users and at the very least had a plan in place and begun first steps (those requiring minimal investment); lining up hardware, schedules for overtime, additional personnel, etc. Six weeks ago when they saw their highest user numbers i think would have been the time implement said plan so as to keep that cushion of bandwidth/space below their cap/point at which the servers are overloaded. At least that's what we did with my department when we saw significant business/data increases coming, with enough time to be proactive of course.

Long story short, the fact that they use the crashing of the servers to trigger hardware addition/upgrades rather than a predetermined point (X% of server capacity/load and or x# of errors, x# of new/returning users, etc.) as a red flag, is the real concern and drastically increases the chances that we'll see this again down the road. Granted being more proactive may hurt your bottom line upfront by increasing cost of hardware, billing hrs, personnel, etc. that you may not need down the road, but it will keep the player base happy, bring in more customers and provide more stable long term growth and revenue. Food for thought Zenimax...

Thank you for the amazing game so far!

WhiskeyDad

FierceSam

L

UrQuan wrote: »

Verbal_Earthworm wrote: »

UrQuan wrote: »

Verbal_Earthworm wrote: »

lordrichter wrote: »

Verbal_Earthworm wrote: »

Day 3 of the Login Queue Fiasco and still no word on when a fix is coming.

Um. They said it would be about 2 weeks, right at the top of this thread.

It's very easy to be patronising when the issue doesn't even affect you isn't it?

I hope you get a login queue of your own.

Whether the issue affects him or not, he's right and you're wrong. They already said when they expected the fix to be coming, but you chose to ignore that.

They said a lot of things, some of which were patently lies, so pardon me if I need something more concrete and current.

If this isn't concrete and current enough for you, then nothing ever will be:

ZOS_MattFiror wrote: »

We expect the time frame for this to be about two weeks, but we will keep you posted.

It's from just 2 days ago; were you expecting hourly updates on whether they're still expecting the time frame to be the same?

I get that you're upset (and understandably so), but your posts prove that there's literally nothing ZOS can do that will satisfy you, so why should they do anything other than ignore you?

There is something ZOS can do - provide a properly working EU server.

It’s that simple.

However, they have shown that despite spending a lot of time planning and preparing for a year long ‘Season of the Dragon’, they seem to have spent no time or effort in planning for success or managing their EU system. EU players are angry and frustrated that despite being repeatedly told their infrastructure failings were proving detrimental to the game on the EU server, ZOS did nothing. Aside from implementing a queueing system, which operates with ZOS-like precision, there is little evidence ZOS is doing anything now.

The notion that this suddenly happened or was unpredictable is as fatuous as the notion that ZOS thought the group finder issues had magically ‘been fixed’, despite copious evidence to the contrary. And that has turned out so well.

With the PTS launching on Monday, which always leads to a loss of performance on the regular EU server, my guess is that queueing will be here for a lot longer than 2 weeks.

If you use the EU server, prepare to get locked out of Elsweyr on a very regular basis. It will make FO76 look like a roaring success.

888and888

Khaadrun wrote: »

So, they are telling all the PC/EU players that they've been aware of an issue for the last 3 months and a half and yet didn't do anything until it broke something?

How in the seven hells can you come out, with a straight face, and say to your loyal player base and say: "Hey, I knew that the population was starting to overcrowd the servers and yet i didn't upgrade anything".

...

Are you surprised? I think this is ZOS’ modus operandi.

They know that they have arguably the best MMO available and that their customer base is loyal, no matter how many lies they throw around.

Sad, but true...

Valenor

A minor stepback if we can get a better experience soon.

lelink88

I think server update will come out with Elsweyr chapter

Brandanus

WhiskeyDad6 wrote: »

...the fact that they use the crashing of the servers to trigger hardware addition/upgrades rather than a predetermined point (X% of server capacity/load and or x# of errors, x# of new/returning users, etc.) as a red flag, is the real concern and drastically increases the chances that we'll see this again down the road.

the hardest cap for any IT infrastructure is $budget. Then, an online RPG is very likely not based on autoscaling groups with clear dependencies, more a mix in classical co-hosting of awesome complexity: the monolithic monster you hide in your basement with a huge "no touchy!" sign.
That being said - thanks for budget before fiscal year closes Good luck for the people who need to implement it! Thanks for fixing the queue and thanks to the community for making a stance!

Elsonso

WhiskeyDad6 wrote: »

Long story short, the fact that they use the crashing of the servers to trigger hardware addition/upgrades rather than a predetermined point (X% of server capacity/load and or x# of errors, x# of new/returning users, etc.) as a red flag, is the real concern and drastically increases the chances that we'll see this again down the road. Granted being more proactive may hurt your bottom line upfront by increasing cost of hardware, billing hrs, personnel, etc. that you may not need down the road, but it will keep the player base happy, bring in more customers and provide more stable long term growth and revenue. Food for thought Zenimax...

I have had similar thoughts. Three things came to mind.

First, ZOS is, and always has been, very cagey about the megaservers and capacity. What was said by Firor was what they felt was necessary to say, but it is probably not the whole picture. It is possible that they had mitigation plans in place for this very situation, but that the magnitude was larger than projected, the timing was different than expected, or both. That mitigation might even have included additional hardware capacity that was already in the procurement pipeline. We really don't know what they have not said.

Second, projections and anticipations are tricky things, and they are often wrong. Business wants to be ahead of the curve, so things like this don't happen, but they don't want to over spend by building something that is more than what they will need.

Third, they watch the megaservers very closely. I am sure some people think that ZOS is clueless about EU performance problems, but I highly doubt that. They watch the servers. I have seen the room and the people who do this. I would be surprised if ZOS knew less about the server performance issues in EU than the players.

Out here in the forum, it is easy to armchair, and doubly so when we get to armchair about things in hindsight. It is much harder to be responsible for these things, and have to make decisions based on what is going to happen in the future.

Ultimately, this is all stuff that ZOS is probably discussing internally. That is the best place to discuss it, if we want goodness to come out of this. Discussing it out here is entertaining, perhaps therapeutic, but not something that will help in the future.

KoultouraS

lordrichter wrote: »

WhiskeyDad6 wrote: »

Long story short, the fact that they use the crashing of the servers to trigger hardware addition/upgrades rather than a predetermined point (X% of server capacity/load and or x# of errors, x# of new/returning users, etc.) as a red flag, is the real concern and drastically increases the chances that we'll see this again down the road. Granted being more proactive may hurt your bottom line upfront by increasing cost of hardware, billing hrs, personnel, etc. that you may not need down the road, but it will keep the player base happy, bring in more customers and provide more stable long term growth and revenue. Food for thought Zenimax...

I have had similar thoughts. Three things came to mind.

First, ZOS is, and always has been, very cagey about the megaservers and capacity. What was said by Firor was what they felt was necessary to say, but it is probably not the whole picture. It is possible that they had mitigation plans in place for this very situation, but that the magnitude was larger than projected, the timing was different than expected, or both. That mitigation might even have included additional hardware capacity that was already in the procurement pipeline. We really don't know what they have not said.

There is no "larger than projected".
Stats show a great but not an overwhelming increase.
(If we consider the steam population as a representative slice of the "pie")
Furtermore as I point out (here), and since that post everyone else ;
They can't use the excuse this caught them off-guard because:
1. They have deliberatelly paid campaigns to sites/youtubers and other media related to MMOs so to promote them as a contender for best mmo 2018 and/or 2019
2. They were already established as a leading title of the genre.
3. They have been awarded noumerous awards
4. WoW was already in decline and the rest of the great upcoming titles of the MMO spectrum are still WiP.
5. MOST IMPORTANTLY they didn't had to wait either. The servers were already in a $hity condition looooong before the aforementioned incidents . For a whole year or so, despite the plethora of threads reporting the deterioration of the servers, they just opted to milk the hell out of the existing infrastracture and that was just about it, like it or not!

So , sorry there is NO excuse such as " We didn't expect that"...

bethsheba

Please don't buy the new hardware from the Clearance rack.

Verbal_Earthworm

lordrichter wrote: »

WhiskeyDad6 wrote: »

Long story short, the fact that they use the crashing of the servers to trigger hardware addition/upgrades rather than a predetermined point (X% of server capacity/load and or x# of errors, x# of new/returning users, etc.) as a red flag, is the real concern and drastically increases the chances that we'll see this again down the road. Granted being more proactive may hurt your bottom line upfront by increasing cost of hardware, billing hrs, personnel, etc. that you may not need down the road, but it will keep the player base happy, bring in more customers and provide more stable long term growth and revenue. Food for thought Zenimax...

I have had similar thoughts. Three things came to mind.

First, ZOS is, and always has been, very cagey about the megaservers and capacity. What was said by Firor was what they felt was necessary to say, but it is probably not the whole picture. It is possible that they had mitigation plans in place for this very situation, but that the magnitude was larger than projected, the timing was different than expected, or both. That mitigation might even have included additional hardware capacity that was already in the procurement pipeline. We really don't know what they have not said.

Second, projections and anticipations are tricky things, and they are often wrong. Business wants to be ahead of the curve, so things like this don't happen, but they don't want to over spend by building something that is more than what they will need.

Third, they watch the megaservers very closely. I am sure some people think that ZOS is clueless about EU performance problems, but I highly doubt that. They watch the servers. I have seen the room and the people who do this. I would be surprised if ZOS knew less about the server performance issues in EU than the players.

Out here in the forum, it is easy to armchair, and doubly so when we get to armchair about things in hindsight. It is much harder to be responsible for these things, and have to make decisions based on what is going to happen in the future.

Ultimately, this is all stuff that ZOS is probably discussing internally. That is the best place to discuss it, if we want goodness to come out of this. Discussing it out here is entertaining, perhaps therapeutic, but not something that will help in the future.

So you are saying that they watch the server so closely that they have known that their login queue is broken and have done for 4 days?

They can clearly see how the grace period upon disconnect they speak of is not functioning?

But rather than fix it they are discussing it internally so that "goodness" can come out of it?

Do they pay you to type this apologist nonsense?

Varana

lordrichter wrote: »

Third, they watch the megaservers very closely. I am sure some people think that ZOS is clueless about EU performance problems, but I highly doubt that. They watch the servers. I have seen the room and the people who do this. I would be surprised if ZOS knew less about the server performance issues in EU than the players.

Yes, I assume that is true.
The problem is the utter silence on any of this. Not once have they acknowledged, in the past months (let's be honest, this is not an issue only since January), that there may be problems with server capacity. Sure, at the time, if they thought they would be able to silently fix it, it may have seemed to be a good idea to just ignore the complaints and work on whatever they were trying to do.
But then, this has been ongoing for several months now, without them being able to solve the problem, and they still decided to not say anything at all. They totally deserve the backlash. It's only internet rage, they will survive it, but I feel they shouldn't be let off the hook without at least a mild sh**storm.

Ananoriel

I am sorry but there are many pathetic comments in this thread. Yes, the queues are annoying, and yes we have to hope that things will get better. But you don't have to get a subscription if you don't think it's worth it. And no, the game is not dead. And they are not lying about it, they are finally communicating something.

Fixer2050

As someone stated in few previous posts, they had forseen this would happen since January '19 and they still waited till the problem happen to act.

For me this is unacceptable for a such big company that hosts one mega server for the whole Europe and the problem occur to all of them and not just a small portion and they KNEW IT!

The worst thing is many players/refugees from WoW and other MMOs who joined the hyped train will leave because they did not have the chance the game as they should .

Verbal_Earthworm

I shouldn't have to remind anyone that we are in the middle of a 5 week event.

EU-PC players cannot properly participate in the event as they are constantly in login queues and disconnecting in a vicious loop.

We are lucky just to log in to eat some cake and get the daily tickets.

We are not able to remain in the server long enough to collect jubilee boxes.

Whatever is contained in those boxes is out of our reach now.

This will mean we will have to try and buy any limited items, costing us a lot of gold.

Our trade guilds will all be struggling this week and perhaps for two more weeks (probably longer knowing ZOS).

This is not at all fun and so you can expect that the affected players will be a tad peeved.

Unaffected players can post all they like, but they just appear as ignorant knee-jerk apologists.

EntolomaBlu

1) The last post in the old thread says .. new message 'over here', there is no link, I just searched for five minutes before I found where to look...

2) I did it, I just cancelled my subscription. 15 Minute queue is just not what I am paying for, I don't even want to wait 5 minutes, and then I forfeit more daily rewards. (It is not about 'frustration', 17year olds are 'frustrated', any adult knows that he is not getting what he pays for, and should know when not to accept this.)

3) The US may have different time zones, Europe is much more splintered by different languages (and cultural rifts). Can't imagine splitting it up would really be a problem. Why not start a mandatory survey about possible split scenarios on login, if that's a no-cost-solution and almost everyone is happy?

Elsonso

Do they pay you to type this apologist nonsense?

No need to get nasty. I don't work for ZOS, am not paid by ZOS, and I have no inside information to share about this, but they are not treading on unfamiliar ground for me. I've been on both sides of this sort of thing, and I have learned a lot in my prior experiences. I can remember a time when I would have gotten angry about stuff like this. Now, I know better.

Varana wrote: »

The problem is the utter silence on any of this. Not once have they acknowledged, in the past months (let's be honest, this is not an issue only since January), that there may be problems with server capacity.

I will not argue against the idea that they need to communicate better.

I appreciate the commentary from Firor, and that fact that he makes these statements. However... In my mind, when they need to push out a statement from Firor about a situation that has been brewing for several weeks, it is a sign that maybe they needed to be talking about it more, before it got to that point. He's a busy guy.

Again, this is part of what they need to discuss internally to decide whether there were improvements in what they could have done. Communication should be on the agenda, I think.

Fixer2050 wrote: »

As someone stated in few previous posts, they had forseen this would happen since January '19 and they still waited till the problem happen to act.

One of the things that I have been trying to inject into this discussion is that we do not know what actions they have been taking. To suggest that they waited to act is probably inaccurate. What we do know is that it was not enough.

Hopefully, in a couple of weeks, this will pass. While that happens, there are login queues on the EU servers. Could they have done something different to avoid this before it happened? I cannot say. They are working to fix it, and that is the next best thing.

Valdek

I'd be very interested to know what kind of investment it would take to improve the servers to the point that the most intense areas (e.g. large scale fights in cyro) run smoothly. I imagine that it would be quite a lot, however, perhaps a crowdfunding campaign or something similar could provide a big injection. This could be integrated into the crown store so you get a cool 'hamster' pet or something as a token reward.

I've played many MMOs and the players in ESO have shown by far the most generosity, altruism and willingness to help. Guilds are often showing that they are all keen to work together to support the creation of expensive guild houses. I'm sure that with this kind of player base, many ESO fans would be up for supporting ZOS in this way, if the goal was clear and benefits noticeable.

@ZOS_GinaBruno @ZOS_MattFiror what do you think?

Verbal_Earthworm

lordrichter wrote: »

I have no inside information to share about this

Except you already did:

lordrichter wrote: »

They watch the servers. I have seen the room and the people who do this.

Then to add insult to injury you state:

lordrichter wrote: »

I can remember a time when I would have gotten angry about stuff like this.

Except you aren't even on the PC-EU server, have not experienced the Login Queue Fiasco and are only here to troll apparently.

I tend not to get angry about things that don't affect me whatsoever also.

At this point you are just an unwelcome heckler.

Can't you find some other pointless cause to champion?

ManwithBeard9

Valdek wrote: »

I'd be very interested to know what kind of investment it would take to improve the servers to the point that the most intense areas (e.g. large scale fights in cyro) run smoothly. I imagine that it would be quite a lot, however, perhaps a crowdfunding campaign or something similar could provide a big injection. This could be integrated into the crown store so you get a cool 'hamster' pet or something as a token reward.

I've played many MMOs and the players in ESO have shown by far the most generosity, altruism and willingness to help. Guilds are often showing that they are all keen to work together to support the creation of expensive guild houses. I'm sure that with this kind of player base, many ESO fans would be up for supporting ZOS in this way, if the goal was clear and benefits noticeable.

@ZOS_GinaBruno @ZOS_MattFiror what do you think?

I think the hindrance on Cyrodil is more tied to the engine and netcode than it is the servers themselves.

Varana

I'm quite certain that lag during large-scale battles in Cyrodiil is not something that is (mainly) caused by server capacity. The problem with that, I think is the general code or programming of the game, and maybe connection speed, or other things.

Verbal_Earthworm

Been trying all day to login now,
the queue timer gets to 10 mins and stops,
forcing me to force-quit the game before trying again,
and again,
and again...

Happy Easter.

Cążki

PC_EU

Jaraal

Valdek wrote: »

I'd be very interested to know what kind of investment it would take to improve the servers to the point that the most intense areas (e.g. large scale fights in cyro) run smoothly. I imagine that it would be quite a lot, however, perhaps a crowdfunding campaign or something similar could provide a big injection. This could be integrated into the crown store so you get a cool 'hamster' pet or something as a token reward.

I've played many MMOs and the players in ESO have shown by far the most generosity, altruism and willingness to help. Guilds are often showing that they are all keen to work together to support the creation of expensive guild houses. I'm sure that with this kind of player base, many ESO fans would be up for supporting ZOS in this way, if the goal was clear and benefits noticeable.

@ZOS_GinaBruno @ZOS_MattFiror what do you think?

Wait, are you seriously suggesting a GoFundMe account for a multi-billion dollar corporation?



I've heard it all now.....

SydneyGrey

Your "leave queue" button doesn't work at all. It's completely broken.
Also, why do you do these things in the middle of events? Couldn't you have waited to "fix" stuff before or after the event? This is the third time you've done some major update in the middle of a festival or event that I've noticed.

Most_Awesome

20 MINUTE LOG IN QUE 30+ MINUTES CYRODIIL QUE FOR THE PACT COMPLETE ***

red_emu

So I get stuck in perpetual loading character...15 minutes I get kicked from game. Now it's a 48 minute queue. Why am I getting punished for character not loading?

Trueblue

This is funny... first queue 22 min and time drop 21, 20, 19... now 11:48 almost 20 min lock on 11:48... it's not funny. They give free game account for test and now all player with ESO Plus suffer.
Back to game from almost 2 year and still queue...

AH93

The queue is constantly getting stuck on 17/18 minute mark, when in reality I have been in the queue for 30-40min+.

This is not what I am paying for. Therefore, until this is fixed and this queue system removed, I will be unsubscribed and no longer spend any money with you.

LtBottle

22 minutes to 13 minutes.
Game client crash.
Back to 23 minutes.

This is what i call customer respect. Like a real free to play. Except you are not.
It really took you 4 months to realize there was a problem? your ingeneers are the problem.