PC-EU Server Update - April 11

me_ming

I really hope they DON'T extend the event, it's already quit dragging as it is. Can they just increase the number of tickets per day instead, so people still get the same number of tickets as expected?

Zer0oo

Since it was just the EU server ignore it till we see also that the NA server gets close to the limit?


The EU server performance was really bad since ages and was reported on the forums quite often but complete ignored. But now we got some news about this and it sound just like the big reason there will be a server upgrade is that the NA server will also get problems soon.

UrQuan

The Uninvited wrote: »

I call fake news. Remember when half the US server came over to EU and vice versa because we could win a villa? Performance was fine then and all those players were in 1 zone...

Do you think all the NA players logged on to the EU server during EU prime time? And all the EU players logged on to the NA server during NA prime time (which, as Matt pointed out, is spread across more time zones than EU prime time is, so if the player bases are about the same size NA is going to have a lower peak player count)?

The number of players logging on during a given day isn't the issue. It's the peak number of concurrent players that's the issue, and that's where you'll see server strain: if the peak concurrent players starts getting higher and higher (which it has over the last few months).

Check out Steamcharts:
https://steamcharts.com/app/306130
The highest number of concurrent players ever for ESO on Steam (which is only a fraction of the PC player base, but should be representative) was 31,851, which just happened on March 31st - less than 2 weeks ago. The peak number of concurrent players for ESO on Steam during the month when everyone was logging on to both servers for the villa (Sep 2018) was 22,288.

Miswar

FleetwoodSmack wrote: »

As a PC-NA person, I decided to see what this hubbub was all about.

I'm extremely disappointed in the handling of PC-EU. For years they've been treated like second class players and for years they've been the scapegoat during crappy maintenance times. Not only have you ( @ZOS_MattFiror ) have ignored the trends that catered toward ESO, you also ignored following the global markets in terms of how your product has sold. You claim that all of your staff reads the forums, but this post here blatantly tells us that you haven't been and that's just disappointing. PC-EU deserve better treatment, especially considering that this is (again) happening around a major ESO event.


For those affected (and who are tired of it), had I not seen it for myself, I'd have brushed it off. This is ridiculous.

..add the fact that console server performance is a utter joke most of the time and you have bingo.

Sad thing is that mr Firor seems to indicate that console servers are functioning "normally" which is a bad joke.

I have not used eso+ for over 6 months now and encourage others to the same until ZoS actually start improving their worsening game performance. Words are just words and until that miracle happens that they actually do improve performance not dropping another coin for this game.

KoultouraS

UrQuan wrote: »

The Uninvited wrote: »

I call fake news. Remember when half the US server came over to EU and vice versa because we could win a villa? Performance was fine then and all those players were in 1 zone...

Do you think all the NA players logged on to the EU server during EU prime time? And all the EU players logged on to the NA server during NA prime time (which, as Matt pointed out, is spread across more time zones than EU prime time is, so if the player bases are about the same size NA is going to have a lower peak player count)?

The number of players logging on during a given day isn't the issue. It's the peak number of concurrent players that's the issue, and that's where you'll see server strain: if the peak concurrent players starts getting higher and higher (which it has over the last few months).

Check out Steamcharts:
https://steamcharts.com/app/306130
The highest number of concurrent players ever for ESO on Steam (which is only a fraction of the PC player base, but should be representative) was 31,851, which just happened on March 31st - less than 2 weeks ago. The peak number of concurrent players for ESO on Steam during the month when everyone was logging on to both servers for the villa (Sep 2018) was 22,288.

What I see is a continuously increased influx since December 2017.
Mind to interpret that too?

The Uninvited

KoultouraS wrote: »

UrQuan wrote: »

The Uninvited wrote: »

I call fake news. Remember when half the US server came over to EU and vice versa because we could win a villa? Performance was fine then and all those players were in 1 zone...

Do you think all the NA players logged on to the EU server during EU prime time? And all the EU players logged on to the NA server during NA prime time (which, as Matt pointed out, is spread across more time zones than EU prime time is, so if the player bases are about the same size NA is going to have a lower peak player count)?

The number of players logging on during a given day isn't the issue. It's the peak number of concurrent players that's the issue, and that's where you'll see server strain: if the peak concurrent players starts getting higher and higher (which it has over the last few months).

Check out Steamcharts:
https://steamcharts.com/app/306130
The highest number of concurrent players ever for ESO on Steam (which is only a fraction of the PC player base, but should be representative) was 31,851, which just happened on March 31st - less than 2 weeks ago. The peak number of concurrent players for ESO on Steam during the month when everyone was logging on to both servers for the villa (Sep 2018) was 22,288.

What I see is a continuously increased influx since December 2017.
Mind to interpret that too?

Exactly!

Varana

First, it's nice to hear that there are more players than ever.

Second, it's nice to finally get an acknowledgement that there even is an issue. The silence about any group finder hiccoughs, lag spikes, and similar problems has been deafening for months now.

Third, it's also nice to hear that at last, something is being done about the problem.

But.

This didn't start in January. We had massive group finder problems during the Undaunted event, and even back then there was that infuriatingly stupid message (from Rich Lambert, iirc) that you thought the group finder had been fixed. Obviously not, as even a short look into your effing bug forum would've told you. GF on PC/EU has been acting up at least since Wolfhunter or Murkmire.
And now this. Either you're lying, or group finder is broken just generally, or you have no clue.

ManwithBeard9

Varana wrote: »

First, it's nice to hear that there are more players than ever.

Second, it's nice to finally get an acknowledgement that there even is an issue. The silence about any group finder hiccoughs, lag spikes, and similar problems has been deafening for months now.

Third, it's also nice to hear that at last, something is being done about the problem.

But.

This didn't start in January. We had massive group finder problems during the Undaunted event, and even back then there was that infuriatingly stupid message (from Rich Lambert, iirc) that you thought the group finder had been fixed. Obviously not, as even a short look into your effing bug forum would've told you. GF on PC/EU has been acting up at least since Wolfhunter or Murkmire.
And now this. Either you're lying, or group finder is broken just generally, or you have no clue.

GF is just generally broken, they stated that after the Undaunted event. They said they wouldnt have another event that required the use of it until it was properly fixed.

NeillMcAttack

Lol at the reference to the console servers. We all know they are gonna be fine.......

UrQuan

KoultouraS wrote: »

UrQuan wrote: »

The Uninvited wrote: »

I call fake news. Remember when half the US server came over to EU and vice versa because we could win a villa? Performance was fine then and all those players were in 1 zone...

Do you think all the NA players logged on to the EU server during EU prime time? And all the EU players logged on to the NA server during NA prime time (which, as Matt pointed out, is spread across more time zones than EU prime time is, so if the player bases are about the same size NA is going to have a lower peak player count)?

The number of players logging on during a given day isn't the issue. It's the peak number of concurrent players that's the issue, and that's where you'll see server strain: if the peak concurrent players starts getting higher and higher (which it has over the last few months).

Check out Steamcharts:
https://steamcharts.com/app/306130
The highest number of concurrent players ever for ESO on Steam (which is only a fraction of the PC player base, but should be representative) was 31,851, which just happened on March 31st - less than 2 weeks ago. The peak number of concurrent players for ESO on Steam during the month when everyone was logging on to both servers for the villa (Sep 2018) was 22,288.

What I see is a continuously increased influx since December 2017.
Mind to interpret that too?

It's not really a continuous increase since then - it's been relatively up and down, with Summer 2018 being particularly high (June & August peak concurrent player numbers were very high, while July's peak concurrent players was much lower, but still higher than average - obviously Summerset is responsible for that summer surge), but there are also months in that time frame when the number of peak players was significantly lower. It is a general upward trend over that time, but this is the first time there have been 4 consecutive months of growth in the Steam numbers.

Ideally ZOS would have anticipated this situation based on the general upward trend over the last year or so, but I'd imagine that those dips in concurrent players that you see some months would have made them think twice about whether they were really going to see sustained long-term growth.

KoultouraS

You do understand that steam players are not the majority of Eso users?
Also you do understand that during the summer months a lot leave their residence on vacation?
Right?
What would you make of it ,if less than 1/2 of a population is increasing rapidly whereas there should be more of the same , having some ups and downs?
Since december the stats show increasing influx during prime times. All except some few cases.
If that's not enough to warrant investment into better equipment, I dont know what is.

On a side note I bet the stats Zos had , regarding influx of non steam users is definitely against your argument as for why they hesitated all these years..

UrQuan

KoultouraS wrote: »

You do understand that steam players are not the majority of Eso users?

You do understand that I explicitly said that right?

UrQuan wrote: »

The highest number of concurrent players ever for ESO on Steam (which is only a fraction of the PC player base, but should be representative) was 31,851, which just happened on March 31st - less than 2 weeks ago.

It's still a representative slice of the player population, and although we can't use the numbers to say what the actual peak concurrent players were on any given server (especially as Steam numbers don't separate them between NA and EU), the sample size is big enough that the ups and downs should be pretty damn consistent with what will be seen in the overall concurrent players.

TheTwistedRune

The queueing system is just crafty way of dissuading people to log in. If you are only able to play an hour a night, and have to wait 20 minutes to log in, how many people are just not going to bother?

What bothers me most is they’re almost giving out the impression it’s the players fault. “Yea, too many of you are logging in you see, you are causing issues.”

It’s really shocking to me that a company that already has a very rocky reputation when it comes to server performance could handle a problem they have known is coming in such a dismissive and ham fisted way. For a large developer of one of the largest MMORPG’s this is simply not good enough.

Nobody is saying ZOS have to be perfect or that the game won’t have issues, but it’s your handling and the number of said issues and lack of prompt communication that constantly lets down EU players and draws their ire.

ESO is fortunate that no other new big western MMORPG’s are around, as I’d wager their overloaded server problems would vanish pretty quickly....

Korah_Eaglecry

Acrolas wrote: »

Korah_Eaglecry wrote: »

Short answer: It wasnt impacting their bottom line.

Long answer: They would rather wait until things get ugly and have to go with extreme measures rather than spend money up front to get ahead of an issue.

The real answer is that budgets aren't approved overnight, particularly ones dependent on supporting evidence, and work orders aren't immediate because funding and labor just don't work like that. That the expected turnaround time is only two weeks is evidence that they're not making this *** up as they go along.

These inconveniences will be forgotten in a month. But it's the Internet and people will do what they normally do in the comments section.

No ones claiming that a budget needs to be approved overnight. But budgets arent as tightly sealed up as you make it out to be. Depending on what is broken, not working or damaged. Its very likely the company has a list of things that have almost immediate releases of funds for repairs and replacements. Id wager a company that relies entirely on an online service has funds they can release if something was to go sideways with that service. As their means of making a profit would be absolutely impacted if they couldnt do this.

Matt states in his letter that they knew the servers were experiencing issues due to traffic as early as January. They had 2+ months to address the issue, if they are being honest about when they noticed the increased traffic, and request funding for an expansion of their server farm.

Im not putting this on Matt or any of the developers at ZOS. Its very likely the people in charge of said funding waited until the last minute to do anything and the developers had no other choice but to place a queue on log-in to hold things over til things can get fixed. And now that the issue of server load has now come to pass on the EU server, they have enough "proof" to go ahead with getting the NA servers expanded.

sh_shatter

EU players are a tiny bit more than 2 time zones considering Asia, Russia and some of ANZ folks are also playing on EU server

Caligamy_ESO

Inaccurate minute estimates are pretty frustrating, perhaps if the login queue was more like Cyrodil where it told you your actual place in line it would be less of a pain.

MashmalloMan

@ZOS_MattFiror Just wanted to say thank you for the transparency. It's refreshing. Please don't take upset comments as a reason to stop, this is the type of changes your community like to see, good or bad, keep us posted.

Puupi

Nice!

Mojmir

As much of a *** i am too the devs. I understand what matt is going through. *** basically has to go this far south before his bosses will lift a finger and drop coin to improve.

valeriiya

So we have @ZOS_MattFiror saying they knew about the issue but let it get worse. We have @ZOS_GinaBruno saying a new communication plan is coming...since November, no rush, and
@ZOS_RobGarrett saying "a majority of the bugs and issues faced are not ones everyone encounters or are drastically halting gameplay, as such they need to continue creating content for people who want new content to play and don’t play at the higher level to notice if a skill or mechanic works a little wonky."

ZO$

Recremen

Great to hear that the game is getting more popular to the point that we even need more infrastructure to handle the increased demand, and I'm glad that steps are being taken to address the problem. One question though, and I don't mean this to come off as too controversial, though it probably will, so I apologize in advance. Is there any estimate for how many accounts are just bot farmers of one type or another, and is there anything in the works to address the existing botting problems across all platforms? It just sucks that some of this server loud is probably driven by people with nefarious intentions, and I hope it doesn't account for too much of the increase.

Marto

Keep in mind guys, server infrastructure in NA (specially once huge corporations like Google and Amazon are involved) is just more advanced and readily available.

Setting up a server in Europe or upgrading an existing one there is just not as easy.

Add to that things like timezone differences, language differences, and the fact that Bethesda has employees and studios in Texas (Where the NA servers likely are) while they don't have any employees in the UK or Germany... and of course EU is going to take a while longer.

BlueViolet

Thanks for giving us a more in depth explanation of what's going on, I'm sure many appreciate that, but I have to admit, I'm really disappointed about how things have been handled.

Why is it that after so many months of people reporting incidents of server degradation, does it have to come to the point that a login queue has to be turned on because things were allowed to go this far without seemingly any investigation? It's almost like the EU and its players are just an afterthought.

It is also sad that this kind of thing has happened during a major event. An event that I hope you will consider extending, for all the people who are missing out on playtime because of this debacle.

FleetwoodSmack

valeriiya wrote: »

So we have @ZOS_MattFiror saying they knew about the issue but let it get worse. We have @ZOS_GinaBruno saying a new communication plan is coming...since November, no rush, and
@ZOS_RobGarrett saying "a majority of the bugs and issues faced are not ones everyone encounters or are drastically halting gameplay, as such they need to continue creating content for people who want new content to play and don’t play at the higher level to notice if a skill or mechanic works a little wonky."

ZO$

This comment is so much win right now.

Uryel

Thank you, @ZOS_MattFiror , for FINALLY acknowledging the problem. The past few months have been very frustrating for us players as we have been repeatedly notifying you guys of the server issues, only to apparently be ignored. It seems you were actually well aware of the issue, but, as per usual with this company, you didn't say anything.

Proper communication goes a long way in customer relations. Even when all you have to say is "we are aware of the issue and are trying to fix it, sorry for the inconvenience". It's WAY better than saying nothing at all. Your kind of communication, or lack thereof, can really cause your customers to become really angry. Which you probably noticed, since our reports were more and more hostile as time went on without any official word. And threads on the matter being removed from the boards certainly didn't help.

As of today, many folks feel like you tried to sneak that one under the rug, and therefore will have a hard time trusting you. It's basically the same with pretty much every issue or perceived issue : the lack of of communication will anger the customers, and that may have a negative impact on your business.

It's been the same with the racial changes in Update 21, by the way : we are STILL waiting for official statements on why some changes, obviously made only to improving some PvP experience and without any PvE use whatsoever, where pushed to live against the lore and against common sense (yes, I'm talking about the Bosmers again). What I'm saying is, you guys really, really need to learn to communicate more with your community. Or, in this company's case, communicate at all, but it seems you're getting to it, and for that, I thank you. Let's hope the lesson has been learnt, but as I mentioned, people might have a hard time trusting you now, and I'll wait and see. Maybe you are working on your communication, maybe it's just a one time thing. Let's hope it's the former.

There are still a few points I'd like to adress in your statement.

ZOS_MattFiror wrote: »

We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

So, what you are saying is, you MacGyvered the existing hardware to avoid spending cash on expanding it.

Been there, done that. I used to be in IT, specifically in sytem and networks aministration. When the executives refuse to spend a penny, you're f****ed. I once had to convince my boss to spend 5 000 euros on a new server because running on the backup one only incured the risk to lose 80+ millions euros worth of sales and delivery datas. It took the whole afternoon. Stubborn executives are the worse.

Still, it took me an afternoon to convince my boss that losing 80+ millions was a risk we couldn't afford, and I thought that is was VERY long a time to achieve a proper result. It took you guys 3-and-a-half months. What the heck took you so long ?

ZOS_MattFiror wrote: »

these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time"

Actually, Europe has 4 time zones. Eastern Europe is in one, western Europe in another, United Kingdom and Portugal are in a third, and Iceland is in its own time zone. Granted, it's not the small number of people from Iceland that will cause the biggest strain on the servers, but still, Europe has 4 time zones.

Also, it seems a non negligible amount of players from Asia play on the EU server. Which would spread alot the number of time zones the server covers. And if you include players from the Russian federation, then we're talking about TEN to TWELVE time zones. With 10 time zones you'd expect that peak concurrent connexions would be spread more evenly.



I assume you guys know what you're talking about when counting connexions and whatnot, but the major bias here is limiting the EU server to actual geographic Europe. I don't know how many players from Africa there are in game, but if there are any (pretty sure you'll find some from the Maghreb area, at least), they won't play on US server. Asian players might be spread between US and EU, so load should be expected. Seems to me that you guys didn't make a proper market study, there, because it sure feels like you had no idea how many people would actually play the game.

Which, in turn, led many EU player to believe you guys didn't give a skeever's bottom about us "second rate" players from the EU server, due to your absolute lack of communication. At any rate, late is better than never, but please try to learn from that and do communicate more in the future.

Also, why isn't there an asian server ? Historically, if you look back to the pre-World of Warcraft era, Asia was the main market for MMOs. Before WoW, the most successful MMOs were Lineage 1 and 2, which, at a time when the very concept of paying a monthly fee to play a game on that weird thing called "internet" was still outlandish for many occidental cultures, has respectively over 1 and 2 millions active users. The number may seem small by today's standards, but back then, a successful MMO in the western world would have had a few hundred thousands. Sure, preferences vary greatly and you guys might have considered your game wouldn't appeal to the asian market. Maybe it doesn't, and people from Asia are only marginally represented, I wouldn't know. Geographically speaking, the only asians we can be sure of are the russians (and yes, anything east side of the Oural mountains is in Asia, so the major part of the russian land mass is in Asia). But I find it weird that a MMO company would entirely neglect the historically strongest market in their field.


Anyway, thank you again for keeping us informed, now and hopefully in the future.

Androconium

ZOS_MattFiror wrote: »

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

So, here is what we are doing to fix the problem:

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

Matt

Well, we're all really sorry that you were caught unawares by the fact that your current marketing of ESO was so successful. By current, I mean the 2019 promotions to attract new players, after the exodus of long term players. Too bad then that you did this before you had capacity to cope.

As stated elsewhere, this problem has been identified by players for the past 24 months at least. There are no 'Fix the Server' posts currently in the General Discussion forum for two reasons:

• The long-term players that were concerned with this ongoing issue have LEFT IN DISGUST.
• Any remaining posts are being moved to other topic headings, where less people will see them.

So your interim solution is to limit the number of players that can logon at once? So this is now mMO (minorly multiplayer online) and not big-M MMO? How proud you must be.

Somewhere in your organisation, someone elected to run the marketing agenda against the advice from the technical teams.
Agile development philosophy suggests that would be the Product Owner; or the Senior Producer. It's like that they are the same person.

When we see the firing of this incompetent individual, we'll know that you are serious about treating your customers with the respect that paying customers deserve. I have spent several thousand dollars on the base game, DLC, and Crown Store purchases.

Regardless of what you end up doing, I will not buy another Bethesda-badged product. I'm not insane.

I bought my first new car in 1984, it was a General Motors product. The overall experience of dealing with that company, despite however good the car itself was, was enough that I have never bought another GM product in 35 years.

I am, however, still telling the story.

James-Wayne

I've been playing on EU recently due to the speed of my ping on NA and its much much better then NA, perhaps while you are getting more servers ready you can drop one in down under to Australia!

Cheers mate!

Uryel

Marto wrote: »

Keep in mind guys, server infrastructure in NA (specially once huge corporations like Google and Amazon are involved) is just more advanced and readily available.

Setting up a server in Europe or upgrading an existing one there is just not as easy.

I call Ox-droppings on that. I used to admin a corporate server farm in a company that had subsidiaries all over the world, we had about 200 servers that needed to be available 24/7 from USA, Europe, China and whatnot, and that was back in 2007.

What you describe is the late 90's.

Banana

Excellent. Ill be hoping for that 280 ping I had at launch

Acrolas

Uryel wrote: »

Actually, Europe has 4 time zones. Eastern Europe is in one, western Europe in another, United Kingdom and Portugal are in a third, and Iceland is in its own time zone. Granted, it's not the small number of people from Iceland that will cause the biggest strain on the servers, but still, Europe has 4 time zones.

The quote was "almost all of our EU players are located in two time zones".
Likely specifically referring to Western European Time and Central European Time.

ZOS_MattFiror wrote: »

Asian players might be spread between US and EU, so load should be expected. Seems to me that you guys didn't make a proper market study, there, because it sure feels like you had no idea how many people would actually play the game.

Latin America and Asia are developing markets. Most people would acknowledge that the opportunities are theoretically there. But there's also a lot of financial risk and far less protection for intellectual property. It's not about just playing the game. There are so many more hard decisions involved.