PC-EU Server Update - April 11

Tasear

😅 wipes sweat...NA could of been next...if not for EU speeding up things.

SoggyBizkit

Returned to ESO the last few weeks after a year break due to the poor server issues and major let down on the steam client. I was hyped for the next expansion which made me return but once again server problems which have plagued this game for too long have made my choice clear. I won’t invest money into something Zenimax won’t invest themselves.

chaz

ZOS_MattFiror wrote: »

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.



Matt

So, @ZOS_MattFiror , like I've been saying for years now, maybe now you will listen to me.

NOW IS THE TIME TO PLACE A NEW SERVER, right here in Honolulu. This will alleviate server taxation from EU and NA as we are right here in between them both.

Others keep throwing it in your face, key word BLIZZARD.

I'm telling you, it is best to place that server here in Honolulu for multiple reasons. I sometimes meet up with a lot of Hawaiians that are the original tournament champions and millionaire gamers and all agree they would love to see a new server out here and maybe even you host a million dollar PvP national tournament.

Silly2202

ZOS_MattFiror wrote: »

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

So, here is what we are doing to fix the problem:

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

Matt

I got a idee why not make more mega servers? With of course character transfer?

andreasv

@ZOSmarketing How about you ask the intern who developed this if he/she can make it so that you can browse the Crown Store while waiting in queue? You're welcome!

Mathius_Mordred

SaltySudd wrote: »

Unless there really hasn't been an influx in players and this is a tactic to convince people that ESO has a growing community in order to convince them to buy the game...



On a more serious note: It's nice seeing a post from Matt huzzah for communication!

Well I can tell you that after decisions that Star Trek Online have taken recently at least 15 of our group have come over to ESO and I understand many more from the game are also trying ESO. I understand there are some WoW refugees too, so this may well be correct.

red_emu

this is laughable. Typical strategy of customer service when things go wrong. Pulling wool over our eyes:

ZOS talks about servers, using big words and technical terms - ZOS forgets there are players who are experts in this field here.

ZOS talks about how difficult the situation is with upgrading the servers - ZOS forgets there are players who are experts in logistics and proper planning

ZOS talks about how expensive it all is and how impossible - ZOS forgets there are players here who are experts in marketing and corporate finance.


TL:DR - We're not all 16 year olds gamers who throw tantrums. You can't fool us with your textbook "Thank you for your patience, please hold. Here's some big words in the meantime"

KoultouraS

red_emu wrote: »

this is laughable. Typical strategy of customer service when things go wrong. Pulling wool over our eyes:

ZOS talks about servers, using big words and technical terms - ZOS forgets there are players who are experts in this field here.

ZOS talks about how difficult the situation is with upgrading the servers - ZOS forgets there are players who are experts in logistics and proper planning

ZOS talks about how expensive it all is and how impossible - ZOS forgets there are players here who are experts in marketing and corporate finance.


TL:DR - We're not all 16 year olds gamers who throw tantrums. You can't fool us with your textbook "Thank you for your patience, please hold. Here's some big words in the meantime"

What we are dealing here with is amateurism in its finest.
Amateurism in PR, amateurism in planing, in financing, in coding, amateurism in balancing a game via nerfs.
In almost everything besides CROWN STORE placement.
They got the best in monetizing department for sure.
I can't explain why we're still paying fot this game, if the latter doesn't do an extraordinary job.

On a side note , ZOS its just fuckingly extremely lucky they were trusted such a AAA+ title as ESO.
If it wasn't for the hype and memories this game creates *, it would have been doomed to eternal hiatus upon aborning .


* and extreme fun , whenever -those few moments in history- it works ok,

Mygalomorpea

I have delayed posting this since I thought it was only me being highly annoyed by the way the situation was handled... But I just can't shake the feeling that EU is only seeing a upgrade because someone convinced someone else that makes financial decisions that the same will happen to the pc na server at this rate.

If this is the case then pc eu will yet again be the first to suffer during any new user peak at a later stage because the capacity of the EU server will then never be greater than that of the NA server.

Also - there is the claim that the EU only works over 2 major time zones, if they want a more wide spread activity roster, why haven't they invested more in any region other than those time zones?

Why is there no effort in place to make Scandinavian users feel more welcome? There is a LOT of them. Then also the further out laying countries like Iraq, Iran, Russia, Madagascar, Italy, South Africa... There is players from all those places playing the game on the EU server. If they actually felt appreciated then you would see more people logging in outside of the German and British prime times to do content. Many of them are currently playing at later times than they would prefer because there is not enough people around when they would like to do stuff like progression trials. (I do not like having to trials from 9pm to midnight and then getting up again at 5am for work, but I have to do so to do vet trials)

There is other solutions that could have been implemented years ago already that would have prevented this problem. If only the higher ups cared about the growth of the EU server...

Turned out as a rant after all...

Alienoutlaw

would be nice if the good paying "fans" were compensated in someway after 4 month of horrendous service

profundidob16_ESO

@ZOS_MattFiror

Thanks for the update. I feel I must add that after a queue of 10minutes last night Cyrodill was pretty much unplayable (where it normally is always ok for me) because of lag.

So you might have to reduce the maximum allowed number of players even further (even if that meanslonger queues)

Itzmichi

Less talk more action please.

NoTimeToWait

Thanks for the response, greatly appreciate the honest communication here. It is good to know that EU PC server is moving towards better performance.

Though, I would say, it doesn't help much at this moment. At this moment I am NOT enjoying the game unfortunately. I am looking forward to start enjoying TESO again though.

Zorgon_The_Revenged

I've not really had many problems since January.

Then again, I've greatly reduced my play time since January due to the often disgusting server performance over the previous 18 months.

I don't believe the "World is flat" claims, but I look forward to the day when all the performance upgrades make a lasting improvement.

pod88kk

Is there a way of better detecting bot accounts to cut down on the the amount of people logging in at one time therefore bringing down the queue tie for everyone. For example on Xbox most of these bot accounts usually only have a gamerscore of between 5 and 20 whereas normal players would have gamerscores in the, 000's . Is there a way you could analyse the player's gamer score before they log in?

Salvas_Aren

The best way would be to send gamemasters who inspect the sites where they farm.

If they behave suspicious, spawn a WB unto them, talk to them or force them to log out, if there is proof they repeat their breach of ToS, 3 day ban and then perma-ban.

Would be about 12 wages more to pay for 6 servers.

CleymenZero

ZOS_MattFiror wrote: »

Everyone,

After seeing the commentary about the queuing system being enabled on the PC EU servers yesterday, I thought I'd let everyone know what has been leading up to this, and what we are doing to alleviate the situation.

Please note that the situation that led to the queuing system being enabled is not related to the overall client performance issues that we talked about at the end of last year - those engine optimizations are still underway and hopefully you'll see them in-game at some point this year. The situation that led to queuing has nothing to do with client performance - it is because our servers are overloaded.

Starting in January we've had an influx of new players - we've have had some of our highest weekly average user, daily active user, and peak concurrent user numbers in the last six weeks since we launched on console back in 2015. So, it was in January when EU players started noticing degraded server performance. We made tweaks and updates to the service to free up extra database resources and other boosts to performance, which is why you have been seeing more and longer maintenance outages than normal in the last couple of months.

However, in late March, we really saw a huge spike in the number of users logging in, which led to too many concurrent users for the platform to effectively support. As a result, players on PC EU have seen a severe degradation in service. This degradation started in January, but has greatly accelerated in the last week - lag spikes, disconnects, inability to zone from instance to instance, weird interactions with LFG, etc. These problems are greatly exacerbated by having serious load on the servers, as there are too many users for the hardware to handle.

A quick aside: these problems affect the PC EU more than the other megaservers because it has our highest peak concurrency of any of our six megaservers. The reason: the EU is a highly PC-centric gaming market - and also because almost all of our EU players are located in two time zones. Compared to the North American servers, where there are four time zones to spread out the "peak playing time", the EU has almost all its users logged in at the same time, which leads to very high peak usage.

So, here is what we are doing to fix the problem:

• Short term, we are turning on our queuing system (as you saw yesterday), which will cap the maximum number of players allowed on the PC EU megaserver. You should only see the queue at peak times, when demand is higher than the hardware supports. As was posted yesterday, the estimated time to login may not be 100% accurate, but it will give you an idea of how big the line is. ALSO: if you are disconnected from the server for any reason, you should have a twenty minute grace period to log back in and "skip" the queue.
• Longer term: we are spinning up additional hardware in both our NA and EU datacenters in order to support our growing user base. We expect the time frame for this to be about two weeks, but we will keep you posted. And yes, we are adding more capacity to PC NA as well as EU because we are rapidly approaching the same problem in NA and want to get ahead of it. We will also be looking very closely at our Console megaservers to make sure they have enough capacity.

I know it is frustrating to be stuck in queue, but it is better to do this than have the entire service be unstable and frustrating for everyone. We will do everything we can to get the new hardware online as quickly as possible and stop queuing and have the service return to normal. Stay tuned for details.

Matt

I think being transparent like this is just going to help. Although it seems that the problem isn't as great on NA, the NA server network issues are still problematic.

Thank you for your candor and best wishes to get the solutions up and running ASAP.

Cheers,
Zero

DrOuttaSight

@ZOS_MattFiror

Please please double/treble the server expansion capacity of what you have now,
so later (within the next 5yrs),
you don't need to resort to a queuing system again in the future,

as an every day player of ESO, it really does kill the gaming experience.

Uryel

Acrolas wrote: »

Latin America and Asia are developing markets.

No it's not. As I mentionned previously, historically Asia has always been one of the strongest market in MMO history. Probably still is, but their taste for grindfest rather than storytelling sets them appart in terms of marketing opportunities from a western company.

I do have to agree on the protection of intellectual property you mentionned, though. Especially with China. This country is a nightmare when it comes to corporate espionage. Again, been there, done that, back in the days...

profundidob16_ESO

How about adding extra server realms that are only accessible to ESO+ people ? We pay alot of yearly money after all through subcription, expansions and extra crown stuff...

Uryel

chaz wrote: »

NOW IS THE TIME TO PLACE A NEW SERVER, right here in Honolulu. This will alleviate server taxation from EU and NA as we are right here in between them both

Pray tell, how is Honolulu located between US and Europe when it's in the pacific ocean ? It would make sense for an oceania / asian server, though. Say, how good are the lines in Hawaii ? Just technical curiosity, I wonder how a datacenter would fare on a pacific island.

Uryel

KoultouraS wrote: »

In almost everything besides CROWN STORE placement.

Sorry mate, amateurism in that too. I mean, a single per character outfit slot priced at 1500 crowns, full price, is amateurism. They obviously never heard of profit by volume of sales and would rather use discouraging prices. At this price, I would expect an account-wide unlock.

I personnaly wouldn't purchase a per character slot at this price, even though I have spent quite a hefty buck on that same item in Champions Online. But they are prices 20 times cheaper, so the percieved value is way different.

Since those things have no production costs, they should be priced to appeal to a large amount of players. I understand "elite" items prices higher, such as, say, manor houses... I don't like it, but I understand it. But basic convenience ? Surely there is a profit to be made if everyone can purchase a few and be happy with it rather than wait for some discount or even refuse to buy on the principle of "it's not worth it if it's this expensive".

So... Yeah, amateurism even in the crown store.

Verbal_Earthworm

profundidob16_ESO wrote: »

How about adding extra server realms that are only accessible to ESO+ people ? We pay alot of yearly money after all through subcription, expansions and extra crown stuff...

Check out my thread to see the reaction to such a suggestion:

https://forums.elderscrollsonline.com/en/discussion/465509/dedicated-server-for-eso-subscribers/p1

profundidob16_ESO

Verbal_Earthworm wrote: »

profundidob16_ESO wrote: »

How about adding extra server realms that are only accessible to ESO+ people ? We pay alot of yearly money after all through subcription, expansions and extra crown stuff...

Check out my thread to see the reaction to such a suggestion:

https://forums.elderscrollsonline.com/en/discussion/465509/dedicated-server-for-eso-subscribers/p1

oh boy...wow...just wow

there are some valid arguments in there for sure but the bulk of responses is indeed -as can be expected from the masses- simply beyond ridiculous. I guess I should be glad I'm not lynched by now =P

UrQuan

Uryel wrote: »

KoultouraS wrote: »

In almost everything besides CROWN STORE placement.

Sorry mate, amateurism in that too. I mean, a single per character outfit slot priced at 1500 crowns, full price, is amateurism. They obviously never heard of profit by volume of sales and would rather use discouraging prices. At this price, I would expect an account-wide unlock.

I personnaly wouldn't purchase a per character slot at this price, even though I have spent quite a hefty buck on that same item in Champions Online. But they are prices 20 times cheaper, so the percieved value is way different.

Since those things have no production costs, they should be priced to appeal to a large amount of players. I understand "elite" items prices higher, such as, say, manor houses... I don't like it, but I understand it. But basic convenience ? Surely there is a profit to be made if everyone can purchase a few and be happy with it rather than wait for some discount or even refuse to buy on the principle of "it's not worth it if it's this expensive".

So... Yeah, amateurism even in the crown store.

When it comes to Crown store items, from a business perspective the best strategy is to have multiple different strategies for different types of items.

1. DLCs should be relatively cheap or otherwise easily accessible (like through ESO+) because you want a significantly large portion of the player base to have access to them. More things to do & a less fragmented player base (ie. fewer cases of "Oh I want to group up with my friends to do XYZ, but they don't all have the DLC") leads to better long-term engagement and keeps people playing the game.
2. Convenience items and services should be relatively cheap if they're not one-time-only purchases, as you ideally want them priced cheaply enough that people will buy them without spending too much time considering whether it's worth the price. If they're one-time-only purchases (like the any race any alliance upgrade, or the warden class, or the imperial race) then you want to price them somewhat higher, but still low enough that a large portion of the player base will consider it to be worth the price.
3. Cosmetic items should be priced all over the map. You want some priced cheaply enough that people will impulse buy them even if they only kind of like them, you want others priced moderately so that if they appeal to a given player that player is more likely than not to consider it to be worth the price, and you want others priced too high for most players to consider them to be worth buying because they will appeal to some whales (and to some whales the very fact that they're expensive, and the idea of relative exclusivity will be what attracts them to those items).

The per character outfit slots fit into the 2nd category, and are not one-time-only purchases, so I agree with you that they've missed the mark on the pricing for them. I think they'd make a lot more money off outfit slots if they were priced around 500 crowns.

888and888

It is amazing to see how many people believe what the Devs say.

We have been promised better Cyrodiil performance for years and we have been promised better Vendor performance with the new database, maybe we have been told something else that I don’t remember.
Now a queue system has just been slapped on paying customers and more promises have been made.

What about providing, for once, some solid facts and evidence, instead of empty words?

Thanks.

red_emu

UrQuan wrote: »

Uryel wrote: »

KoultouraS wrote: »

In almost everything besides CROWN STORE placement.

Sorry mate, amateurism in that too. I mean, a single per character outfit slot priced at 1500 crowns, full price, is amateurism. They obviously never heard of profit by volume of sales and would rather use discouraging prices. At this price, I would expect an account-wide unlock.

I personnaly wouldn't purchase a per character slot at this price, even though I have spent quite a hefty buck on that same item in Champions Online. But they are prices 20 times cheaper, so the percieved value is way different.

Since those things have no production costs, they should be priced to appeal to a large amount of players. I understand "elite" items prices higher, such as, say, manor houses... I don't like it, but I understand it. But basic convenience ? Surely there is a profit to be made if everyone can purchase a few and be happy with it rather than wait for some discount or even refuse to buy on the principle of "it's not worth it if it's this expensive".

So... Yeah, amateurism even in the crown store.

When it comes to Crown store items, from a business perspective the best strategy is to have multiple different strategies for different types of items.

1. DLCs should be relatively cheap or otherwise easily accessible (like through ESO+) because you want a significantly large portion of the player base to have access to them. More things to do & a less fragmented player base (ie. fewer cases of "Oh I want to group up with my friends to do XYZ, but they don't all have the DLC") leads to better long-term engagement and keeps people playing the game.
2. Convenience items and services should be relatively cheap if they're not one-time-only purchases, as you ideally want them priced cheaply enough that people will buy them without spending too much time considering whether it's worth the price. If they're one-time-only purchases (like the any race any alliance upgrade, or the warden class, or the imperial race) then you want to price them somewhat higher, but still low enough that a large portion of the player base will consider it to be worth the price.
3. Cosmetic items should be priced all over the map. You want some priced cheaply enough that people will impulse buy them even if they only kind of like them, you want others priced moderately so that if they appeal to a given player that player is more likely than not to consider it to be worth the price, and you want others priced too high for most players to consider them to be worth buying because they will appeal to some whales (and to some whales the very fact that they're expensive, and the idea of relative exclusivity will be what attracts them to those items).

The per character outfit slots fit into the 2nd category, and are not one-time-only purchases, so I agree with you that they've missed the mark on the pricing for them. I think they'd make a lot more money off outfit slots if they were priced around 500 crowns.

This thread is now a 'Let's teach ZOS 101 of running a business'

Love it

moonsugar66

“We’re spinning up additional hardware... “ ZOS has got the OPA making artificial gravity. Does that mean Ceres sized server for EU? We can hope.

Jaraal

More like six new cassette drives for the Commodore 64 server array!

Scootter

I play on NA and EU. If EU was the last server I played on the next time I log in I go directly into the queue and if I click leave to try to select NA it automatically bumps me back to the EU queue.