Update on PC NA & EU Issues [Updated 10/27 @ 8:40pm]

Numerikuu

Nemesis7884 wrote: »

stop praising them for doing the absolute bar minimum

Agreed. Sorry, but this has been going on for too long.

Tldr the EU server is an afterthought (and has been for years), has a lesser host/server compared to NA, and they're 'doing everything they can' (lol limiting their damn customers from entering the game/adding queues) except shelling out for a better server/host... GG I'm glad I unsubbed.

Kadoin

Rhaegar75 wrote: »

on the bright side: I spent the last hour in Cyro and worked like a charm!!!!

I saw you in EU...

FierceSam

Schattenfluegel wrote: »

FierceSam wrote: »

Schattenfluegel wrote: »

FierceSam wrote: »

DrOuttaSight wrote: »

Does this mean that ESO is a more popular game in the EU, than the US of A?

Short answer - yes

We have a bigger population, a larger landmass and more ESO players

If PC EU falls.. we’re coming to PC NA

And we are legion

Its not only EU, Russians, China, Japan and some other Non-EU Countrys are on our server, too..

Dude ... have some geography

Russia is a part of the European landmass (although because it’s really rather large, parts of Russia are in Asia)

We welcome players from all over the world, cos that’s how we roll.

Yeah but they are not a part from the European Union...EU means normally the Union not the Landmass. In the last case your going to say Europa.

Except in this case the EU in PC EU doesn’t mean European Union. It stands for European. (Or more bluntly Non-North American from a ZOS

Katheriah wrote: »

KyraCROgnon wrote: »

I'm feeling sorry for Gina, she's the one taking the direct heat while she can't do a thing about the problems. Like the receptionist at a plumbing company getting the angry call from people angry because of the plumber butchered the job or didn't show up , she not responsible for them not having hot water or a working toilet, but she still gets all the yelling..

Isn't that like... her job?

Ofc they're putting a nice person up front. For us she's still *the face of ZoS*.

What else are we going to do? Shout at some infra engineer who's name we don't know? His boss, who's name we don't know? His boss's boss?

Her job is community manager

Her job description does not include “being shouted at by rude people”.

You can be mad as hell, as I am, and still have the brains to understand that while she is passing on messages you might not like, she is not responsible for their contents.

You don’t shout at the postman when he delivers bills.

Have the guts to be a civilised person when you’re making your points. It gives them so much more weight.

daemonios

Schattenfluegel wrote: »

p_tsakirisb16_ESO wrote: »

Schattenfluegel wrote: »

These are the playernumbers from two recommend WoW-Servers (New Classic WoW). US has less than an half active players then EU. Yeah its not ZOS i know, but i think its similar to it. The most EU Players are on PC, not on consoles, like in US.

One thing to add the EU server doesn't have people from EU only.
Has quite a few from South America which have better connection to EU than NA. Has Asian players due to the peak time zone is closer to them also and of course African players.

Also the even base European population, is twice as big than USA - AUS - CAN put together which is the base population demographic for NA server. And that applies to all games.

You forget Russian, Asia for EU..

I have zero interest in discussing how many players or time zones are involved in PC-EU. That is immaterial at best, and at worst a distraction from the thing we need to discuss: why is PC-EU permanently operating near or past its stable capacity, so that every time more players want to play - whether because of new content, or an event, or simply at prime time - the server gives up the ghost?

This is what has been happening for years, this is what we've complained about for years, this is what ZOS promised it has addressed several times before, yet it keeps happening. Every. Single. Time.

Dread_Viking

daemonios wrote: »

Schattenfluegel wrote: »

p_tsakirisb16_ESO wrote: »

Schattenfluegel wrote: »

These are the playernumbers from two recommend WoW-Servers (New Classic WoW). US has less than an half active players then EU. Yeah its not ZOS i know, but i think its similar to it. The most EU Players are on PC, not on consoles, like in US.

One thing to add the EU server doesn't have people from EU only.
Has quite a few from South America which have better connection to EU than NA. Has Asian players due to the peak time zone is closer to them also and of course African players.

Also the even base European population, is twice as big than USA - AUS - CAN put together which is the base population demographic for NA server. And that applies to all games.

You forget Russian, Asia for EU..

I have zero interest in discussing how many players or time zones are involved in PC-EU. That is immaterial at best, and at worst a distraction from the thing we need to discuss: why is PC-EU permanently operating near or past its stable capacity, so that every time more players want to play - whether because of new content, or an event, or simply at prime time - the server gives up the ghost?

This is what has been happening for years, this is what we've complained about for years, this is what ZOS promised it has addressed several times before, yet it keeps happening. Every. Single. Time.

i agree it seems like we are more players EU so what is the problem in permanently upgrading the server so thy can handle events and Prime time

StarMightyMaster

ZOS_GinaBruno wrote: »

Update: To ensure that the game can run as smoothly as possible, we are going to disable the Activity Finder until we determine the root cause of the issues that are impacting game performance. As stated before, we’ll look into extending the Witches Festival, but first we need to identify and fix the problems that are impacting player experience.

For this change to take place, we'll need to reboot both PC NA and EU to stabilize everything, then turn off the Activity Finder; maintenance for this will begin around 3:30am EDT and should take an hour or two. Note that you'll still be able to form a group manually, but you will not be able to queue for Battlegrounds while the system is disabled.

We'll have another update for everyone on the situation later today (Sunday). Thanks again for your patience as we get to the bottom of this.

So base on what you said i guess this will not be fix today so the game will have no Activity Finder today and many +++ days any news on the update.

dagrdagaz_5912

After Gina's post about saturday evening/night's maintenance i seem to understand the main/root problem is somewhere in patch 24, not server capacity.
And increasing server capacity this time will not solve the problems.

The above does not mean they should not increase server capacity in general, or with events/festivals.

Heatnix90

Thanks ZOS.

Thanks for finally giving me the motivation to unsub from this dumpster fire of a game.

You cancelled Midyear Mayhem, one of two PvP events a year, for the sake of "performance improvements."

You didn't include an arena/mini-trial in Update 24 for the sake of "performance improvements."

Where are they?

ZOS_JesC

Greetings, we've removed a handful of comments that contained personal insults. This is a friendly reminder to remain constructive and civil when posting. It's fine to disagree with other forum members, but bashing and insults are never appropriate. Thank you.

Carbonised

Bethesda is quickly becoming the laughing stock of the entire gaming industry due to their handling of F76.

ZOS, as a sister company and subsidiary of Zenimax, might want to try avoiding the same fate with their handling of ESO.

Mitrenga

I haven't logged in since Thursday because of the server issues and wanted to wait and see if ZOS can fix this. Now, I'm here to see that nothing has changed.
Well, PvP servers' issues haven't been solved in 5 years and now servers all together went nuts.
Based on the previous work of ZOS, I'm 99% sure, PC EU will not be fixed. Therefore, I'm taking another break.
If ZOS fixes the game by Q1 2020 as they've promised, I will return. I have 1% hope though.
The Outer Wolds is keeping me busy, thanks to one of the most responsible devs out there.

Bam_Bam

Communication is everything and works wonders - thank you very very VERYmuch for the update @ZOS_GinaBruno

KyraCROgnon

Katheriah wrote: »

Isn't that like... her job?

Ofc they're putting a nice person up front. For us she's still *the face of ZoS*.

What else are we going to do? Shout at some infra engineer who's name we don't know? His boss, who's name we don't know? His boss's boss?

Yes it's her job, i still feel sorry for her. I doubt she was expecting that when she took the position. Dealing with never happy players after each game balance patch ? sure. But facing a wall of angry customers because they can't play at all , alone with nothing but the BS passed from the higher food chain ? That seems about as confortable a position as being the insurance company for the Titanic...

NicGroover

Peut-être qu'avec plus de moyens/serveurs les problèmes seraient en partie résolus.

Perhaps with more resources / servers the problems would be partly solved.

KyraCROgnon

tonne.backlinderb16_ESO wrote: »

Well I work within IT and since we've had this problem for so long and they reboot server and turn of this and turn on that but still we have these problems I'm pretty shure they have no clue how to solve this

Agreed. since it's working ok when server is half empty and it gets exponentially worst as the number of people online increase, i'd say something somewhere in their load-balancing algorithms there is a flaw. But one that cannot be easily replicated with "fake" traffic, like what you can do wit a test simulation. Reminds me of what was happening with network hubs before we moved to switches , where all trafic was replicated to all ports. Below 30% load, all is fine. Then you reach the critical mass, and suddendly it all collapses.
Changing some of the algorithms behing the game requires rewriting the stuff, and first you have to know what causes the pbm / where the flaw is.
Until you solve that, adding more servers / calculating power will not solve anything, just increase (and maybe only by a small %) the treshold where the system gets unstable.

Not saying Zenimax is not to blame, they most likely underestimated the stuff, told their IT team "what ? rewrite the core engine ? can't do that, all our coders are busy making new games exept the part time job that design crown items! We'll just tell our HW supplier to bump up our server size a notch and i'm sure that will do" ....

FrancisCrawford

ZOS_GinaBruno wrote: »

We thought we had fixed the problems that have plagued the last few big in-game events by introducing an updated LFG system, some server optimizations, etc., but obviously it wasn't enough. The updated LFG system appears to be working as it should, but it has exposed another underlying issue that is causing certain parts of our servers - especially our database clusters - to become overloaded and eventually unresponsive. This is why we put a cap on the number of concurrent players that can log in last Friday, and then published the patch this morning trying to alleviate that issue.

Classic case of Bottleneck Whack-A-Mole.

http://www.dbms2.com/2009/08/21/bottleneck-whack-a-mole/

Morgul667

Some things never change....

Nulami

Pyvos wrote: »

The_Saint wrote: »

@ZOS_GinaBruno
i think there is a time mistake. Banner and ingame message are correct with german time. but server doesnt go offline half an hour ago. this night we have -1 hour here. so server goes offline for maintenance in ~30min or?'

Daylight savings time ended at 0400, so clocks "went back an hour" already. The server is offline, went offline about 30 minutes ago.

Daylight Savings ends next Sunday in the US.

Junkriid

4/4/2020 will be 6 years and still same problems, just saying..

Nevaee

LFG or joining Dungeon Groups not working since yesterday anymore, no matter what twink, no matter what time of day

RefLiberty

Nevaee wrote: »

LFG or joining Dungeon Groups not working since yesterday anymore, no matter what twink, no matter what time of day

It is been switched off for now, you have the official statement few pages before.

StarMightyMaster

Any ETA on the update you said you was going to do today sometime nice for EU would be nice

LFG coming back online today i would guess not but one can hope right

Elsonso

Nemesis7884 wrote: »

stop praising them for doing the absolute bar minimum

Of course we should. Think of it like house training a dog. Think about how Bruce did it in Bruce Almighty. ZOS is very hard to house train, but we are making slow progress, and each time they show progress, we have to reinforce that behavior.

Edit: With dedication and effort, in 50 or 60 years, they will have real communication and customer support.

KallistaBlackheart

returning player here. love the game... hate the constant kicks and being unable to login for 15 or so minutes after said random kicks.

Pyr0xyrecuprotite

dagrdagaz_5912 wrote: »

After Gina's post about saturday evening/night's maintenance i seem to understand the main/root problem is somewhere in patch 24, not server capacity.
And increasing server capacity this time will not solve the problems.

The above does not mean they should not increase server capacity in general, or with events/festivals.

From other comments by ZoS, it sounds like there may actually be some software design problems. Throwing extra hardware at the issue MIGHT improve performance by 5-10%, but figuring out and fixing any software design problems can improve performance by an order of magnitude, or at least get rid of a major bottleneck or two. (In complex systems like this, the bottleneck could easily just move somewhere else - it's VERY hard to figure out in advance.) Complex, high-demand, multi-user systems are a nightmare to fully understand; it's not possible for one person any more, and teams can usually only try figure out solutions on individual parts. That said, I hope they figure something out soon...

Kudos and many thanks to @ZOS_GinaBruno and all the support folks that were working on this issue over the weekend. I hope the effort is rewarded.

stefan.gustavsonb16_ESO

DakhathKilrathi wrote: »

Okay, you do understand that they can't test something like activity finder until it's live, right? Sure, they can test if it works and they can even test a moderate load, but it's difficult to accurately replicate real world conditions.

This is part of software development. It happens. This is the first heavy activity weekend since this launched. They are at leas trying to make it better. I'm willing to give them the time to do that. Six days after launch isn't enough time.

It is becoming quite obvious that ZOS doesn't test their systems properly before deployment, but honestly, stress testing the internal data paths of a cluster system under simulated heavy external load is fundamental to success in the kind of software development represented by MMO games. Letting software go live without proper testing is a recipe for disaster, and we are watching the disaster unfold right now. There is no longer any excuse for such bad practices in large software projects.

There are useful and well established methods for stress testing large scale networked data-intensive software solutions without letting real end users take the hit like this. Sure, corner cases and unforeseen user behavior patterns can still catch you off guard, but by proper testing you can avoid most of the nasty surprises. Surprises like the one we are getting right now.

I am hoping for the developers to work out a solution, but I would really wish for them to have a better development model and not have to work nights and weekends to fix fundamental errors that could have, and should have, been caught in testing. It would make their life easier, and our game more enjoyable.

StarMightyMaster

So this update they was going give us will it be a time EU people are up or a time everyone in bed as it getting on now for EU time zone

btaddict2014

I'm sorry, but this is ridiculous.

So now as a low-level that I'm trying to level, my only option is to go sit over 1 world/delve boss for 10 minutes until the game decides to drop me another skull?

questing gives almost no skulls except possibly those rare end of map encounters
Dolmens are overrun, no loot from the final boss most of the time. -> make the final chest drop the skull instead of the boss?
can't run trials as level 20-something
can't run arenas as level 20-something
can't really run public dungeons alone at my current level.

So as far as I'm concerned, I'm already not doing the event. your solution to take away LFG fixes nothing.
I'd rather you had to do a hard reboot every few hours than this.

ZOS_GinaBruno

Thanks for being patient, everyone. We're going to be deploying a hotfix later today (no downtime required) that should fix the issue where server instability occurs when the Activity Finder gets overloaded. We will deploy this on PC NA around 6:00pm EDT, and will also turn the Activity Finder back on at that time. Note the exact time could shift depending on internal testing this afternoon, but will update everyone with the exact time when we know it. We will then plan to deploy the hotfix to PC EU once we have validated the hotfix on NA. Stay tuned for further details, and thanks again.

StarMightyMaster

ZOS_GinaBruno wrote: »

Thanks for being patient, everyone. We're going to be deploying a hotfix later today (no downtime required) that should fix the issue where server instability occurs when the Activity Finder gets overloaded. We will deploy this on PC NA around 6:00pm EDT, and will also turn the Activity Finder back on at that time. Note the exact time could shift depending on internal testing this afternoon, but will update everyone with the exact time when we know it. We will then plan to deploy the hotfix to PC EU once we have validated the hotfix on NA. Stay tuned for further details, and thanks again.

The NA become the test server for EU.

A hotfix with no downtime is a real hotfix