ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

RexyCat · June 30

Where level up my Alchemy when I get kicked out with error 307, logged in to continue my levelling, then again it happened. This on PC EU around 9:20 in the morning, Sweden.

EDIT:

Forgot to add that when this happened I was kicked out from Craftstation at West Weald (PC could see craft station and the rest around instead of having UI for crafting), game looked like it had frozen, before I was kicked to log in screen with error message.

Xarc · June 30

PC-EU
Hi, i can't play anymore with this lag

Extreme lag spikes which send me back to login menu with error 103 if I want to come back

from 75ms to 999+ each 20sec, this since Update

I put off all my addons, it changes nothing.
No vpn, no download, only ESO is ON and I cant stay online more than 5min, this for many days now, it's unplayable.

Never had such problem since 2014.

JustLovely · June 30

Prime time on weekends is a good time to find something to do other than ESO now days.

manukartofanu · June 30

Photosniper89 wrote: »

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

Well, people started talking about problems in PvE precisely when we were announced this very server upgrade. Is that coinsidence?
The problems became more serious and even more people started experiencing them when, on our 10th anniversary, we received an update with faulty configuration. Coinsidence again?
I've been in the development field long enough not to believe in such coincidences.

SeaUnicorn · June 30

kyle.wilson wrote: »

Kappachi wrote: »

I've seen the complaint in zone chat, but every time I checked /latency it was fine. Sure it's not y'alls internet connection? your ISP could be having an issue even if it's just pinging one specific location, make sure it's whitelisted everywhere it can be and set all your ports properly/etc.

Do you pvp? there is 2 conversations that have been going on in this thread.
One is the slight problem of PVE lag, and the other is the massive problem in Cyrodiil lag.

PVE may be ISP for the ones seeing it most.
PVP is not. when I run ping tests to anything that doesn't belong to ZOS its usually around 12-28ms.

Its definitely not ISP issue for PVE. Issues happen in certain trial instances, one instance you are fine, everything is smooth, another instance comes with higher ping and lag spikes and potential DCs. It is definitely infrastructure issue that handles instancing. And PVP maps are also instances of sort, so you experience it constantly. Basically you spend more time in exposed tot he issue, but it does not necessarily mean the issue if different.

SeaUnicorn · June 30

Photosniper89 wrote: »

manukartofanu wrote: »

Photosniper89 wrote: »

Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....

I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).

I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.

Photosniper89 · June 30

Stridig wrote: »

kyle.wilson wrote: »

4 second lag spikes tonight PCNA Greyhost
@ZOS_RichLambert @ZOS_BrianWheeler

Had to just give up for the second night in a row. 300+ just standing at High Rock and 999+ anytime an actual fight was happening anywhere else. Stuck in combat most of the rest of the time.

I know I know.... waa, waa, waa

Unfortunately, this is about the same vibe most of my guildies have. They have slowly started moving to other games.

SeaUnicorn · June 30

manukartofanu wrote: »

Photosniper89 wrote: »

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

Well, people started talking about problems in PvE precisely when we were announced this very server upgrade. Is that coinsidence?
The problems became more serious and even more people started experiencing them when, on our 10th anniversary, we received an update with faulty configuration. Coinsidence again?
I've been in the development field long enough not to believe in such coincidences.

We actually had issues in PVE since forever, some were solved, others added. We had notion of "instance farming" where you would load into the trial and check the ping and how 1st trash pack behaves, if mobs were moonwalking or random spinning you would reset and try to get better instance. Teams would spend 70% of prog time instance farming during Harrowstorm DLC. Those were addressed by performance improvements. But the issue where 1 or 2 people on the team would experience random DCs and lag spikes just because of their routing were not resolved. People would solve that with VPN, basically try to find VPN server that goes through "good" route. This was the issue back then and seems to have become a more widespread now.

abakzn · June 30

I had these problems since the launcher update, so now I start the game without the launcher and the problems are gone for me.

SeaUnicorn · June 30

SeaUnicorn · June 30

Jaraal · June 30

SeaUnicorn wrote: »

Photosniper89 wrote: »

manukartofanu wrote: »

Photosniper89 wrote: »

Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....

I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).

I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.

Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.

https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1

He also mentioned that they would let us know when they had any success with the coding upgrade. And as we are still awaiting word on that, it's reasonable to assume that there's still much work to do.

SeaUnicorn · June 30

Jaraal wrote: »

SeaUnicorn wrote: »

Photosniper89 wrote: »

manukartofanu wrote: »

Photosniper89 wrote: »

Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....

I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).

I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.

Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.

https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1

that's a year old post. Multithreading is compute i.e. how game processes calculations. Those would be issues with skills not firing correctly, or not feeling "responsive", mob AI not behaving correctly, general lag in large populated areas. Those issues would cause everyone to experience lag. Issues reported here are network related, some people experience lag and can't stay connected, while others are perfectly fine.

React · June 30

PC NA, GH, 6/30 2:00PM EST. Currently seeing ping spiking up to 500+ for no reason whatsoever in the middle of an empty field riding across the map with not a single other player around, and with little to no action happening on the map.

Photosniper89 · June 30

React wrote: »

PC NA, GH, 6/30 2:00PM EST. Currently seeing ping spiking up to 500+ for no reason whatsoever in the middle of an empty field riding across the map with not a single other player around, and with little to no action happening on the map.

Current state of PvP 2024 edition. @ZOS_Kevin

SeaUnicorn · June 30

SeaUnicorn · June 30

Another DC just testing skills in my house

Jaraal · July 1

SeaUnicorn wrote: »

Jaraal wrote: »

SeaUnicorn wrote: »

Photosniper89 wrote: »

manukartofanu wrote: »

Photosniper89 wrote: »

Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....

I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).

I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.

Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.

https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1

that's a year old post. Multithreading is compute i.e. how game processes calculations. Those would be issues with skills not firing correctly, or not feeling "responsive", mob AI not behaving correctly, general lag in large populated areas. Those issues would cause everyone to experience lag. Issues reported here are network related, some people experience lag and can't stay connected, while others are perfectly fine.

If you are referring to the physical network of the game servers, then yes, they have mentioned having to replace failing network ports:

https://forums.elderscrollsonline.com/en/discussion/600966/update-33-pc-launch-postmortem/p1

It could also be related to the sharding, where players on certain shards are having the dysfunction, while those on other shards are having no issues. That could indicate some sort of hardware or software malfunction where the shards connect. It could also be packet loss or data bottlenecking at Akamai, or whatever other anti-DDoS scrubbing service they are currently using. Recent traceroutes have been showing a variety of IPs, whereas it was exclusively Akamai in the past.

blktauna · July 1

2 year old post. No further mention. While I get they don't want to say what's happening, I wish they just get it dealt with asap.

ElderSmitter · July 1

Barely have played eso the last few weeks. Not even logging in once a day anymore for 1 class scrap script. Lag has ruined this game for me and is over the top. PC NA and always around 200-350 range so when i force myself to log in for a class scrap script i do a writ and log out. CP 3200 + I have played a lot and never encountered Lag like this before EVER.!!

Photosniper89 · July 1

ElderSmitter wrote: »

Barely have played eso the last few weeks. Not even logging in once a day anymore for 1 class scrap script. Lag has ruined this game for me and is over the top. PC NA and always around 200-350 range so when i force myself to log in for a class scrap script i do a writ and log out. CP 3200 + I have played a lot and never encountered Lag like this before EVER.!!

If it makes you feel any better you're not alone.

ElderSmitter · July 1

Photosniper89 wrote: »

ElderSmitter wrote: »

Barely have played eso the last few weeks. Not even logging in once a day anymore for 1 class scrap script. Lag has ruined this game for me and is over the top. PC NA and always around 200-350 range so when i force myself to log in for a class scrap script i do a writ and log out. CP 3200 + I have played a lot and never encountered Lag like this before EVER.!!

If it makes you feel any better you're not alone.

I just filled my vendors for the first time in weeks. I am glad guild did not kick me out. I donate to guild weekly for raffles still. It took me forever to fill almost 60 slots. It was painstakingly slow.

Araneae6537 · July 1

I really hate crashing / getting kicked from the server in the midst of an IA run.

Of course, I can’t log back in in time because it thinks I’m still logged in and then I get that Error 103.

ElderSmitter · July 1

Kappachi wrote: »

I've seen the complaint in zone chat, but every time I checked /latency it was fine. Sure it's not y'alls internet connection? your ISP could be having an issue even if it's just pinging one specific location, make sure it's whitelisted everywhere it can be and set all your ports properly/etc.

I have ATT Fiber Optics and i did see somewhere they have been part of the ongoing attack so until this is resolved I am kind of screwed, I guess. I do not have issues elsewhere online or streaming. Just ESO. Very frustrating.

React · July 1

@ZOS_Kevin

PC/NA. 7/1 12:30AM EST. Imperial City and Grayhost.

Look at the delay on my light attacks in this video. This is happening on and off constantly throughout every day. It doesn't always correlate to the ping spiking, but as you can see from my ping in the bottom left hand side of the screen, it often does. The booting and rolling back that you'll see in the second clip has been happening frequently as well.

This is flat out unplayable. You cannot do any meaningful combat when there is a high chance everything is just going to stop working altogether. I haven't seen performance anywhere this bad since before the servers were replaced, and I am not exaggerating.

Just wanted to add in, this was my ESO experience tonight;

Sign in. Jump into IC, immediately get booted and rolled back upon entering a district. Queue out into cyrodiil, 2-3 second ability/weave delay and 600-999+ ping spikes without even leaving the home base. Sign out.

https://youtu.be/50wb-6nyzu4?si=b8HNxxbzYMGnj8UQ

abakzn · July 1

Try starting ESO directly, no launcher, the launcher is gobbling up a lot of resources after update.

edward_frigidhands · July 1

abakzn wrote: »

Try starting ESO directly, no launcher, the launcher is gobbling up a lot of resources after update.

These issues predate the launcher update and the last patch update as well.

vsrs_au · July 1

ElderSmitter wrote: »

Barely have played eso the last few weeks. Not even logging in once a day anymore for 1 class scrap script. Lag has ruined this game for me and is over the top. PC NA and always around 200-350 range so when i force myself to log in for a class scrap script i do a writ and log out. CP 3200 + I have played a lot and never encountered Lag like this before EVER.!!

Oh, how I wish my network ping was that low.

Anifaas · July 1

Jaraal wrote: »

Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.

https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1

He also mentioned that they would let us know when they had any success with the coding upgrade. And as we are still awaiting word on that, it's reasonable to assume that there's still much work to do.

It's like the video-game equivalent of living through a vivisection. We're all beta testers!

edward_frigidhands · July 1

Photosniper89 wrote: »

manukartofanu wrote: »

Photosniper89 wrote: »

Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....

I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).

I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.

They just upgraded the servers last year and we are right back to where we are. It's not the servers.

Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.

Hey. The server hardware upgrades/changes addressed the issues they were meant to.

We were told for the longest time that the performance issues were not tied to server hardware several times and the investment would yield no results.

However, once the server hardware upgrades happened there was a remarkable change in server performance which persisted for a very long time.

The investment of funds addressed the issues it was meant to, which were hardware related.

Server hardware is meant to be cycled out at the max every 3-5 years. There are competing theories that say the optimum period is 2-3 years.

Despite this information, ZOS waited since the launch of the game to do this and the national pandemic was blamed for the delay without acknowledging the fact that this very key maintenance was put off until the quality of server performance was nearly unplayable.

It is important for a business to do right by its customers and recognize and acknowledge missed opportunities like this in order to hold themselves accountable for future instances.

All of that being said, we don't know if this current issue is hardware related. Hardware can be faulty or poorly configured. Sometimes it is neither and just happens to be subject to unpredictable circumstances that shorten its lifespan or quality of output.

I purchased a PC in 2017 which ran until 2022 without issues. I purchased a second one in 2022 which ran into issues at the 11 month mark. I then purchased an alternative brand that has had no problems.

It is best for the possibility that it is a hardware issue to be entertained and explored in order to ensure all avenues for fixing the issue are assessed.

As customers it's probably not really our place to be getting into ZOS's cost of doing business. But I think it goes without saying throwing some kind of money at the issue will have to happen for there to be investigation and progress where there hasn't been for months.

The alternative may mean droves of people abandoning investing time and money into the game, which I don't think is what ZOS wants.