SeaUnicorn wrote: »Photosniper89 wrote: »manukartofanu wrote: »Photosniper89 wrote: »Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....
I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).
I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.
They just upgraded the servers last year and we are right back to where we are. It's not the servers.
Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.
you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.
Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.
https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1
He also mentioned that they would let us know when they had any success with the coding upgrade. And as we are still awaiting word on that, it's reasonable to assume that there's still much work to do.
edward_frigidhands wrote: »SeaUnicorn wrote: »Photosniper89 wrote: »manukartofanu wrote: »Photosniper89 wrote: »Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....
I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).
I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.
They just upgraded the servers last year and we are right back to where we are. It's not the servers.
Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.
you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.
Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.
https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1
He also mentioned that they would let us know when they had any success with the coding upgrade. And as we are still awaiting word on that, it's reasonable to assume that there's still much work to do.
That is from a while back. There is a specific set of issues that began and worsened during May 2024 and have been ongoing since.
Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability.
Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability.
drip_fromtheinkwell wrote: »it's been a minute since it happened again, but did anyone else figure out what to do in a "booted from server" > "unexpected internal error" loop? ,_,
edward_frigidhands wrote: »SeaUnicorn wrote: »Photosniper89 wrote: »manukartofanu wrote: »Photosniper89 wrote: »Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....
I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).
I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.
They just upgraded the servers last year and we are right back to where we are. It's not the servers.
Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.
you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.
Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.
https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1
He also mentioned that they would let us know when they had any success with the coding upgrade. And as we are still awaiting word on that, it's reasonable to assume that there's still much work to do.
That is from a while back. There is a specific set of issues that began and worsened during May 2024 and have been ongoing since.
The point is that as they are tweaking and changing things, sometimes things get worse. We expect the changes behind the scenes to make things better, but that's not always the case.Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability.
Bammlschwamml wrote: »Well if you tweak your apple pie while the oven is on, you will end up with a pile of goo and burn yourself in the process...
SeaUnicorn wrote: »SeaUnicorn wrote: »Photosniper89 wrote: »manukartofanu wrote: »Photosniper89 wrote: »Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....
I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).
I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.
They just upgraded the servers last year and we are right back to where we are. It's not the servers.
Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.
you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.
Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.
https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1
that's a year old post. Multithreading is compute i.e. how game processes calculations. Those would be issues with skills not firing correctly, or not feeling "responsive", mob AI not behaving correctly, general lag in large populated areas. Those issues would cause everyone to experience lag. Issues reported here are network related, some people experience lag and can't stay connected, while others are perfectly fine.
If you are referring to the physical network of the game servers, then yes, they have mentioned having to replace failing network ports:
https://forums.elderscrollsonline.com/en/discussion/600966/update-33-pc-launch-postmortem/p1
It could also be related to the sharding, where players on certain shards are having the dysfunction, while those on other shards are having no issues. That could indicate some sort of hardware or software malfunction where the shards connect. It could also be packet loss or data bottlenecking at Akamai, or whatever other anti-DDoS scrubbing service they are currently using. Recent traceroutes have been showing a variety of IPs, whereas it was exclusively Akamai in the past.
SeaUnicorn wrote: »edward_frigidhands wrote: »SeaUnicorn wrote: »Photosniper89 wrote: »manukartofanu wrote: »Photosniper89 wrote: »Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....
I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).
I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.
They just upgraded the servers last year and we are right back to where we are. It's not the servers.
Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.
you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.
Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.
https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1
He also mentioned that they would let us know when they had any success with the coding upgrade. And as we are still awaiting word on that, it's reasonable to assume that there's still much work to do.
That is from a while back. There is a specific set of issues that began and worsened during May 2024 and have been ongoing since.
The point is that as they are tweaking and changing things, sometimes things get worse. We expect the changes behind the scenes to make things better, but that's not always the case.Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability.
Game developers are tweaking the game engine, it has nothing to do with actual hardware, especially network. You don't just "tweak" drivers and software of network components, those come from the manufacturer, maintained by manufacturer, and updates of those are released by manufacturer. ZOS engineers are only responsible for patching those on timely manner. In majority of cases that software is proprietary all together. What your saying is equivalent of "I am tweaking the recipe of my apple pie, and because of it my oven does not work anymore"
Well, when it comes to the game.... I seriously struggle to not see it as a joke at this point. Thought I'd log on just to keep up the daily rewards and immediately got the unable to connect to server error. Tried entering my password three or four times and just gave up because clearly ESO doesn't want me playing it. And you know what game doesn't randomly boot me or just not let me in at all? Many, but specifically my one of choice lately is RuneScape. So.... I know where I'm going for the evening/night.edward_frigidhands wrote: »I hope to see how things look today once I am done with work.
SeaUnicorn wrote: »SeaUnicorn wrote: »SeaUnicorn wrote: »Photosniper89 wrote: »manukartofanu wrote: »Photosniper89 wrote: »Okay - thats kind of what I thought happened but wasn't sure. Disappointing. I'm assuming the decision was made that it's not financially worth it for them to invest in refactoring the game but....
I'd challenge anyone in a leadership role in the company to factor in technical debt they have because they refuse to refactor the code. This is a conversation that hassssssss to happen (most leaders don't even know what technical debt is so it's never even discussed).
I don't think it's a matter of technical debt. You can refactor endlessly, but when the issue is with the servers, it doesn't solve anything. If the problem were in the code, everyone would be experiencing exact same issues. But we see that only a portion of the players are affected simultaniously. This kind of suggests that the questions should be directed not at the programmers, but at the administrators of the whole system.
They just upgraded the servers last year and we are right back to where we are. It's not the servers.
Now - are the servers configured improperly some how... sure that is possible. But throwing money and hardware at the issue didn't fix it.
you do realize one could cheap out on hardware and purchase not enough or of questionable quality. In addition, there are other components of infrastructure besides the servers. New compute does not solve issues with the network. Routers and switches handle network traffic and therefore have direct impact on the ping. Few faulty routers means some people experience issues because their traffic gets routed through those, while remaining population does not experience issue because their traffic gets routed through functional ones. In addition there are network components that connect servers themselves, one faulty router there means certain areas that the server behind faulty router would experience issues.
Mr Firor himself said that the ongoing multithreading/code rewrite was causing instability. So it's not necessarily just a hardware issue.
https://forums.elderscrollsonline.com/en/discussion/637691/server-multithreading-update-july-2023/p1
that's a year old post. Multithreading is compute i.e. how game processes calculations. Those would be issues with skills not firing correctly, or not feeling "responsive", mob AI not behaving correctly, general lag in large populated areas. Those issues would cause everyone to experience lag. Issues reported here are network related, some people experience lag and can't stay connected, while others are perfectly fine.
If you are referring to the physical network of the game servers, then yes, they have mentioned having to replace failing network ports:
https://forums.elderscrollsonline.com/en/discussion/600966/update-33-pc-launch-postmortem/p1
It could also be related to the sharding, where players on certain shards are having the dysfunction, while those on other shards are having no issues. That could indicate some sort of hardware or software malfunction where the shards connect. It could also be packet loss or data bottlenecking at Akamai, or whatever other anti-DDoS scrubbing service they are currently using. Recent traceroutes have been showing a variety of IPs, whereas it was exclusively Akamai in the past.
Shards are storage drives that are spread across servers and are connected into single DB via network. Your character data like achieves, outfits, inventory is stored there. Realtime events of you moving, firing skills, applying buffs de-buffs are not persisted in the database, because its unreasonable amount of unnecessary data and databases are not designed to process such amounts of data. Such real-time "streams" of data are processed in memory on compute instances and then summaries of what states of characters and the world those actions resulted in are sent to the database. So the skill not firing and all that does not even involve interaction with the shards. It's processed on the compute instances. If there was a miss-communication between compute and store everyone would experience something like rollbacks, or corruption of data, such as items missing, achieves not applying and so on. I am sure ZOS pays ALOT of attention to that part, as this has direct monetary impact and is probably ranked as high inherent risk item. We only really seen major issues with data corruption may be twice in history of ESO: once was straight up PTS was connected to live, so it was saving onto live DB (just a human error, pretty easy to make too, just copy-paste one wrong config line), and there was this one time when servers rolled back by few minutes so people lost some store purchases. Normally company has log summaries of important stuff like purchases, so there would be tools to address that too. In any case issues with shards would be a major problem and we def don't want to experience those.
Current issues we are experiencing and reporting are limited exposure (i.e. some people but not others), hence no pressure to fix them immediately. It's all just simple product management and issue prioritization at this point. But more voices make those issue raise higher on the priority list. So we should keep reporting them.
What does not help is discounting and dismissing people's experiences. Issues are happening. There is no explanation from dev/engineering team regarding the root cause. There are indication of what those issues might be. Posts of 2023 are not an explanation of the root cause of current issue.
Launcher is the problem.
Playing directly from eso64.exe = normal ping.
Just tried it and works fine.
@ZOS_GinaBruno
edward_frigidhands wrote: »I am wondering if the server restart helped or it was just that I was playing at a low population hour and didn't have the time to enter cyrodill.
What server/platform do you play on?
What is the nature of the issues you started experiencing or noticed got worse after the beginning of May 2024? (ex. did characters rubberband/warp, did abilities get delayed/fail to fire, etc.)
Around what time of day have you experienced the new and/or worse issues reported in this thread?
Where in-game have you experienced these issues? Please be as specific as possible (name/area of overland zone, delve, dungeon, Trial, etc.)
Have you experienced these issues outside of combat situations (in houses, for example)
Have you noticed these issues are worse during any specific times of day? If so, what times?
Have you noticed any changes/improvements as of today?
PCNAWhat server/platform do you play on?
Rubberbanding, block not blocking, player abilities are failing to fire, new enemy casts are starting after enemy deathsWhat is the nature of the issues you started experiencing or noticed got worse after the beginning of May 2024? (ex. did characters rubberband/warp, did abilities get delayed/fail to fire, etc.)
Occurs at all times of day.Around what time of day have you experienced the new and/or worse issues reported in this thread?
lucent citadel, blackrose prison, the cauldron, graven deep, banished cells, fungal grotto, sanity's edge, rockgrove, sunspire, southern elsweyr, northern elsweyr, craglorn, apocrypha, the rift, eastmarchWhere in-game have you experienced these issues? Please be as specific as possible (name/area of overland zone, delve, dungeon, Trial, etc.)
yes, multiple inputs are sometimes required to open storage, interact with NPCs in houses, as well as open chests after combat has ended. the worst of which is at the end of Lucent Citadel. Character will cancel crouch animation 10+ times in a row before actually opening chestHave you experienced these issues outside of combat situations (in houses, for example)
issues don't seem to be worse at any given time of dayHave you noticed these issues are worse during any specific times of day? If so, what times?
noHave you noticed any changes/improvements as of today?
PC NAWhat server/platform do you play on?
Around June/chatper launch the load from login to character select started to take an extremely long time. Once in game, load screens went back to normal.What is the nature of the issues you started experiencing or noticed got worse after the beginning of May 2024? (ex. did characters rubberband/warp, did abilities get delayed/fail to fire, etc.)
My sleep schedule is erratic and I've been on at various times, time of day does not seem to affect anything related to the issues on my end.Around what time of day have you experienced the new and/or worse issues reported in this thread?
Initial login lag is literally the first login to characters select to in game world. Hopping characters afterwards, hopping zones, etc are fine as long as I don't fully log back out.Where in-game have you experienced these issues? Please be as specific as possible (name/area of overland zone, delve, dungeon, Trial, etc.)
Doesn't seem better or worse in or outside of houses, in or outside of trials, etc. Just a persistent lag.Have you experienced these issues outside of combat situations (in houses, for example)
Nothing noticeably better or worse time of day wise.Have you noticed these issues are worse during any specific times of day? If so, what times?
The borderline unplayable lag started after Monday's maintenance for me. The character login is not better either.Have you noticed any changes/improvements as of today?
Hi all. After chatting with the team, we're looking to gather more information from players who have been experiencing new and/or worse latency/lag spikes/rubber-banding issues in ESO since the beginning of May 2024 (when the emergent issues in this thread were initially reported.) To help us with our ongoing investigation, could you please answer the following questions?
- What server/platform do you play on?
- What is the nature of the issues you started experiencing or noticed got worse after the beginning of May 2024? (ex. did characters rubberband/warp, did abilities get delayed/fail to fire, etc.)
- Around what time of day have you experienced the new and/or worse issues reported in this thread?
- Where in-game have you experienced these issues? Please be as specific as possible (name/area of overland zone, delve, dungeon, Trial, etc.)
- Have you experienced these issues outside of combat situations (in houses, for example)
- Have you noticed these issues are worse during any specific times of day? If so, what times?
- Have you noticed any changes/improvements as of today?
Hi all. After chatting with the team, we're looking to gather more information from players who have been experiencing new and/or worse latency/lag spikes/rubber-banding issues in ESO since the beginning of May 2024 (when the emergent issues in this thread were initially reported.) To help us with our ongoing investigation, could you please answer the following questions?
- What server/platform do you play on?
- What is the nature of the issues you started experiencing or noticed got worse after the beginning of May 2024? (ex. did characters rubberband/warp, did abilities get delayed/fail to fire, etc.)
- Around what time of day have you experienced the new and/or worse issues reported in this thread?
- Where in-game have you experienced these issues? Please be as specific as possible (name/area of overland zone, delve, dungeon, Trial, etc.)
- Have you experienced these issues outside of combat situations (in houses, for example)
- Have you noticed these issues are worse during any specific times of day? If so, what times?
- Have you noticed any changes/improvements as of today?