ESO Performance and Lag - Technical Discussion

hrothbern · March 2016

KhajitFurTrader wrote: »

hrothbern wrote: »

This is why you should never use TCP for networking time-critical data!

While everything being said in that article applies well to online first person shooters, it doesn't necessarily do so to MMOs -- because they are not time critical in the sense that they need to provide small, guaranteed maximum answer/turnaround times (e.g., like a real-time system does). MMOs, by definition, are not "twitchy" games -- ESO with its outstanding combat system might be a bit different, but reaction window times to combat indicators are well above multiple times of expectable network latency.

When years ago I first heard that WoW internally runs with a delay of 500 ms calculated in, I was baffled. How can everything seem to be so immediate, so smooth, or look so synchronised, when everything you see in your viewport "happens" with more than a half second difference? The answer is: the client is cheating, a lot! When you press [W], your character immediately starts moving forward, but the server might only know about it at half the current roundtrip time plus an arbitrary amount of processing time, which will depend on zone load, later. Only then will it broadcast your movement to others within visible vicinity, and again, at half of their roundtrip time later, they will see you start moving. Global synchronicity within half a second is precise enough for MMOs, and a tradeoff between needed computational power, i.e. costs, and continuity for players -- only when message transmission times, caused either by delayed messages due to server load, or by high network latency, are becoming greater than the internally calculated delay, the system starts to break down. In the case of moving, the server calculates where you have been 500 ms ago and where you should be now, within reasonable boundaries. If the next status message from the client contains an implausible vector (position + direction + speed), the server overrules the client and your position is reset. Rubberbanding or slingshotting are the visible effects of this.

So, MMOs are not time-critical, or real-time online applications, although through the use of trickery, they very much try to look like them.

Thanks again @KhajitFurTrader ,

So we get the illusion of a higher responsiveness than that is actually there, just like our brain does with the data our eyes supply in effect also cheating us.
Must be a thrilling job to work on this topic by the involved ZOS people.

Another question, if you allow me:
Is it so that this time window is super critical for network latency.
Or in other words: is it so that if a smaller number of players have network latency issues, that the issues arising cause rapidly more server load, that on her turn rapidly spreads the issue to many more players

If something like that would happen, you could stay in control by having cut-off algorithms in place, cutting off players with highest latency (moving historical record), to protect the server load from reaching critical values.
Then you contain the issue to the connections (players) causing the issue, but you never break completely down.

And if too many connections/players are switched off, you freeze and roll back.

Malmai · March 2016

KhajitFurTrader wrote: »

RinaldoGandolphi wrote: »

.

The 1st step to fixing the lag issues with ESO is to re-write the netcode using RUDP(Reliable UDP). This will then allow ZOS to write their own latency control issues into the custom protocol that would kinda make it a mix between Layer 4(UDP) with custom code built on top of it to handle these issues(Layer 7) functionality.

Useful References
https://1024monkeys.wordpress.com/
https://1024monkeys.wordpress.com/2014/04/08/udp-vs-tcp-a-follow-up/

Ok, first off, I miss a link to a relevant article from the author mentioned above: https://1024monkeys.wordpress.com/2014/04/01/game-servers-udp-vs-tcp/

Second, trashing and rewriting the entire netcode (which needs to be done simultaneously both on client- and server-side, naturally) might be a very expensive option in terms of both programming and testing time and resources. The fundamental design decision to go with TCP has been made a long time ago, and the corresponding code has been tested over years in internal, closed, open, and stress test betas. It may simply not be affordable to conduct even a partial test schedule in a live environment, while the PTS does not even come close to live concurrency numbers, let alone stress test numbers. An operation on such a basic component of the code base without proper testing beforehand would be a high risk endeavor without guarantee of improvement -- after all, we don't know for sure whether the main cause for server-side lag is, e.g., packet loss due to network saturation (which UDP wouldn't change, btw), or high server-side response times due to load on script engines or RDBMS. The latter case could be solved easier and cheaper with optimizations, an ongoing process that has already begun.

Besides, the consoles would be affected as well, since they share the same, common code base, meaning that the whole approvement process on both PSN and Xbox Live would have to be repeated.

All in all, while of course improvements to server/network performance are important, the question should be asked whether they are feasible. Changing network protocols this late in the lifetime of the game might just not be, but none of us are privy to internal evaluations of the company.

U just said this game is done... A community manager and writing this horse pie LOL

Morozov · March 2016

hrothbern · March 2016

Malmai wrote: »

KhajitFurTrader wrote: »

RinaldoGandolphi wrote: »

.

The 1st step to fixing the lag issues with ESO is to re-write the netcode using RUDP(Reliable UDP). This will then allow ZOS to write their own latency control issues into the custom protocol that would kinda make it a mix between Layer 4(UDP) with custom code built on top of it to handle these issues(Layer 7) functionality.

Useful References
https://1024monkeys.wordpress.com/
https://1024monkeys.wordpress.com/2014/04/08/udp-vs-tcp-a-follow-up/

Ok, first off, I miss a link to a relevant article from the author mentioned above: https://1024monkeys.wordpress.com/2014/04/01/game-servers-udp-vs-tcp/

Second, trashing and rewriting the entire netcode (which needs to be done simultaneously both on client- and server-side, naturally) might be a very expensive option in terms of both programming and testing time and resources. The fundamental design decision to go with TCP has been made a long time ago, and the corresponding code has been tested over years in internal, closed, open, and stress test betas. It may simply not be affordable to conduct even a partial test schedule in a live environment, while the PTS does not even come close to live concurrency numbers, let alone stress test numbers. An operation on such a basic component of the code base without proper testing beforehand would be a high risk endeavor without guarantee of improvement -- after all, we don't know for sure whether the main cause for server-side lag is, e.g., packet loss due to network saturation (which UDP wouldn't change, btw), or high server-side response times due to load on script engines or RDBMS. The latter case could be solved easier and cheaper with optimizations, an ongoing process that has already begun.

Besides, the consoles would be affected as well, since they share the same, common code base, meaning that the whole approvement process on both PSN and Xbox Live would have to be repeated.

All in all, while of course improvements to server/network performance are important, the question should be asked whether they are feasible. Changing network protocols this late in the lifetime of the game might just not be, but none of us are privy to internal evaluations of the company.

U just said this game is done... A community manager and writing this horse pie LOL

Not everybody is a smart as you @Malmai

that they can shoot instant solutions from the hip

KhajitFurTrader · March 2016

Malmai wrote: »

U just said this game is done... A community manager and writing this horse pie LOL

I am not, nor have ever been, a community manager. Please read up on the community ambassador program here.
Could you kindly lay out to me where I said the game is "done", when I didn't? I made the assessment, to the best of my knowledge of the matter at hand, that switching core communication protocols at this point in time would hardly be feasible because the amount of testing needed for such a feat is barely affordable in a live environment. I also made the assessment that this kind of change would hardly be necessary, because the supposed advantages over the status quo are questionable, and performance improvements might also be achieved by other means, without having to rewrite major portions of core code components. Which, incidentally, also makes sense from a business/HR point of view.

Enraged_Tiki_Torch · March 2016

Excellent post. There are pros and cons to both. I am sure when the server code was written, there was a discussion of which protocol would be best for the game. Although UDP is faster and has less overhead, TCP is more reliable especially with large data transmission. This game sends a lot of data and even though packet loss sucks, it might suck even more if packets were not sent/received at all. Say you were about to die, and you wanted to cast a heal on yourself; sorry but that packet was lost and you died. Also "ghosting" is an issue with UDP.

People complain about Lag but I have been playing Online games for quite a while and haven't found one lag-free yet.

WalkingLegacy · March 2016

You gotta pick the necessary evil.

Yes, we might lose some packets here and there with UDP (reliable UDP is pretty reliable - enough so for gaming)

But, that heal you're trying to get off....not going to help you when your packets are waiting in line to execute your heal and roll Dodge but the enemy Proxy det happens first.

I'd pick rUDP over their TCP/IP any day of the week for gaming. Especially with how performance hogging their anti hack bot is.

Removing everything but players from Cyrodiil won't fix the issues either. Torchbugs aren't the issue. Players spamming skill after skill after skill is network extensive when it is all handled server side. Need network speeds to match how the game was designed.

hrothbern · March 2016

Enraged_Tiki_Torch wrote: »

Excellent post. There are pros and cons to both. I am sure when the server code was written, there was a discussion of which protocol would be best for the game. Although UDP is faster and has less overhead, TCP is more reliable especially with large data transmission. This game sends a lot of data and even though packet loss sucks, it might suck even more if packets were not sent/received at all. Say you were about to die, and you wanted to cast a heal on yourself; sorry but that packet was lost and you died. Also "ghosting" is an issue with UDP.

People complain about Lag but I have been playing Online games for quite a while and haven't found one lag-free yet.

, it might suck even more if packets were not sent/received at all.

I cleaned my inventory issues today by sending a lot of stuff to my second account.

My stomach would really turn if data integrity transmission would not be guaranteed for that stuff.

Enraged_Tiki_Torch · March 2016

Yeah, it really is pick your poison here.

With TCP, your heal will still sync with the server. The game will lag but everything is still done in order. If proxy would of killed you, it would of done so with or w/o the lag. Nobody gets to fast forward their gameplay with either solution.

Enraged_Tiki_Torch · March 2016

I still feel the first priority is to improve the performance of the Server and how it handles the game. It all starts with lowering the stress of the servers. After that, UDP and TCP can be re-evaluated based which will provide the best results after the core issues are fixed.

Elsonso · March 2016

Enraged_Tiki_Torch wrote: »

I still feel the first priority is to improve the performance of the Server and how it handles the game. It all starts with lowering the stress of the servers. After that, UDP and TCP can be re-evaluated based which will provide the best results after the core issues are fixed.

Yup.

Rune_Relic · March 2016

Merlight wrote: »

RinaldoGandolphi wrote: »

I think the 1st step towards fixing ESO lag issues lies in the netcode and its reliance on TCP. I think a custom RUDP(Reliable UDP protocol) implementation would go a long way in reducing server overhead and making the game lag significantly less in high stress situations. TCP simply isn't the best choice for a game the scale of ESO and this becomes woefully apparent on the PVP side of the game.

I agree that UDP might help, but abandoning TCP might not be as straightforward as it seems. I don't know ESO's application protocol, though, so I'll give you an example from a game using TCP whose application protocol I do know.

In Lineage 2, communication with the game server in both directions is encrypted using a very simple, fast and insecure stream cipher -- it's just a XOR with a short key (server's encryption key equals client's decryption key, and vice versa), and each sent / received byte modifies one byte of the corresponding key. I guess they modify the key to make it harder for an adversary to tackle with the data stream -- but it also rules out dropping packets, as everything that was encrypted on the server must be decrypted on the client to keep their keys in sync. A lost packet means decryption of the next received packet will use stale key, which inevitably leads to disconnect as it produces garbage.

Was it a poor choice for an MMO? Well, their servers were able to handle thousands of players in 2006 -- there were no megaservers, no instances, only 2000+ players on one physical server -- and 100+ people fighting over a castle wasn't uncommon. To be honest, there was not that much AoE back then, but heck it's been 10 years ago, and the map was as large as Cyrodiil...

Agreed with this and OP to a greater extent.
Will add more details of a fix for some players later.
I have removed my +999 latency issues.
This will be a good place to put it after the OP has gone out of his way to explain everything within.

hrothbern · March 2016

Rune_Relic wrote: »

Merlight wrote: »

RinaldoGandolphi wrote: »

I think the 1st step towards fixing ESO lag issues lies in the netcode and its reliance on TCP. I think a custom RUDP(Reliable UDP protocol) implementation would go a long way in reducing server overhead and making the game lag significantly less in high stress situations. TCP simply isn't the best choice for a game the scale of ESO and this becomes woefully apparent on the PVP side of the game.

I agree that UDP might help, but abandoning TCP might not be as straightforward as it seems. I don't know ESO's application protocol, though, so I'll give you an example from a game using TCP whose application protocol I do know.

In Lineage 2, communication with the game server in both directions is encrypted using a very simple, fast and insecure stream cipher -- it's just a XOR with a short key (server's encryption key equals client's decryption key, and vice versa), and each sent / received byte modifies one byte of the corresponding key. I guess they modify the key to make it harder for an adversary to tackle with the data stream -- but it also rules out dropping packets, as everything that was encrypted on the server must be decrypted on the client to keep their keys in sync. A lost packet means decryption of the next received packet will use stale key, which inevitably leads to disconnect as it produces garbage.

Was it a poor choice for an MMO? Well, their servers were able to handle thousands of players in 2006 -- there were no megaservers, no instances, only 2000+ players on one physical server -- and 100+ people fighting over a castle wasn't uncommon. To be honest, there was not that much AoE back then, but heck it's been 10 years ago, and the map was as large as Cyrodiil...

Agreed with this and OP to a greater extent.
Will add more details of a fix for some players later.
I have removed my +999 latency issues.
This will be a good place to put it after the OP has gone out of his way to explain everything within.

@Rune_Relic ,

You mean self help to diminish latency issue from your own PC ?

that would be awesome

The only thing I could think of in that respect was using the cable instead of Wi-Fi

Perhaps also some Add-ons.

Rune_Relic · March 2016

INCOMING WALL OF TEXT.
WARNING - ANYTHING YOU ATTEMPT AS A RESULT OF USING THIS POST IS DONE AT YOUR OWN RISK.
PLAYING WITH NVRAM IN YOUR ROUTER CAN KILL IT.
YOU ARE ADVISED TO USE THIS POST FOR INFORMATION ONLY.

Ok. so heres the background. I like to do 50/50 pvp, pve. I am no pro player, just mediochre. Regardless: When jumping between PvE shrines I would get +999 for upto a minute.
If I was in PvE and getting PvP scroll reports on screen, my latency would be +999
If it was prime time, my latency would be up and down like a yoyo.
In pvp... keep takes +999 etc.
Rezzing at a keep would be instant solo but have an endless loading screen in a group.
You get the picture and many of you will be familiar no doubt.
My nominal quiet latency is around 70.

The OP and several of the posters have done an excellent job explaining the technicalities of TCP and its issues as a guaranteed delivery service rather than best effort UDP.
So not much point adding to that.
I will focus on the practical effects it has as a demonstration.
This will go into some depth on the problems with microsofts TCP stack windowing, netgear routers, wan_mtu size, widows 10 issues and IP fragmentation.

I am running netgear A6210 USB AC wifi to a netgear D6200 ADSL router.
The most evil of evil network gaming setups in the eyes of many.
I connect in the UK to SkyUK using this non-standard adsl modem.
SkyUK uses PPPoATM rather than PPPoEthernet.
There is no fibre in my location, but it wont help with this particular issue anyway.

My website loading would be choppy, sometimes fast and sometimes slow.
Multiple devices on the network would aggravate the situation.
Some graphics would not fully display although a refresh often helped.
Massive pages with high bandwidth, cpu and memory requirements would really suffer.
So after recently upgrading to windows 10 x64 I had to dig into the issues.
I had few if any problems on windows 7 x64.
Most of this time was using Bethere and PPPoE however.

Issue 1 - The Intel USB stack drivers used for windows 7 break in windows 10 upgrade.
You will get USB device conflicts unless you install the latest intel chipset drivers specifically for windows 10.
My razer naga was clashing with my keyboard and wifi.
Issue 2 - These new chipset drivers break the existing USB device drivers which have to be rewritten for windows 10 too. So USB wifi is borked.
Once these issues are addressed we can get onto the real problem.

For those that dont know how, right click my computer or this pc > manage > event viewer > system events.
If you see this TCP-IP "Warning Event" in your error logs, you have a major problem....

Windows EventID 4230 - Tcpip - Warning.... (System Event)
TCP/IP has chosen to restrict the congestion window for several connections due to a network condition.
This could be related to a problem in the TCP global or supplemental configuration and will cause degraded throughput.

...If you dont see this error, than this wont help and ignore this post.

Whats happening to create this event is as outlined by the OP.
Packets are being lost, so windows is dropping the windowing size of packets sent in an attempt to adress the packet loss.
So why am "I" getting this packet loss ?
It comes courtesy of Netgears built in maximum wan_MTU size.
When I am using PPPoATM I can select anywhere from 616-1458 (including message header) on the netgear wan_mtu setup.
So its basically 1430 bytes + 28 byte header by default.
This is the size of the packets that my router sends to my ISP SkyUK.

My wifi and ethernet LAN run at 1500 bytes (1472 bytes + 28 byte header).
This is pretty typical and the default for all home LANs these days.
So I send 1472 bytes to my router (+28 header...keep up).
My router instantly fragments that into a 1430+28 packet and a 28+28 packet.
Yes. Every packet I send from any device on the LAN, now becomes two packet when it hits the internet.
The first packet having the fragmentation bit set to let the receiver know that another part follows.
Latency is attached to both packets which cannot be rebuilt until they both arrive via any random path.

So, your next act is to right click ...all apps > windows system > command prompt as admin (right click)
When its up, you need to type the following in...
ping google.co.uk -f -l 1472
...what this does is sends 4 packets to google (arbitrary but other sites might change their MTU) with the "do not fragment -f" bit set.
The packet size used in this case is 1472 bytes long (+28 byte header).

If your wan_MTU was 1472+28 and your lan_MTU was 1472+28 you would get a response like so...
C:\WINDOWS\system32>ping google.co.uk -f -l 1472
Pinging google.co.uk [216.58.213.67] with 1472 bytes of data:
Reply from 216.58.213.67: bytes=64 (sent 1472) time=790ms TTL=57
Reply from 216.58.213.67: bytes=64 (sent 1472) time=614ms TTL=57
Reply from 216.58.213.67: bytes=64 (sent 1472) time=495ms TTL=57
Reply from 216.58.213.67: bytes=64 (sent 1472) time=570ms TTL=57
Ping statistics for 216.58.213.67:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 495ms, Maximum = 790ms, Average = 617ms
(NB I was running a cloud sync here at the same time hence the high latency)

If your wan_MTU was less than 1472+28 you would get this...
C:\WINDOWS\system32>ping google.co.uk -f -l 1473
Pinging google.co.uk [216.58.213.67] with 1473 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 216.58.213.67:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
(NB I used 1473 here to force fragmentation after hacking/fixing Netgears nvram)

If you got this far the chances are you received the latter ping response.
Your first fix is to change your router wan_mtu to 1500 if it is not already so.
Once done you should be able to ping google with 1472 bytes without fragmentation.
However, SkyUK has an MTU of 1500 and other ISPs do not.
If you can not ping at 1472 bytes (even with the router mtu set at 1500) or cant set the router at 1500 then read on.

Option 1 (if you can set the router wan_mtu at 1500).
Ping google using a smaller and smaller packet size until you find the highest value without fragmentation
This will be your ISPs mtu seting.
You now have to do two things to stop fragmentation.
Step 1 - change you router wan (and lan) mtu to the ping framesize + 28 bytes for the header.
Step 2 - change every device on your lan that connects to the internet to use the same mtu size
Have fun with that.
This information is readily available on the internet so I leave this here.
Your ISP will optimise its own gateways and internet backbone from here.

Option 2 (if you cant set the router wan_mtu at 1500).
I cant help everyone with this.
For Netgear routers though....open a browser and replace the ip address of your router to connect....
http://192.168.1.1/setup.cgi?todo=debug
....this will force the router into debug mode and enable telnet access.
open command prompt as above and...
telnet 192.168.1.1 [you may need to add windows builtin optional telnet client in add remove programs]
...use your normal admin username and password at the prompt to access the router command line and change the IP address to suit.
Then type...
# nvram get wan_mtu ...entering this will show....
wan_mtu=1458 ....the current wan_mtu size...
# nvram set wan_mtu=1500 ....entering this will set the wan_mtu to 1500
# nvram commit ....entering this commits the change to nvram memory now.
# reboot ...will restart your router with the new mtu settings.
If you now goto your routers web page and check the mtu it should say 1500 instead of 1458.
At this point your lan devices, router and ISP should all be running the same size MTU with no more fragmentation at this end.
My +999 at waystations has gone from the past two days and my highest has been 350 so far.
The tcpip events no longer show in event manager and my performance is no longer degraded.
(# ip link list | grep mtu ....you can use this if you want to check the mtu setting of all the routers interfaces like wifi and lan.)

TL;DR
Reducing MTU size reduces network performance.
Some router/modem manufacturers limit the wan_mtu below the standard ethernet framesize and induce fragmentation that the user can do nothing about.
Some ISPs run their own reduced mtu which in itself will possibly induce even more fragmentation.
ZOS run their game server at 1448 bytes + 28 byte header....so any frames that enter their domain are fragmeted again if your native mtu is higher than 1448+28
The only people that will experience no fragmentation when talking to ZOS is where the end to end MTU is 1448 bytes (including your lan devices) that dont cross different types of networks.
Crossing the internet involves traversing many differnet types of media (ATM for instance) which have their own optimal mtu that isnt necessarily compatible with 1448.
The further you are away from ZOS, the greater the chance the media will change and performance will suffer.
Many firewalls view fragmentation as a possible attack and will deal with it in any number of ways...which will limit performance.
https://en.wikipedia.org/wiki/IP_fragmentation_attack

zerosingularity · March 2016

hrothbern wrote: »

Enraged_Tiki_Torch wrote: »

Excellent post. There are pros and cons to both. I am sure when the server code was written, there was a discussion of which protocol would be best for the game. Although UDP is faster and has less overhead, TCP is more reliable especially with large data transmission. This game sends a lot of data and even though packet loss sucks, it might suck even more if packets were not sent/received at all. Say you were about to die, and you wanted to cast a heal on yourself; sorry but that packet was lost and you died. Also "ghosting" is an issue with UDP.

People complain about Lag but I have been playing Online games for quite a while and haven't found one lag-free yet.

, it might suck even more if packets were not sent/received at all.

I cleaned my inventory issues today by sending a lot of stuff to my second account.

My stomach would really turn if data integrity transmission would not be guaranteed for that stuff.

I think this might explain why TCP was chosen over UDP. Can you get UDP to guarantee data transmission for stuff like this? If not, then with the game how it is, UDP cannot be used, ever.

ralonasan · March 2016

Lol, they might as well make a new game. Changing it to RUDP would take way too much time and effort. Not worth.

KhajitFurTrader · March 2016

Awesome post @Rune_Relic, thank you for sharing your findings. I agree that MTU/MSS sizes is still a minefield(*), even today (which by now should be nobody's concern anymore). But given the vast number of possible combinations in network gear and consumer-/carrier-grade broadband tech that's being used, it's inevitable that some people will encounter problems which other's don't. So it's important that well informed people share their knowledge with others.

*) Ever tried to use jumbo frames in an enterprise network environment comprised of gear made by different manufacturers, who all vouch for supporting them? The only sensible advice in this case is: don't.

Septimus_Magna · March 2016

Can you explain why the latency in large scale pvp battles pretty decent before the lightning patch?
(update 1.4 if I remember correct)

Agalloch · March 2016

The game supported large scale battles in the early days . I saw hundreds of players on one screen, even a thousand. In the beginning the engine was great with no lag and no problems at all.
I think they must upgrade their servers and internet band. Also the engine.
Also they must launch a FREE DLC named REAL ESO dedicated only to removal of perfomance issues and bugs.
I'm a fan of this game and I'm sure the issues will be fixed.

Please no DB until you fix the issues ZOS .

We need engine improvements, large scale PVP with no lag ( remember how ESO was advertised ZOS?), removal of the bugs and issues launched with Thieves Guild. Also please fix the other issues we know about them since one year ...or maybe more.

Please ZOS fix the issues, improve the engine.

AFTER THAT U CAN LAUNCH DB.

Also u can add free content also like other Buy to Play games did ( look at Diablo 3, or Guild Wars 2..they have one big expansion that u must buy ..the rest of content is free )

Also u can offer the DB DLC for free after u fix the issues of the game . Yes for free as an incentive to regain the trust of players.
Please ZOS I want to play this game ...please fix it ..

Excuse my english.

hrothbern · March 2016

Hereby a link to a very old discussion on latency reduction in WOW from 2011 that mentions a software named "Latency optimiser"
http://www.wowhead.com/forums&topic=182606/reduce-latency-addon

Digging further, here a link to a Badosoft describing the general latency issue and the Latency optimiser software when ESO just started
https://www.badosoft.com/knowledgebase/the-elder-srolls-lag-fix.php

And here a link to a free download of the latest version of "Latency Optimiser".
http://latency-optimizer.en.softonic.com/

Question: has anybody experience in using this software on ESO ?

helediron · March 2016

TCP isn't necessarily the root of evil. If server side is already coded to work asynchronously, changing to UDP does not help. I think this is still classical server congestion. And i don't think we can do any worthwhile analysis from outside.

sirinsidiator · March 2016

@RinaldoGandolphi that's a neat theory, but it does not match up with what I see during these situations.
1) I have a network meter gadget running on my second screen at all times and if packet loss really is the reason, it should reflect in the graph it shows, which it does not. Instead of throttling I see it going along like nothing happend, which should not happen if I lost packets.
2) It does not explain why all parts of Cyrodiil get the same amount of insane lag, but not every player everywhere. If there was so much outgoing traffic that packets get dropped, it should affect the traffic for all zones all the same, as we only use one IP for connecting to the server.
3) If packet loss really was the reason, my interaction with an enemy should feel differently. The order in which things happen during these massive lags does not really look like a delay because of connection issues, but more like a server that cannot handle calculations in time.

Not to say that packet loss is not a problem in some cases. Especially what has been said about packet loss via WLAN connection is true. You can test this easily by using the ping command on your local router. Just type "ping /n 100 <ip-of-router>" in a command line window. You can assume that every packet that takes longer than 1ms had to be sent more than once. Usually you will see between 5% and 70% "bad" packets depending on your connection quality.

KhajitFurTrader · March 2016

hrothbern wrote: »

Question: has anybody experience in using this software on ESO ?

I view software like this as snake oil, and I strongly advise against paying any amount of money for it. This kind of software always comes with an elaborate, if not confounding visual display to show its (pretended or greatly exaggerated) effects, when all it does is to tweak a few registry settings (if they still have any effect in modern versions of Windows), which could've also been tweaked manually by users, if they knew what they're doing.

A very common trick, aside from altering MTU/MSS, is to lower the size of both send and receive window buffers way down, which of course will positively influence latency, while throughput/transmission rates will inevitably tank. Since these changes affect the whole system, this might help in a minority of special cases, but for a majority it will very likely become more of a hindrance.

aubrey.baconb16_ESO · March 2016

Zyle wrote: »

How was the game able to run without excessive latency in the earlier days? I get the advantages/disadvantages between the two protocols, but I don't get why it's an issue now if there was a time where TCP being used didn't cause terrible latency. I still think it's horrendous lack of optimization rather than improper protocol selection. Not saying UDP wouldn't be a better option, I just don't think it's the solution to the ridiculous latency we're seeing.

Because significant code was moved client side to server side because of cheats exploiting the client side code.

Rune_Relic · March 2016

I am rusty and a little busy to read up at the moment.
Does IPv6 enable packet by packet authentication so a datastream is not required ?
rather than session encryption and authentication.

That way you can lock down the client for peer to peer and use packet based gated access for DDoS.
All data would have to be locked in to single packet and become a layer 4 routing protocol kind of thing.
Payload size would have to be 1400 or something to eliminate fragmentation cross internet.

I am thinking a client server tcp initialisation to exchange keys.
Then best effort peer-peer udp transfer packet by packet.
Problem here would be recovery if you copied the packets to zos backup but udp packets were lost.
Clients might get upset if their gameplay was regressed at startup and progress lost.
Do we not get that with rollbacks and rubber banding anyway.

EDIT: some light reading I couldn't resist...
https://www.arbornetworks.com/blog/asert/ipv6-fragmentation/

cazlonb16_ESO · April 2016

Zyle wrote: »

How was the game able to run without excessive latency in the earlier days?

It never did. As soon as people started using turtle/bomb groups in PvP it was pretty much game over.

Zyle · April 2016

cazlonb16_ESO wrote: »

Zyle wrote: »

How was the game able to run without excessive latency in the earlier days?

It never did. As soon as people started using turtle/bomb groups in PvP it was pretty much game over.

It most definitely did prior to the bot patch, but as someone has explained to my inquiry it's partly because they shifted a bunch of client side things to server side to deal with bots and it resulted in a more intense strain on the server.

rfennell_ESO · April 2016

The question regarding UDP vs TCP would be exploiters and those not playing on the up and up.

I mean we already have both of them, and from my experience in pvp things tend to be fine until the exploitive minority arrives Currently in game you can have nearly lag free (or some lag, not crippling lag) fights nearly everywhere. But when the macroed out crowd (that obviously sends more information than "non exploitive" players do and usually concentrates on abilities that are going to lag things out more) turns up like a bad penny things tend to degrade.

I can tell you for a fact that I've seen the arboretum melee be totally responsive and nearly lag free. I've seen emp keep dethrone fights also be nearly lag free. Then I see "the usual suspects" turn up and lag always follows (the usual suspects being the elitist exploitive types that speak in "salt" "Git gud" and other nonsense speech).

It's been like that since Havoc would spend 30 minutes aeing terrain just to fubar the servers.

I've little doubt that the malcontented exploitive minority will always play the way they do (until they move on to ruin CU). That's their modus operandi, that's their chosen play style... it's not good enough for them to "win", their enjoyment is ruining others play.

How does UDP handle people that will deliberately mess with the game versus TCP?

Oh and yes Da Nile isn't just a river in Egypt, we know... the NA DC contingent of guys dodging 50% of attack while being magicka builds and exploiting knock downs out of nowhere and skill bar lockouts with lagged out charges will feel compelled to respond with "git gud" "prove it" and other nonsense. It's OBVIOUS when the cheats show up and all performance begins degrading.

To any and all still mystified by the lag, always note the names that turned up when it started. You will find, all on your own, it's not a coincidence (and never has been).

ThanatosXR · March 2019

https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol
would be the the best method allowing Ordered and Unordered packets to be sent.

Technically a mmo with hybrid peering would be the fastest but hard to design
All clients could talk to each other and the server
but, security could be a issue and the host would be more passive, allowing players to lagswitch cheat, also customer complaining that the game is using all there internets,