Networking TCP offload big performance gain

NeoXanthus · January 2016

I ran with 5760x1080 NVidia surround for about a year in ESO with no issues in regards to performance. I use FRAPs to limit the top end to 100FPS because I want to limit total variation of min and max FPS. I upgraded to 7680x1440 and have sense been experiencing all sorts of performance related issues with FPS as low as 18 and to 100 FPS. I thought I just was reaching the limits of my first gen quad-SLI GTX Titans. Something I believed I would just have to live with until I upgraded my video cards. Through various diagnostic software I see the cards are actually fine it is just the CPU is getting overwhelmed with the ESO process under the increased resolution requirements. When I use a single 2560x1440 monitor the ESO process is fine from a CPU consumption point a view. On a hunch I spanned the workstation from my switch to a promiscuous sniffer and looked at the networking traffic to my surprise ESO uses mainly TCP instead of UDP communication. TCP by its very nature is more CPU intensive than UDP. The traffic from the computer running only a single monitor at 2560x1440 was fine no issues, when I increased local load and went to 3 monitors so with a resolution of 7680x1440 I saw many hiccups on my side causing many TCP retransmits and overall slower responses. From here I see two options if I want to keep 7680x1440 resolution and improve the performance. Replace my current CPU, motherboard, and memory with a newer version, or upgrade my NIC with one that has true parallel offload stack. Because I eat, sleep, and breathe networking I did the later. For everyone’s information there are only two companies still around that make real true parallel offload cards Chelsio and Qlogic. (These types of cards are used in commodities exchange environments, Hadoop clusters, and for other low latency needs.) I went with a Chelsio unified wire card and the total system performance was night and day different under high CPU load with my resolution at 7680x1440. My low end min FPS is 90 and my cap of 100 FPS is hit all the time. I can’t believe I waited so long to use a real ToE card the game response is just awesome!

Shyfty · January 2016

That sounds awesome. Could you tell me what card you actually got? I want to see if this will help my performance problems.

ced · January 2016

yeah, if this is the solution it seems to be I'd like to be in on it too

any chance you could get some footage from cyrodiil at primetime, showing if it improves performance there?

kojou · January 2016

I too would be interested in the model of card you selected and what the rest of your network architecture looks like.

WalkingLegacy · January 2016

I can't imagine an MMO using UDP

Nestor · January 2016

NeoXanthus wrote: »

TCP by its very nature is more CPU intensive than UDP.

It should not be. UDP relies on Layer 7 or Application Layer to assure delivery, where as TCP relies on Layer 4 to assure delivery using Windowing and Acks. Of course that all depends on how involved the application layer gets in the delivery assurance. The CPU is usually involved in Layer 4, but is a lot less overhead than having the game itself assure delivery of packets.

Still off loading the traffic to a more intelligent NIC can offer some benefits in some cases. Some older NICs would allow for memory upgrades which could help with this, but most people have on board nics which are just really a transceiver with some shift registers.

Docmandu · January 2016

WalkingLegacy wrote: »

I can't imagine an MMO using UDP

DAoC

and as to OP: rofl... sounds like you're a KillerNIC customer.

SirAndy · January 2016

WalkingLegacy wrote: »

I can't imagine an MMO using UDP

Many of us who have worked on client/server systems and game engines have pointed out many of the flaws of the ESO network code to ZOS since early beta.

I and others first suggested a prediction based model utilizing UDP for transport more than 2 years ago.

I've worked as a programmer in the gaming industry for many years and as stated in this thread, TCP is the worst choice when it comes to client/server communication for a real-time online game.

UDP combined with a client/server prediction model would eliminate most, if not all, of the lag in Cyrodiil since the server would never wait for information from players with a slow ping/connection before broadcasting updates to other players.

With the current implementation, a player with a crappy ping can actually lag out everyone else around them in PvP, which is a disaster for large scale battles.

wayfarerx · January 2016

WalkingLegacy wrote: »

I can't imagine an MMO using UDP

There's lots of great uses for UDP in an MMO. Voice chat is an easy one.

Lefthy · January 2016

Hey guys i was reading the thread and i saw some many of you knowing about networking that i cant avoid to ask about my problem.

The simple version is:
i have 150ms ping with every software on NA server (games in general, speedtests, etc) but when i play TESO i get 250ms - is this normal? do any of you recognize this issue?

The full thread about this is here: http://forums.elderscrollsonline.com/en/discussion/240990/latency-diferences

Sorry for coming from nowhere and ask this, i just got tempted whit all the networking discussion and i have a real bad time dealing whit this.

SirAndy · January 2016

Lefthy wrote: »

i have 150ms ping with every software on NA server (games in general, speedtests, etc) but when i play TESO i get 250ms - is this normal? do any of you recognize this issue?

Do a trace route, that will show you where the slowdown is happening.
I suspect it may be the ZOS server(s) itself.

You can use your Resource Monitor to find the QFDN/IP of the ESO server you're connected to, then open a DOS box and type:

tracert xx.xx.xx.xx

where xx.xx.xx.xx is the server IP ...

Robbmrp · January 2016

SirAndy wrote: »

WalkingLegacy wrote: »

I can't imagine an MMO using UDP

Many of us who have worked on client/server systems and game engines have pointed out many of the flaws of the ESO network code to ZOS since early beta.

I and others first suggested a prediction based model utilizing UDP for transport more than 2 years ago.

I've worked as a programmer in the gaming industry for many years and as stated in this thread, TCP is the worst choice when it comes to client/server communication for a real-time online game.

UDP combined with a client/server prediction model would eliminate most, if not all, of the lag in Cyrodiil since the server would never wait for information from players with a slow ping/connection before broadcasting updates to other players.

With the current implementation, a player with a crappy ping can actually lag out everyone else around them in PvP, which is a disaster for large scale battles.

Hello @SirAndy, from your post your the perfect person to ask this. Would what the OP is indicating help others out by switching to a NIC with Offload capability? I am interested in smoothing my gameplay as much as possible and if one of these would work, I am definately going to look into getting one. I don't have issues with generally with FPS but do have some dips in it when I don't turn off all of my addons prior to going into Cyrodil, especially MM and anything Guild store related. My game suffers from the same latency spikes as others. If there's anything I can do on my end, I'll do it.

Thanks!

SirAndy · January 2016

Robbmrp wrote: »

Hello @SirAndy, from your post your the perfect person to ask this. Would what the OP is indicating help others out by switching to a NIC with Offload capability? I am interested in smoothing my gameplay as much as possible and if one of these would work, I am definately going to look into getting one. I don't have issues with generally with FPS but do have some dips in it when I don't turn off all of my addons prior to going into Cyrodil, especially MM and anything Guild store related. My game suffers from the same latency spikes as others. If there's anything I can do on my end, I'll do it.

Part of the problem with TCP is the added low-level overhead within the protocol itself. Offloading some of the processing can indeed free up CPU cycles and improve network speed, especially if done with a true parallel hardware implementation.
https://en.wikipedia.org/wiki/TCP_offload_engine

But before you sink some serious money into upgrading your NIC, make sure you don't have any other bottlenecks in your local network that would negate any gains from a NIC upgrade. It's the sum of the parts that really make a network perform.

Robbmrp · January 2016

SirAndy wrote: »

Robbmrp wrote: »

Hello @SirAndy, from your post your the perfect person to ask this. Would what the OP is indicating help others out by switching to a NIC with Offload capability? I am interested in smoothing my gameplay as much as possible and if one of these would work, I am definately going to look into getting one. I don't have issues with generally with FPS but do have some dips in it when I don't turn off all of my addons prior to going into Cyrodil, especially MM and anything Guild store related. My game suffers from the same latency spikes as others. If there's anything I can do on my end, I'll do it.

Part of the problem with TCP is the added low-level overhead within the protocol itself. Offloading some of the processing can indeed free up CPU cycles and improve network speed, especially if done with a true parallel hardware implementation.
https://en.wikipedia.org/wiki/TCP_offload_engine

But before you sink some serious money into upgrading your NIC, make sure you don't have any other bottlenecks in your local network that would negate any gains from a NIC upgrade. It's the sum of the parts that really make a network perform.

Great, thank you for the information. It helps a lot.

WalkingLegacy · January 2016

SirAndy wrote: »

WalkingLegacy wrote: »

I can't imagine an MMO using UDP

Many of us who have worked on client/server systems and game engines have pointed out many of the flaws of the ESO network code to ZOS since early beta.

I and others first suggested a prediction based model utilizing UDP for transport more than 2 years ago.

I've worked as a programmer in the gaming industry for many years and as stated in this thread, TCP is the worst choice when it comes to client/server communication for a real-time online game.

UDP combined with a client/server prediction model would eliminate most, if not all, of the lag in Cyrodiil since the server would never wait for information from players with a slow ping/connection before broadcasting updates to other players.

With the current implementation, a player with a crappy ping can actually lag out everyone else around them in PvP, which is a disaster for large scale battles.

Wouldn't it make loss of player input more frequent since the realibility isn't as good as TCP?

Perhaps their QoS needs some adjusting?

SirAndy · January 2016

WalkingLegacy wrote: »

Wouldn't it make loss of player input more frequent since the realibility isn't as good as TCP?

Not really. While technically true that UDP packets aren't guaranteed to make it to the destination in reality packet loss is very rare.

I have worked on extremely high volume client/server implementations and adding a simple single byte sequence number is enough to ensure data integrity on a UDP stream where packet loss is a concern.

Malmai · January 2016

Why do we have to do researches and ZOS doesn't give a ... for it ?

smacx250 · January 2016

FYI - if you are on a mac you can use "nettop" to see the ESO TCP connection statistics. In the terminal app, type: "nettop -m tcp". You can use the arrow keys to move the list up and down until you can find the ESO app, something like this (sorry - wrapping is ugly):

interface           state        bytes in       bytes out   rx dupe    rx ooo     re-tx
eso.60829                                                                                        204 KiB        3849 B    1448 B       0 B       0 B
    tcp4 192.168.1.16:59051<->198.20.200.26:24503                    en0     Established         140 KiB        2045 B    1448 B       0 B       0 B
    tcp4 192.168.1.16:59055<->198.20.200.28:24108                    en0     Established          59 KiB         896 B       0 B       0 B       0 B

This updates live, and shows such things as RX and TX bytes, duplicates and out of order received, retransmissions, and round trip latency.

dantheman1972 · January 2016

They simply need to use SCTP, which combines the best of TCP and UDP. The telecommunications industry has used this for the past 15 years or so, and is the primary reason that IP based telecom systems are even remotely viable. Back when we used TCP for signaling during the early dawn of VoIP, it was a catastrophe with any form of brief congestion.

cschwingeb14_ESO · January 2016

UDP is actually very good for a bunch of traffic in MMOs. Take position updates for example: if you are getting 5 updates per second on an enemies position, do you really care if you lose the third of 5 updates pet second? No, the enemies position has already been refreshed.

Other traffic, like skill activation so you can see the animation, is more important

NeoXanthus · January 2016

I was so busy at work today I had no chance to check my thread I had no idea it had this much traction.

To answer some questions and statements.

Shyfty: I actually bought two cards both from eBay inexpensively. One I think is an OEM version of a T520-CR current generation of a Chelsio Unified Wire card. I could not get it to work but it was so cheap did not bother sending it back. The second card was an older third generation Chelsio S310E-CR card it worked perfectly and is the current card I have been enjoying for a little over two weeks now. If you Google these cards I know people well say those are 10gb cards and for 99.999% of the home market that would not be useable but there is versions out there both from QLogic and Chelsio that are 1gb like the Chelsio S304E-BT. I just went with 10gb because I have a 10gb fiber switch at home and the 10gb Chelsio cards on eBay believe it or not are less expensive. Note extra bandwidth does nothing for your game performance as you are limited of course by your internet service provider upstream access device.

ztyhurstub17_ESO: For NIC card see post above. For my home network I only have a single chassis switch running and I don’t recommend my configuration as it can be acquired relatively cheap on eBay but it is expensive to run. I have a Cisco 6506e with twin Sup 720-3b, twin WiSMs, a 6548-GE-TX with 48 port PoE, and a 6704-10GE with 3B DFC and dual 4000 watt PS.

Nestor: For a game I would argue assure packet delivery is not a requirement. There is a reason UDP is usually the protocol of choice for applications such as gaming and voice with many small packets if you drop a few no big deal. What is more important, better FPS and lack of jitter under system load, or assuring every packet arrives?

docmandu: I agree with you KillerNIC is complete snake oil as it tried to partially offload UDP. However true parallel TCP offload is a noticeably gain. Here is a clip from wiki on ToE “Freed-up CPU cycles

A generally accepted rule of thumb is that 1 hertz of CPU processing is required to send or receive 1 bit/s of TCP/IP.[3] For example, 5 Gbit/s (625 MB/s) of network traffic requires 5 GHz of CPU processing. This implies that 2 entire cores of a 2.5 GHz multi-core processor will be required to handle the TCP/IP processing associated with 5 Gbit/s of TCP/IP traffic. Since Ethernet (10Ge in this example) is bidirectional it is possible to send and receive 10 Gbit/s (for an aggregate throughput of 20 Gbit/s). Using the 1 Hz/(bit/s) rule this equates to eight 2.5 GHz cores.

Many of the CPU cycles used for TCP/IP processing are "freed-up" by TCP/IP offload and may be used by the CPU (usually a server CPU) to perform other tasks such as file system processing (in a file server) or indexing (in a backup media server). In other words, a server with TCP/IP offload can do more server work than a server without TCP/IP offload NICs.”

https://en.wikipedia.org/wiki/TCP_offload_engine

SirAny: I can’t agree with you more. You sound like you work in a similar field as myself. UDP is the way to go with this game.

Dantheman1972: SCTP is a really good idea ESO should look in to it.

Docmandu · January 2016

NeoXanthus wrote: »

docmandu: I agree with you KillerNIC is complete snake oil as it tried to partially offload UDP. However true parallel TCP offload is a noticeably gain. Here is a clip from wiki on ToE “Freed-up CPU cycles

A generally accepted rule of thumb is that 1 hertz of CPU processing is required to send or receive 1 bit/s of TCP/IP.[3] For example, 5 Gbit/s (625 MB/s) of network traffic requires 5 GHz of CPU processing. This implies that 2 entire cores of a 2.5 GHz multi-core processor will be required to handle the TCP/IP processing associated with 5 Gbit/s of TCP/IP traffic. Since Ethernet (10Ge in this example) is bidirectional it is possible to send and receive 10 Gbit/s (for an aggregate throughput of 20 Gbit/s). Using the 1 Hz/(bit/s) rule this equates to eight 2.5 GHz cores.

Many of the CPU cycles used for TCP/IP processing are "freed-up" by TCP/IP offload and may be used by the CPU (usually a server CPU) to perform other tasks such as file system processing (in a file server) or indexing (in a backup media server). In other words, a server with TCP/IP offload can do more server work than a server without TCP/IP offload NICs.”

https://en.wikipedia.org/wiki/TCP_offload_engine

LOL.. just.. lol..

Let's me quote something else in that article:

This article's factual accuracy may be compromised due to out-of-date information.

https://en.wikipedia.org/wiki/Talk:TCP_offload_engine#Outdated

BTW: The game will have other issues if it's starting to require 5Gbit/s bandwidth...

SirAndy · January 2016

Docmandu wrote: »

LOL.. just.. lol..
Let's me quote something else in that article:

This article's factual accuracy may be compromised due to out-of-date information.

And your point is?

HeroOfNone · January 2016

Hmm, wonder how much I'd gain since I'm playing the ESO and streaming at the same time

Any recommendations on a gigabit network in under the 100 dollar range?

Chims · January 2016

From every aspect I have seen on a server level, TCP chimney offloading is pure garbage. I bet Zenimax can get increased throughput by disabling it on their servers alone (its on by default).

Not sure about the end/client user benefits. I would try to turn it off globally if you have performance issues and see what happens.

To disable these settings, the following statements can be run in a command prompt:

- netsh int tcp set global chimney=disabled
- netsh int tcp set global RSS= disabled
- netsh int tcp set global NETDMA= disabled

The same setting has to be modified at the NIC card levelr. To do so follow the below steps.

a. Click Start, click Run, type ncpa.cpl, and then click OK.
b. Right-click a network adapter object, and then click Properties.
c. Click Configure, and then click the Advanced tab.
d. In the Property list, click Receive Side Scaling, click Disable in the Value list, and then click OK.
e. Go through the list and do this for
- TCP/IP Offload (Make it disable)
- IPV4 Checksum Offload (Make it disable)
- Large Send Offload V2 (IPv4) (Make it disable)
- TCP Checksum Offload (IPv4) (Make it disable)

NeoXanthus · January 2016

Chims wrote: »

From every aspect I have seen on a server level, TCP chimney offloading is pure garbage. I bet Zenimax can get increased throughput by disabling it on their servers alone (its on by default).

Not sure about the end/client user benefits. I would try to turn it off globally if you have performance issues and see what happens.

To disable these settings, the following statements can be run in a command prompt:

- netsh int tcp set global chimney=disabled
- netsh int tcp set global RSS= disabled
- netsh int tcp set global NETDMA= disabled

The same setting has to be modified at the NIC card levelr. To do so follow the below steps.

a. Click Start, click Run, type ncpa.cpl, and then click OK.
b. Right-click a network adapter object, and then click Properties.
c. Click Configure, and then click the Advanced tab.
d. In the Property list, click Receive Side Scaling, click Disable in the Value list, and then click OK.
e. Go through the list and do this for
- TCP/IP Offload (Make it disable)
- IPV4 Checksum Offload (Make it disable)
- Large Send Offload V2 (IPv4) (Make it disable)
- TCP Checksum Offload (IPv4) (Make it disable)

Chims,

This issue I was talking about was client side CPU exhaustion not server side networking performance. I agree chimney offloading is not a good thing however what I was referencing is complete parallel TCP stack not some form of OS based hybrid TCP offload. Also I don’t know but I would doubt the back end servers host are Microsoft and instead some form of Linux distro. It would make sense from a scale perspective especially when it comes to licensing and management.

NeoXanthus · January 2016

HeroOfNone wrote: »

Hmm, wonder how much I'd gain since I'm playing the ESO and streaming at the same time

Any recommendations on a gigabit network in under the 100 dollar range?

With true parallel offload, I don’t know of any in that price range that are copper 1gb.

Chims · January 2016

NeoXanthus wrote: »

Chims,

This issue I was talking about was client side CPU exhaustion not server side networking performance. I agree chimney offloading is not a good thing however what I was referencing is complete parallel TCP stack not some form of OS based hybrid TCP offload. Also I don’t know but I would doubt the back end servers host are Microsoft and instead some form of Linux distro. It would make sense from a scale perspective especially when it comes to licensing and management.

I agree they are probably linux based servers and as I said I am not sure what the impact would be on a client. I am just saying it might be something to try on the client side if your CPU activity is high as that's what it does for windows, offloads network activity to the CPU. If your CPU is spiking its worth trying and takes all of 5 minutes to implement.

KhajitFurTrader · January 2016

For anyone interested in the matter, here's an in-depth background article on TCP vs. UDP in game network communication. The gist: try to use the best tool for any given task (well, duh). UDP has its drawbacks, e.g. no packet ordering, as has TCP, e.g. increase in latency when re-transmitting on packet loss. It depends on what you'll need to achieve -- the linked article explains it quite well.

Docmandu · January 2016

SirAndy wrote: »

Docmandu wrote: »

LOL.. just.. lol..
Let's me quote something else in that article:

This article's factual accuracy may be compromised due to out-of-date information.

And your point is?

My point: don't bother spending money on a super duper network card... just forget it's there and be happy. On board motherboard networking is more than performant enough to run the game while streaming to twitch and playing a youtube vid at the same time, without having impact on your CPU.

and the other point... not all info on the internet is correct and/or up-to-date.

NeoXanthus · January 2016

Docmandu wrote: »

SirAndy wrote: »

Docmandu wrote: »

LOL.. just.. lol..
Let's me quote something else in that article:

This article's factual accuracy may be compromised due to out-of-date information.

And your point is?

My point: don't bother spending money on a super duper network card... just forget it's there and be happy. On board motherboard networking is more than performant enough to run the game while streaming to twitch and playing a youtube vid at the same time, without having impact on your CPU.

and the other point... not all info on the internet is correct and/or up-to-date.

Out-of-date is just part of the standard Wikipedia template after a length of time this is nothing to get hung up on.

https://en.wikipedia.org/wiki/Template_talk:Update#Time_sensitive_statements

“The whole point of having a wiki-based encyclopedia is that it's continually updated. Why would we treat our readers and editors as if they were too dense to realize that when they read information from any source that information may be out of date?

I've got a couple of single volume encyclopedias on my bookshelf and without looking I can tell you they contain descriptions of the World Trade Center in New York as a contemporary building, and have absolutely nothing about Presidents Clinton, Bush or Obama, or Al Qaeda. Yet somehow I manage to cope with these "out-of-date" works. This doesn't mean I'm unique, it's what humans do.”

As for being happy with your onboard NIC that is certainly your prerogative. I have an issue with a tremendously overloaded CPU when running ESO at such a high resolution. My solution was to use a true parallel offload NIC card with an ASIC processor and memory. I thought I would share this solution with the ESO community, hoping it might help others. The increased FPS speed as a result is quantifiable, however if you have lower CPU taxing then you are right your onboard card may be completely adequate.