ESO PvP Update [Updated June 2022]

Mojmir

Show dont tell

Elsonso

.

ks888 wrote: »

Murky_Dregs wrote: »

Her remarks were incredibly candid and are definitely a real glimpse into how your people feel.

Given the glib customer disregard displayed, how do you plan to address that issue, meaning your team's professional resolve regarding the matter, in order to ensure delivery of the quality product that many of us pay well over hundreds of dollars a year for and deserve?

^^^ this

The thing that must be remembered is that Ms Lambert is not a representative of ZOS and her views should not be considered the views of ZOS. I certainly don't consider her to be a ZOS representative. I don't even consider her to be representative of Rich's opinions. Never have. Her opinions are hers. If people choose to consider her to be a ZOS representative, that is their choice, but it does not change the fact that she is not.

Should Rich have stepped up and turned her off? Possibly, as would be expected any time a user got out of hand during a stream. She is not a skilled streamer, and is opinionated, and that makes her an easy target for people baiting her in chat. Should the moderators have been quicker about nuking that person? Probably.

As for what ZOS needs to do? My feeling is nothing at all. The stream belongs to Rich, so he should handle things more effectively in the future. He should address that in a future stream, if there are any. As her comments were not statements from ZOS, ZOS has nothing to say.

VaranisArano

LightningWitch wrote: »

YOU. NEED. TO. CREATE. A. CONSOLE. PUBLIC. TEST. SERVER.

It's long overdue. Each time you give out PTS testing, the only feedback you all seem to get is what the new procs do.

There are far more players on console than on PC, so it's best if this audience can test these changes.

I understand in-house development is using PC based SDKs, but this shouldn't be an excuse not to build a PTS server for consoles.

I believe will provide far more feedback from the active player base than the few who then rush to their Twitch channel the moment the NDA is lifted to talk about the new builds.

If I had to guess, I'd think that's on Playstation and XBOX for much the same reason that consoles don't have addons: those companies are more careful about what code gets put into their version of ESO.

There's been a couple times that ZOS has talked about needing to get patches approved by Sony, so I don't know how well a PTS with multiple updates would play with the certfication process.

blktauna

Sylvermynx wrote: »

Einstein_ wrote: »

EF321 wrote: »

A year ago at around this time we were promised new servers. No new server.
Just saying.

to be fair, last year they said new STORAGE servers, which will NOTimprove performance.
they said we start 2022, with the acuall new hardwear for PC.

True - and there are still supply chain issues with chips which is going to affect that too. I just checked with my machine supplier, thinking I'd order my new machine now because I don't expect they'll be able to ship it before March or April. Wrong. It's already looking to David like August or September. Maybe.

One hopes that they will be using cloud to develop on. They have the whole of MS cloud at their disposal, which releases them from dependency on supply chain. It can also allow them to develop the code to scale more easliy. They are then able to customise disk images for quick deployment and backup.

blktauna

VaranisArano wrote: »

LightningWitch wrote: »

YOU. NEED. TO. CREATE. A. CONSOLE. PUBLIC. TEST. SERVER.

It's long overdue. Each time you give out PTS testing, the only feedback you all seem to get is what the new procs do.

There are far more players on console than on PC, so it's best if this audience can test these changes.

I understand in-house development is using PC based SDKs, but this shouldn't be an excuse not to build a PTS server for consoles.

I believe will provide far more feedback from the active player base than the few who then rush to their Twitch channel the moment the NDA is lifted to talk about the new builds.

If I had to guess, I'd think that's on Playstation and XBOX for much the same reason that consoles don't have addons: those companies are more careful about what code gets put into their version of ESO.

There's been a couple times that ZOS has talked about needing to get patches approved by Sony, so I don't know how well a PTS with multiple updates would play with the certfication process.

This. ZOS has no ability to make swift update on anything console. They need to submit code to MS and SONY who then run it through a process of their own which can take up to 2 weeks... and then its deployed. Not really a process that will work for a PTS

kwinter

Since Microsoft now owns eso they could do pts for xbox if they want to

Elsonso

VaranisArano wrote: »

LightningWitch wrote: »

YOU. NEED. TO. CREATE. A. CONSOLE. PUBLIC. TEST. SERVER.

If I had to guess, I'd think that's on Playstation and XBOX for much the same reason that consoles don't have addons: those companies are more careful about what code gets put into their version of ESO.

There's been a couple times that ZOS has talked about needing to get patches approved by Sony, so I don't know how well a PTS with multiple updates would play with the certfication process.

I mean, it is possible to do this. The PTS version would need to be a different SKU and separately installable. They would undoubtedly have to have the same approval process, so there might only be one or two iterations during a 4-5 week PTS cycle. The big thing is the additional megaservers, which is not a trivial investment. I would question whether it is worth all the extra work, though. I expect that the PC PTS server is not heavily used by people actually doing testing, and so two more megaservers so a relatively few people on console can do testing seems extravagant.

blktauna wrote: »

One hopes that they will be using cloud to develop on. They have the whole of MS cloud at their disposal, which releases them from dependency on supply chain. It can also allow them to develop the code to scale more easliy. They are then able to customise disk images for quick deployment and backup.

They have talked about this. ESO is not written for "cloud" in the sense that people like to throw around in here. To get the megaserver that ESO has, with the concurrency that ESO has, ESO lives in a data center on dedicated hardware.

blktauna

Elsonso wrote: »

blktauna wrote: »

One hopes that they will be using cloud to develop on. They have the whole of MS cloud at their disposal, which releases them from dependency on supply chain. It can also allow them to develop the code to scale more easliy. They are then able to customise disk images for quick deployment and backup.

They have talked about this. ESO is not written for "cloud" in the sense that people like to throw around in here. To get the megaserver that ESO has, with the concurrency that ESO has, ESO lives in a data center on dedicated hardware.

Note I said develop, not deploy. You can spin up testing and building servers in various configurations at will and not have to wait for hardware. It also helps refine the hardware requirements. Also not as expensive for the early stages.

I'm sure they have an excellent idea of what they need but I've found letting devs spin their own server up to try stuff out works better than having people sit around and wait for machines. As long as everything gets turned off when it no longer needed... Theres a hidden cash sink lol

Elsonso

blktauna wrote: »

Elsonso wrote: »

blktauna wrote: »

One hopes that they will be using cloud to develop on. They have the whole of MS cloud at their disposal, which releases them from dependency on supply chain. It can also allow them to develop the code to scale more easliy. They are then able to customise disk images for quick deployment and backup.

They have talked about this. ESO is not written for "cloud" in the sense that people like to throw around in here. To get the megaserver that ESO has, with the concurrency that ESO has, ESO lives in a data center on dedicated hardware.

Note I said develop, not deploy. You can spin up testing and building servers in various configurations at will and not have to wait for hardware. It also helps refine the hardware requirements. Also not as expensive for the early stages.

I'm sure they have an excellent idea of what they need but I've found letting devs spin their own server up to try stuff out works better than having people sit around and wait for machines. As long as everything gets turned off when it no longer needed... Theres a hidden cash sink lol

Ahh. I get it. Yeah, but then the big question is whether the environment is so different from what they develop in now that it will just set things back while people and processes move over to the new environment. 🤔

blktauna

Elsonso wrote: »

blktauna wrote: »

Elsonso wrote: »

blktauna wrote: »

One hopes that they will be using cloud to develop on. They have the whole of MS cloud at their disposal, which releases them from dependency on supply chain. It can also allow them to develop the code to scale more easliy. They are then able to customise disk images for quick deployment and backup.

They have talked about this. ESO is not written for "cloud" in the sense that people like to throw around in here. To get the megaserver that ESO has, with the concurrency that ESO has, ESO lives in a data center on dedicated hardware.

Note I said develop, not deploy. You can spin up testing and building servers in various configurations at will and not have to wait for hardware. It also helps refine the hardware requirements. Also not as expensive for the early stages.

I'm sure they have an excellent idea of what they need but I've found letting devs spin their own server up to try stuff out works better than having people sit around and wait for machines. As long as everything gets turned off when it no longer needed... Theres a hidden cash sink lol

Ahh. I get it. Yeah, but then the big question is whether the environment is so different from what they develop in now that it will just set things back while people and processes move over to the new environment. 🤔

Always the real question, yes.

SammyKhajit

Reading through these posts and really feel the pain for PVP players.

Now ZOS needs to make regular official updates about what they’ve been doing, the progress and so on. This one doesn’t doubt that the team is working hard, but silence can be, and has been interpreted, as no accountability.

Aldoss

Elsonso wrote: »

The thing that must be remembered is that Ms Lambert is not a representative of ZOS and her views should not be considered the views of ZOS.

While true, the disconnect for me is that Rich, due to very nature of who he is and what his role is at ZOS, will personally benefit from streaming a game that he creatively directs.

It would be unreasonable to suggest that he never answer a question from the perspective of his official role. That's quite literally the draw to his stream and why he typically has 200+ concurrent viewers.

As someone who watches ESO streamers on a daily basis, Rich's stream really isn't that great and I'm not sure I'd be able to believe anyone who claims that they go to his stream because of him as a person and not because of his position at ZOS.

With all of that in mind, I can't accept that he isn't personally responsible for what gets said or what messages are conveyed through the stream. He chose to let his wife talk and his wife chose to react poorly to (arguably) someone who said something in poor taste.

I don't want for Rich to have to stop streaming. I love that it exists and that it's a direct example for anyone who asks "do the devs even play this game?!". I also don't think this should result in Terri never coming on, but to suggest that Rich or his wife can say whatever they want to the viewers with no repercussions is absurd.

If he wants that protection, he needs to either stream a different game or stop fielding questions related to ESO - the latter I believe to be realistically impossible.

xaraan

I hope they figure something out to help the performance in cyrodiil (and trials) but I'll believe it when I see it.

It seems to me they don't really care to let it drag on this long and the video doesn't really do anything to make me trust zos. This person may not be a zos representative, but it makes me think two things:

1. If any stream team member had someone on their stream that was talking about zos devs like that, you can be Rich would be all over them and if it was allowed to happen again they would lose their status. So there is no off the hook IMO for everyone saying they don't represent zos. It also shows the hypocrisy of "mr. always be constructive and not mean in your feedback".

and 2. This is someone Rich has probably talked/griped to after a long day of dealing with these types of complaints and it makes me wonder what he says about them behind closed doors. Sure we all have long days and anyone we are griping to about work knows that, but if he (or more than just he at zos) are saying this consistently and feel like pvpers are doing nothing but whining, then they probably won't ever take the complaints seriously.

In the end, they have felt the game in a complete enough state to charge money for it, so "working on it" doesn't fly when it's been YEARS since the game worked decently in PvP. It seems like we get a post like this every year, but here's hoping this 'fix' works.

Lysette

Abnaxos wrote: »

Awesome post, Flang.

Flangdoodle wrote: »

So, here's where the speculation comes in: My guess is that they've known for a long time what causes all the problems in PVP, but every time they told the execs what it would take to fix them (a rewriting of the foundational server code), the execs asked how much it would cost, and when the devs told them, they said "Nope. Find a cheaper way."
[...]
Put yourself in a devs shoes for a second: you know what's wrong and how to fix it, and you tell your boss, but your boss says "No, too expensive. Try something else". You *know* nothing else will fix it. So, what do you do? SHOW them that nothing else will work.

I work in software development. Not in gaming, but in the end, games are software and software is software. This is how it works. Been there, done that, again and again and again.

Developer sees the code, says: "We have to rewrite this from scratch."
Project manager: "How long does it take?"
Developer: "X months, minimum."
Project manager: "No way. It's too expensive and the customer wants a solution now. Find another way."
(Note how the customer – that's us, the players – is part of the problem at this point)

As a result, the developer has to come up with some duct tape fix, that might improve the situation a little bit for the moment. More often than not, it actually makes things worse in the long run, and the developer warns about that – but it's quick and cheap, so management gives green light for the duct tape fix. All these fixes accumulate over the years, each one making the overall situation a little bit worse.

This happens everywhere where software is made, it's part of the business. Such a decision isn't taken lightly and it shows ZOS's commitment to the game.

This is as well my impression - this is a serious approach to get it fixed - no longer just band aids and workarounds.

Lysette

Well, if I would have to redesign it, I would build on what worked before - namely at the launch of the game they had outsourced a lot of the computation to client machines and got awesome performance like this - because most of the computation could be done in parallel on all those client machines. This opened the game unfortunately up to cheating.

Now let's just focus on the "in parallel" part - we know it cannot go back to the client machines without to get cheating again. But the heavy part of the computation can proven be done "in parallel" whilst being independent enough from other computations meanwhile - that is how they had done it - now this can be done in-house on the server side as well - by putting the parallel part of the computation on a CUDA device and compute it in parallel like before, just this time on the CUDA hardware - then performance will get back to where it has been once - and cheating is avoided - this would be my approach.

But to implement that would not take months, but just weeks - so I guess ZOS has not considered a true parallel in-house approach yet using actual true parallel programming hardware for it. What differs computation-wise from doing it on client machines (which is bad) is client machines do not slow down with branching instructions, whilst SIMD hardware is slowing down with any branching instruction in the kernel on CUDA hardware - this is why I said, it would take weeks to find an optimal computation with the least amount of branching instructions, otherwise it would just be days.

And so based on this and that Matt said, it will take the best part of this year, I guess there is a lot of other server-related stuff changed as well - not just the performance part - maybe better scalability, which would eventually be required for vet overland or other services like private instances - if they ever would want to go for that, they could then. So let's give them the time, it will better our gameplay in the end.

gp1680

As usual with updates on PvP performance from Zos, I am heartened by the proposed rewrite, but I’ll hold off on adulation until I see that it works.

Idinuse

Lysette wrote: »

Well, if I would have to redesign it, I would build on what worked before - namely at the launch of the game they had outsourced a lot of the computation to client machines and got awesome performance like this - because most of the computation could be done in parallel on all those client machines. This opened the game unfortunately up to cheating.

Now let's just focus on the "in parallel" part - we know it cannot go back to the client machines without to get cheating again. But the heavy part of the computation can proven be done "in parallel" whilst being independent enough from other computations meanwhile - that is how they had done it - now this can be done in-house on the server side as well - by putting the parallel part of the computation on a CUDA device and compute it in parallel like before, just this time on the CUDA hardware - then performance will get back to where it has been once - and cheating is avoided - this would be my approach.

But to implement that would not take months, but just weeks - so I guess ZOS has not considered a true parallel in-house approach yet using actual true parallel programming hardware for it. What differs computation-wise from doing it on client machines (which is bad) is client machines do not slow down with branching instructions, whilst SIMD hardware is slowing down with any branching instruction in the kernel on CUDA hardware - this is why I said, it would take weeks to find an optimal computation with the least amount of branching instructions, otherwise it would just be days.

And so based on this and that Matt said, it will take the best part of this year, I guess there is a lot of other server-related stuff changed as well - not just the performance part - maybe better scalability, which would eventually be required for vet overland or other services like private instances - if they ever would want to go for that, they could then. So let's give them the time, it will better our gameplay in the end.

Indeed. https://forums.elderscrollsonline.com/en/discussion/comment/7483233/#Comment_7483233

Elsonso

Lysette wrote: »

Well, if I would have to redesign it, I would build on what worked before - namely at the launch of the game they had outsourced a lot of the computation to client machines and got awesome performance like this - because most of the computation could be done in parallel on all those client machines. This opened the game unfortunately up to cheating.

Now let's just focus on the "in parallel" part - we know it cannot go back to the client machines without to get cheating again. But the heavy part of the computation can proven be done "in parallel" whilst being independent enough from other computations meanwhile - that is how they had done it - now this can be done in-house on the server side as well - by putting the parallel part of the computation on a CUDA device and compute it in parallel like before, just this time on the CUDA hardware - then performance will get back to where it has been once - and cheating is avoided - this would be my approach.

But to implement that would not take months, but just weeks - so I guess ZOS has not considered a true parallel in-house approach yet using actual true parallel programming hardware for it. What differs computation-wise from doing it on client machines (which is bad) is client machines do not slow down with branching instructions, whilst SIMD hardware is slowing down with any branching instruction in the kernel on CUDA hardware - this is why I said, it would take weeks to find an optimal computation with the least amount of branching instructions, otherwise it would just be days.

And so based on this and that Matt said, it will take the best part of this year, I guess there is a lot of other server-related stuff changed as well - not just the performance part - maybe better scalability, which would eventually be required for vet overland or other services like private instances - if they ever would want to go for that, they could then. So let's give them the time, it will better our gameplay in the end.

Overall, an interesting comment. Some details worth mentioning... Right now, CUDA hardware is like Santa Claus; some people think this exists, some know it doesn't, and no one has seen it. It could take years to get enough CUDA hardware to make a difference.

The other thing is that i find that player-sourced time estimates are wildly inaccurate, mainly because they deal with estimates about something that even seasoned professionals in the field are not well versed on. ZOS is the expert on what they are doing, though.

To add, ZOS is not replacing the server "engine". Based on the description, they are working on a specific part of it, and that part will fit nicely into the rest of the "engine", which they are not touching. Any design has to take that into to account, but it also has to be tested, and tested well. My thinking is that testing and fixing bugs is going to take the bulk of the time. Even then, once it gets on live, there will be bugs to fix. No QA team can replicate the hammering that a million players will do over the same amount of time.

Anyway, while we know they won't be using CUDA, i will bet the farm that their use of CPU cores and threads is less than optimal and that there is some room for improvement there. Anywhere that there is a single thread bottleneck is going to hurt. CPU single core performance improvements are not matched by how fast they are adding cores.

TequilaFire

I thought CUDA was only compatible with Nividia GPU cores which would leave AMD and console players out.
Although I admit I haven't checked lately but I thought OpenCL was the only choice on AMD.

Elsonso

TequilaFire wrote: »

I thought CUDA was only compatible with Nividia GPU cores which would leave AMD and console players out.
Although I admit I haven't checked lately but I thought OpenCL was the only choice on AMD.

What was being suggested, as I interpreted it, was that ZOS would use NVIDIA GPUs on the server-side and offload client/player/character related tasks to them.

(Edit... a bonus to doing that is that ZOS could farm the latest hot digital currency when the servers were not busy... )

TequilaFire

Makes more sense to optimize server code they have now before the expense of specific hard to get at the moment hardware.
Then there is the whole TCP vs UDP problem.
One could also invest in better anti-cheat software for PC which is the pain point.
I do believe they are on the right track now, time will tell.

Lysette

TequilaFire wrote: »

I thought CUDA was only compatible with Nividia GPU cores which would leave AMD and console players out.
Although I admit I haven't checked lately but I thought OpenCL was the only choice on AMD.

No, I was talking of a server-side computation on a CUDA device - as in a special Nvidia graphic-card with no graphical output - those are purely used to do massively parallel computation on them. What I suggest would be a server-side solution and basically get performance like it was after launch of the game or even better - and as far as the availability of those is concerned - a company which is under the roof of microsoft should not be able to get decent CUDA hardware - that is ridiculous, Nvidia will certainly be supportive with this.

TequilaFire

But that is still trying to compensate for inefficient code with more computational power.
How about a couple of racks of Raspberry Pi. lol I am joking.

Lysette

Elsonso wrote: »

TequilaFire wrote: »

I thought CUDA was only compatible with Nividia GPU cores which would leave AMD and console players out.
Although I admit I haven't checked lately but I thought OpenCL was the only choice on AMD.

What was being suggested, as I interpreted it, was that ZOS would use NVIDIA GPUs on the server-side and offload client/player/character related tasks to them.

Yeah, this - basically what was on client machines after launch and what they have taken back to the server - what was a good step, but now it lacks parallel computation there - but this can be achieved as well, if the parallel workload is put on server-side CUDA hardware, which is designed for this task - and doesn't need graphical output, but is purely for massively parallel computation. It will be running non-stop, so it has to be something really professional, not a consumer market graphic-card - but professional high duty CUDA hardware.

The tricky part of that is not getting it on the CUDA hardware, but to find a good enough data representation and kernel code, which can do the computation while avoiding too many branching instructions - you can imagine that a combat system which has so many conditions to check, has there it's bottleneck - this needs some clever data and code design - and that is why those developing this need to know at start, if their software will run and use actual parallel computation hardware or be just another workaround on normal server blades.

TequilaFire

Interesting concept but not economically desired as hardware is a recurring forever charge whereas correcting the server code is expensive but a one time expense with the expected bug chasing cost built in to the proposal to the suits.
Also there is the redundancy requirement which even further inflates the cost.
Anyway glad it's not my job to have to solve this on a live system. lol

Lysette

TequilaFire wrote: »

But that is still trying to compensate for inefficient code with more computational power.
How about a couple of racks of Raspberry Pi. lol I am joking.

Because that is the way, in which something like this can be solved - if you have samey computation on a lot of datasets, you put that on a SIMD device - same instruction done in parallel on multiple data-sets - just like a graphic-card is doing it with graphical content, the same can be done as well with other data - this is not a joke, this is how this is professionally solved.

It is the principle divide and conquer - this is how graphical things got solved - by splitting the image into smaller chunks and compute all those smaller chunks in parallel with the same instructions on the graphical hardware - otherwise you would never get that high performing graphics, if it wouldn't be done like this.

Lysette

TequilaFire wrote: »

Interesting concept but not economically desired as hardware is a recurring forever charge whereas correcting the server code is expensive but a one time expense with the expected bug chasing cost built in to the proposal to the suits.
Also there is the redundancy requirement which even further inflates the cost.
Anyway glad it's not my job to have to solve this on a live system. lol

then they can forget about it from the very start, because it will not produce the results - the solution is in massive parallel computation of these tasks - which they did before in parallel on client-machines - so it will work, if they go for a true massively parallel solution on hardware designed for this task.

TequilaFire

Yes I agree that it is one way it can be solved but not the only way and is the least likely thing that will happen.
Also do any of us have a hardware map of the existing Megaserver?
There may be even simpler fixes for all we know.
It's up to ZOS, so I will escape the loop! lol

herman.urrejolaub17_ESO

[Please do not delete people comments. Is just to cover the sun with a finger/i]
TESO is a great game but there are so many issues in the PVP experience that makes it unplayable for moments. I live in south america and im kind of used to huge amounts of lag in this game even though my internet connection is very good. People is upset because there was no real intention from game developers to fix this until now...

ArchMikem

Murky_Dregs wrote: »

It's long past time that you upgrade your architecture. It's good to hear that plans are finally in work.

I've run many program, project, & development teams in my career ($600m+portfolio), and the offhand comments made by Rich's wife speak volumes regarding the mindset of the ZoS team.

Her remarks were incredibly candid and are definitely a real glimpse into how your people feel.

Given the glib customer disregard displayed, how do you plan to address that issue, meaning your team's professional resolve regarding the matter, in order to ensure delivery of the quality product that many of us pay well over hundreds of dollars a year for and deserve?

Since when are the comments of an Employee's Wife taken as official representation of that Employee's company?