What Hardware and Backend Sofware Solutions Were Explored for Cyrodiil?

bellanca6561n

I'm a former Kesmai guy and we created some very good backend libs for high user density MMO environments long ago. The project was called Aries I believe but the dominant limitation in those days was hardware. Best we could manage and have it all make sense to players was 500 in one battle. That was in the mid-90s and our efforts would likely seem quaint today.

Also, this was for the very first MMO, Air Warrior. Not for an MMORPG with so, so many player skills employed.

However, I'm curious and could find no discussion on this when I did a forum search. In seeking to improve Cyrodiil performance, what backend software solutions and hardware solutions did you explore before devising these tests?

ShawnLaRock

The entire premise of the tests is to see how much further they can push this decrepit engine past it’s long-expired viability.

S.

trackdemon5512

Three years. I give the game three years before the current gen versions are dropped, next gen is fully embraced, and a new engine is in place.

Current versions are hobbled by the need for base PS4 and base XBOX ONE to fully play the games. Take those limiters off and oh the places you’ll go.

Jaraal

ShawnLaRock wrote: »

The entire premise of the tests is to see how much further they can push this decrepit engine past it’s long-expired viability.

S.

But Cyrodiil ran much smoother with a lot more players a few years ago.

What happened?

Akamai happened.

Tandor

trackdemon5512 wrote: »

Three years. I give the game three years before the current gen versions are dropped, next gen is fully embraced, and a new engine is in place.

Current versions are hobbled by the need for base PS4 and base XBOX ONE to fully play the games. Take those limiters off and oh the places you’ll go.

That's certainly an interesting scenario.

Or, of course, they could just drop support for Cyrodiil...

thegreat_one

Kesmai!!! I played Aliens Online from Gamestorm. My first MMO.
Yeah you could have 200 players in one server and it turned it into a chatroom. Funny how those lessons are still not learned.

Dropping Cyro is not something Zos is willing to entertain.
That much I know.

There is a market for it. Even if its small. Also that would confirm their utter failure in that aspect.

How would that look?

The problem is simple. Yet cannot be fixed. The population today compared to Cyrodiil release has increased. Along with skill and classes that were not incorporated at Launch.
necromancer was planned as it has been in the code forever.

If they had to upgrade would that mean brand new server? Would that mean a complete wipe to migrate the game?
I have seen it happen to games in the past.

The choices are not good.
And considering the uproar over refarming vMA i'd say it would not go over well.

The_Old_Goat

Unfortunately you will not get an educated answer to this. Zenimax just doesn't discuss matters like this, or pretty much anything else with their customer base.

bellanca6561n

Yeah, I know it was rather naive of me to pose a question like this in a public forum but I felt it needed asking and the folks I knew who used to work there have moved on long ago.

Kesmai founder, Kelton Flinn, the guy who first created online games, along with John Taylor, said back in '91...

"The client is in the hands of the enemy."

I'm not sure how much server-side verification was employed at launch of ESO. But it's clear more was added, and this has been perhaps the major difference between the game then and later. And it has been the focus of many combat changes - reducing calculations, server-side.

And, yes, as many of you noted, they add up....



Sadly, for me, the only part of the game that interests me is Cyrodiil. And, as you all know, it's the most problematic.

So I had to ask. Not in a sarcastic way, not in the form of a slam, no tone of disrespect. No developer wants their game to perform poorly.

bellanca6561n

thegreat_one wrote: »

Kesmai!!! I played Aliens Online from Gamestorm. My first MMO.
Yeah you could have 200 players in one server and it turned it into a chatroom. Funny how those lessons are still not learned.

In that particular case, it was Mythic that created that for us. That was back when Mythic was a small shop before someone finally gave Mark Jacobs money and they made Dark Age of Camelot. The Realm v Realm system in that game was simply outstanding....designed, I believe, by the fellow who later founded ZOS

DigitalHype

bellanca6561n wrote: »

thegreat_one wrote: »

Kesmai!!! I played Aliens Online from Gamestorm. My first MMO.
Yeah you could have 200 players in one server and it turned it into a chatroom. Funny how those lessons are still not learned.

In that particular case, it was Mythic that created that for us. That was back when Mythic was a small shop before someone finally gave Mark Jacobs money and they made Dark Age of Camelot. The Realm v Realm system in that game was simply outstanding....designed, I believe, by the fellow who later founded ZOS

Yup. 3 faction AvA has some corollary with the RvR system from Mythic. I remember reading, many years ago, that ESO was actually built upon a design/engine that was originally planned for a DAoC iteration that never came to fruition.

deleted221106-002999

I expect they explored the use of vacuum tubes but were likely too concerned that such technological marvels would be too advanced to utilise.

Gorreck

bellanca6561n wrote: »

I'm a former Kesmai guy and we created some very good backend libs for high user density MMO environments long ago. The project was called Aries I believe but the dominant limitation in those days was hardware. Best we could manage and have it all make sense to players was 500 in one battle. That was in the mid-90s and our efforts would likely seem quaint today.

Also, this was for the very first MMO, Air Warrior. Not for an MMORPG with so, so many player skills employed.

However, I'm curious and could find no discussion on this when I did a forum search. In seeking to improve Cyrodiil performance, what backend software solutions and hardware solutions did you explore before devising these tests?

They seemed to do something for Mid-Year mayhem, but never elaborated on what.

Gorreck

Tandor wrote: »

trackdemon5512 wrote: »

Three years. I give the game three years before the current gen versions are dropped, next gen is fully embraced, and a new engine is in place.

Current versions are hobbled by the need for base PS4 and base XBOX ONE to fully play the games. Take those limiters off and oh the places you’ll go.

That's certainly an interesting scenario.

Or, of course, they could just drop support for Cyrodiil...

"could"?

bellanca6561n

DigitalHype wrote: »

Yup. 3 faction AvA has some corollary with the RvR system from Mythic. I remember reading, many years ago, that ESO was actually built upon a design/engine that was originally planned for a DAoC iteration that never came to fruition.

As Air Warrior was. And the three factions were Red, Blue, and Yellow with the goal of capturing enemy airfields by, first, cutting off their support facilities. Sound familiar? Mark Jacobs was Gunsmoke in Air Warrior.

The big difference between the game I believe they meant to create with ESO, and the game they had to create to make monetary sense was a larger change in both industry and audience.

Air Warrior was a zero loot game. Everyone had access to the same aircraft. There was no religious element of unlocking addition things through devotion. It was purely an action MMO with no PvE. It was never meant to be mass market and could never be. That wasn’t the point.

But just look at the costs of development over the past 20-30 years. The first MMORPG to have a major market impact was Ultima Online. Its budget: 230k. Its biggest fixed cost: bandwidth. Thus they had to try to coerce players to limit their playing time with strategies like The Power Hour where your character could only gain for one hour a day.

By the time you get to an ESO, you’re talking 100 million dollars and perhaps multiples of that. Bandwidth costs are no consideration compared to Key Performance Indices (KPI) of Daily Average Users (DAU) Monthly Average Users (MAU) and similar metrics. In an essentially free to play world, the more a player is online, the more likely they are to buy something, be it Crowns, DLCs, and so on.

And to fit that model you need a loot game.

ESO is two games really. A limited multiplayer loot game, and a massively multiplayer action game. The loot gamers play for loot, the MMO player plays the loot game to play the game with no loot. Not directly. They play for the dynamic challenge and the emotion that delivers.

Sadly, these are often in conflict and prone to grave misunderstandings.



And monetization heavily favors the loot game.

In the loot game, the death of a single player is an emergency. Hence when you show a loot game player a common scene the action game like this….



…they often find it keenly disturbing.

In short, I understand the emphasis of ESO. I may not like it. It’s not the reason I’ve played online games for over 30 years. But I’m not the market. Far from it!

Sorry for the long post

max_only

If they could figure out a way to monetize Cyrodiil (to finally reel in the player base who only play for pvp which from an accountants POV is a loss for the company) that would hopefully incentivize them to make the servers better/ add more.
Maybe load screen advertisements while in Cyrodiil for non subscribers? I don’t currently subscribe btw.

TineaCruris

max_only wrote: »

If they could figure out a way to monetize Cyrodiil (to finally reel in the player base who only play for pvp which from an accountants POV is a loss for the company) that would hopefully incentivize them to make the servers better/ add more.
Maybe load screen advertisements while in Cyrodiil for non subscribers? I don’t currently subscribe btw.

Every hard core PVP player I know has sunk $1,000 plus into the game grinding for gear, mount they want from crown crate etc. Most I know have spent in excess of $3,000 and higher. It's not accurate to claim that PvP does not generate $$$. The end game PvP players are whales a lot of the time when it comes to crown crate spending and other spending on the game. This is why it's so shocking PvP gets dumped on and ignored so harshly.

TequilaFire

I would pay $15.00 a month per account for a Cyrodiil that works without destroying game play.
Oh wait, I already do...

zaria

trackdemon5512 wrote: »

Three years. I give the game three years before the current gen versions are dropped, next gen is fully embraced, and a new engine is in place.

Current versions are hobbled by the need for base PS4 and base XBOX ONE to fully play the games. Take those limiters off and oh the places you’ll go.

Yes, pc requirements has increased plenty from release. They had an 32 bit client back then.
And yes expecting them to drop PS4 and xboxx within 2 years and raise the pc minimum spec to match.
Now this is a bit unfair for players who only do overland.
On the other hand dolmen zergs, dragons and harrowstorms is get into trial level hardware demand.

Again dynamic scaling back effects would solve a lot.

bellanca6561n

TineaCruris wrote: »

max_only wrote: »

If they could figure out a way to monetize Cyrodiil (to finally reel in the player base who only play for pvp which from an accountants POV is a loss for the company) that would hopefully incentivize them to make the servers better/ add more.
Maybe load screen advertisements while in Cyrodiil for non subscribers? I don’t currently subscribe btw.

Every hard core PVP player I know has sunk $1,000 plus into the game grinding for gear, mount they want from crown crate etc. Most I know have spent in excess of $3,000 and higher. It's not accurate to claim that PvP does not generate $$$. The end game PvP players are whales a lot of the time when it comes to crown crate spending and other spending on the game. This is why it's so shocking PvP gets dumped on and ignored so harshly.

Well, so have I, plus I've always either played with Plus or taken a break totally. But when you look under the hood of free to play games, even quite casual ones, it's shocking...shocked me at least. The tiny percentage paying not just the sums you note over years, but monthly is the largest revenue source.

Now, I don't want to get into the moral aspects of this. Game publishers are not akin to the payday loan industry. And there are myriad reasons why Cyrodiil makes business sense.

What distresses me is how a business model, that any publisher would be foolish not to embrace, has altered the relationship between developer and customer as well as the relationship among customers themselves.

No, it's not the FB model where the customer is the product and not a customer at all.

Rather, it's a fundamental change in what the product itself is. It's the gradual, slight, imperceptible change in player behavior and perception that is the product here. You arrive wanting one thing but end up locked in a cycle of wanting something else entirely.

The problem with the MMO action game in ESO is that it falls outside that desired cycle and requires backend resources the loot game's addiction cycle does not.

bellanca6561n

Alas, no answer is an answer meaning either:

• No, we’re not investing is more capable hardware or software to address the issue, or…
• You, as a customer, do have the standing to expect or receive an answer.

The pity of it is we’re in the middle of some celebration of the community. Tamriel Together.

Yet, this second, lengthy beta has been keenly corrosive to the community dedicated to the massively multiplayer portion of this game.

One canary in that coal mine is the resignation of perhaps the most steadfast and durable guild leader over the years, Lord Bashu, founder of the Rough Riders, a DC loyalist PvP guild based on non-CP. No, he’s never been a great individual player. Yet he’s among that one-tenth of one percent of players with the rarest of player qualities: the moral authority and indefatigable determination to build enduring persistent communities, first in Ultima Online and, until today, here in ESO.

Here are the final 30 seconds of the first ESO beta in 2014.

https://www.youtube.com/watch?v=hnFZK_OLe04

I believe the reaction to the end of the current beta will be somewhat different.

Pardon the negative tone. I just can't help it atm, sadly.

Raideen

bellanca6561n wrote: »

Pardon the negative tone. I just can't help it atm, sadly.

I read ever post in this thread starting at the top. I never ONCE looked at your post as being negative, just honest.

idk

I would not expect Zos to divulge details of the backend of this system we play on.

The real issue is twofold. Zos pushed the envelope of a mega-server and seems to have bitten off a little more than they could build out properly. This is evident with the overall performance of the game since launch and most specifically how Cyrodiil has performed.

This baseline performance has degraded over time because Zos has asked the server to handle more and more over the years by adding more buffs and debuffs and other calculations while also moving client-side checks to the server-side. All of this has been done without making meaningful changes to accommodate the performance of the game.

This brings us to where we are now. The only MMORPG I have ever seen where performance has diminished over time.

systema555

Most intelligent & insightful thread I have read on the subject since many moons...

Assuming ZOS did setup their community management teams correctly that thread should reach the Architecture team.

ESO is at a crossroad, we all know that. The next 6 months will define the future of this game.

Its just time...

vvvvv

Alendrin

Backend changes cost money, and Bethesda has shown a lack of interest in investing in their products, or for that matter even selling working products, over the last few years. This isn't limited to just ESO.

gatekeeper13

ZOS is trying to find a way to reduce lag through reduced server load in order to avoid moving to better servers.

I am not buying the "but it's an old game engine" excuse. In the last MYM event, Cyro was pop locked 24/7 and on the same game engine it worked like a charm. No dc, no lag, nothing. Massive battles everywhere and performance was 3 times better than a 10 v 10 battle today. After the event ended, lag and dc returned.

OOJIMMY

thegreat_one wrote: »

Kesmai!!! I played Aliens Online from Gamestorm. My first MMO.
Yeah you could have 200 players in one server and it turned it into a chatroom. Funny how those lessons are still not learned.

Dropping Cyro is not something Zos is willing to entertain.
That much I know.

There is a market for it. Even if its small. Also that would confirm their utter failure in that aspect.

How would that look?

The problem is simple. Yet cannot be fixed. The population today compared to Cyrodiil release has increased. Along with skill and classes that were not incorporated at Launch.
necromancer was planned as it has been in the code forever.

If they had to upgrade would that mean brand new server? Would that mean a complete wipe to migrate the game?
I have seen it happen to games in the past.

The choices are not good.
And considering the uproar over refarming vMA i'd say it would not go over well.

How has the population increased since launch? They reduced to population cap and still usually one campaign is pop locked and the other one hit it miss of it locks.

There use to be more campaigns and almost all of them would be pop locked. I remember having to wait literal hours in que.

Gorreck

gatekeeper13 wrote: »

ZOS is trying to find a way to reduce lag through reduced server load in order to avoid moving to better servers.

I am not buying the "but it's an old game engine" excuse. In the last MYM event, Cyro was pop locked 24/7 and on the same game engine it worked like a charm. No dc, no lag, nothing. Massive battles everywhere and performance was 3 times better than a 10 v 10 battle today. After the event ended, lag and dc returned.

Yep MYM did seem to show they can do something to fix it, they just (for whatever reason) don't for most of the time.

gatekeeper13

Gorreck wrote: »

gatekeeper13 wrote: »

ZOS is trying to find a way to reduce lag through reduced server load in order to avoid moving to better servers.

I am not buying the "but it's an old game engine" excuse. In the last MYM event, Cyro was pop locked 24/7 and on the same game engine it worked like a charm. No dc, no lag, nothing. Massive battles everywhere and performance was 3 times better than a 10 v 10 battle today. After the event ended, lag and dc returned.

Yep MYM did seem to show they can do something to fix it, they just (for whatever reason) don't for most of the time.

They probably bought extra server capacity for the duration of the event.

EmEm_Oh

TineaCruris wrote: »

max_only wrote: »

If they could figure out a way to monetize Cyrodiil (to finally reel in the player base who only play for pvp which from an accountants POV is a loss for the company) that would hopefully incentivize them to make the servers better/ add more.
Maybe load screen advertisements while in Cyrodiil for non subscribers? I don’t currently subscribe btw.

Every hard core PVP player I know has sunk $1,000 plus into the game grinding for gear, mount they want from crown crate etc. Most I know have spent in excess of $3,000 and higher. It's not accurate to claim that PvP does not generate $$$. The end game PvP players are whales a lot of the time when it comes to crown crate spending and other spending on the game. This is why it's so shocking PvP gets dumped on and ignored so harshly.

Yep. Including ESO+, I've spent around $1000. And you're right, no way pvp isn't generating income.

TineaCruris

They still haven't told us what they did during the midyear mayem event that pretty much cleared up most of the issues.

Why isn't doing what ever that was an option or a test?

https://forums.elderscrollsonline.com/en/discussion/538043/why-was-performance-better-during-mym-event-and-now-in-the-trashcan-again#latest

The AOE's they are claiming causing the issues have been in the game since inception. The lag has not, and there used to be triple the players in cyrodiil when performance was much better.

TwinLamps

Answer is prolly close if not the cheapest thing on market, both for hardware and software.