THE MULTIPLE TRADER BID SOFTWARE WAS PROPERLY TESTED

chess1ukb16_ESO

jazsper77 wrote: »

Just checking to see how the NA PC Trader flip went?

Without a hitch!

Reverb

All day yesterday the EU server was circling the drain. While grasping at straws zeni turned off guild history polling. On a Sunday. Right before trader bidding closed. Check the Dev Tracker for more info.

Zeni has always always had a blind spot for how pulling one thread unravels other functionality. And we know that the EU environment can’t handle the number of players, and that the social aspects of the game are hardest hit by the performance shortcomings (grouping, guild functions, friends lists, chat channels). It should be assumed that the stop-gap measures they took yesterday for social functions also prevented the new multi bidding code from executing properly.

If I’m right, and I strongly suspect I am, there’s no good fix in the short term. Zeni will continue desperately plugging one small hole after another after another until the gaping hole at the bow is fixed. At least a year out.

marius_buys

EU server was circling the drain indeed

TriangularChicken

hope they are going to ban the people that exploited the situation...

vgabor

Guizan wrote: »

Then all bids have to be put in a list sorted from highest to lowest and then the top bid win its trader automatically. Once you win a bid all the rest of the bids of that guild will be removed from the list, and the gold for those bids will be returned via mail. (something that FAILED miserably this week BTW)

We also have to have rules what happens in case of a tie when the top two bids are the same amount and on the same trader and this I have not addressed here but it could be either random, or if ZOS want to drain as much gold as possible from the economy they give the bid to the guild with the lower second bid

Then you go to the next highest remaining bid to find out who gets trader #2, and repeat the same procedure until you are out of bids or all the traders are assigned.

If bids still remain after all traders are assigned those bids will be refunded to the bidders.

Kinda but not exactly.

You don't want to do external calls in the middle of the allocation because they might lag out and cause problems like it just happened yesterday, so you want to do all the trader allocation in one go and then record the results and then do all actions based on the results after.

You also do not want to change the bid list itself with removing elements, because finding the affected elements takes long time {it's an O(n) operation for each element so you end up with a O(n^2) overall} instead you want to use hashmaps for storing status information {which with the individual O(1) operations will be just O(n) overall}.

So how I would do:

1. Preparation (get your data ready)
   1. Load up all bids from the database, including relevant guild information which need for the allocation later {complexity O(n)}
   2. Make two assignment hashmaps: one for the guilds from the list {complexity O(n)} and other for all traders {complexity O(1) since trader list static and can be precompiled}
2. Trader allocation, where you do the bulk of the work, no external calls here!
   1. Sort the full bid list in decreasing bid order, break ties with secondary conditions. Reasonable conditions I would use: highest bid > trader with most bids (assuming the traders with more bids are more desirable) > guild with less bids (preferring guilds with focus on a particular trader) > guild with most members (preferring bigger guilds) > random (in case all previous happens to be the same, which is very unlikely) - all the required data for these should and can be loaded up and added into the hashmaps in preparation step, so you have all data already at hand while sorting {complexity is standard sorting complexity, generally O(n * log n)}
   2. Take the first bid in the sorted list, check if guild or trader already allocated using the hashmaps. If any of them already taken then flag the bid unsuccessful. If both are free, record the allocation into the two hashmap and flag the bid in the list as successful. Continue to the next bid with this until you done the list {complexity O(n)}
3. Closing steps after allocation, as when you finished with the above you will end up:
   • your trader hashmap will contain all the trader allocation (the unallocated traders will be the ones free to hire), so save this trader allocation to the database {complexity O(n)} and then activate it in the game {complexity O(1)} so they go live at once (note: I hope you have built the trader system such way that you can simply swap between two trader allocation config)
   • your bid list will contain all the successful/unsuccessful status of the bids, so go through the list now after everything else is done and settle the accounts {since all traders are now live, at this point does not matter much how long it takes, you have time to do the longer outcalls to external services like messaging}
   • your guild hashmap can be simply discarded, it was only for efficiency proposes to do the guild status check fast during allocation

This entire thing is a basic algorithmic design, which any graduates doing even a semi respectable university for software engineering should have done most likely in first year but at worst in second, so how you guys managed to blow it up in this spectacular way is beyond me...

Korah_Eaglecry

What the hell is with all these caps lock titles?

Its monday, is the moderators gonna get off their asses?

Dont_do_drugs

TriangularChicken wrote: »

hope they are going to ban the people that exploited the situation...

not only. i want a complete name list of all banned gm

kalunte

ZoS job is always done correctly. you are mixing up foked up mechanisms with ZoS features.

mook-eb16_ESO

any software that does this is a bot and people should be banned for using it.

Reverb

mook-eb16_ESO wrote: »

any software that does this is a bot and people should be banned for using it.

We all have our grievances with the zeni dev and support teams. But I think banning them would be an overreaction

Rake

Reverb wrote: »

mook-eb16_ESO wrote: »

any software that does this is a bot and people should be banned for using it.

We all have our grievances with the zeni dev and support teams. But I think banning them would be an overreaction

I think banning them is appropriate.
They exploited severe bug that affected entire server.
People were banned for far less in ESO.

Knootewoot

It was properly tested by @zos standards.

Cously

This kinda of change needed a "live" test with someone from ZOS monitoring the situation Sunday night.

AlienSlof

One more thing in a long list of stuff not properly tested - including easy to test things like costumes...

Delparis

Russian Hacker are the best obviously, ZoS devs are completely clueless

OsManiaC

Knootewoot wrote: »

It was properly tested by @zos standards.

yes they even opened a topic in pts section so people can write feedbacks. good for them

agegarton

In response to the OP......

Your Guild almost certainly did not lose to the Guild that is now sitting in the Trader Kiosk. Here's what actually happened:

1. You bid
2. You were outbid when the bids were counted
3. The Guild that won your kiosk could not take up their place in the Kiosk because of some kind of server glitch
4. During the extended gap in time between the bids being counted and the rightful bid winner stepping into their kiosk (created by this server glitch), other Guilds were able to pick up the trader at the minimum fee as if they were vacant
5. When the server issue subsided, there were already Guilds in kiosks, so rightful winning bidders could not step into their kiosk places.
6. Disaster ensues.

So, in short, winning bidders were often left with no trader. Some GM's acted very quickly to take their spot as if it were vacant, but most were not able to do that. Most kiosks were taken by guilds that were "chancing their arm" waiting for a trading spot to come free.

The server glitch was almost certainly a load-based problem. We know the NA servers handled the load OK and we can assume it's the same code (and we can assume that the code was not changed between the EU and NA trader switch as there would not have been time). This is thematic of this year's server issues - they have been exclusive to EU, and we know from ZoS themselves that EU servers have been under too much load pressure. That also goes some way to explaining the multiple guild bid refunds - the servers got stuck in a cycle refunding over and over again. I myself placed 2 bids, but was refunded 7 times (which makes no logical sense). That's typical "overload" behaviour.

Hope that helps. I promise it wasn't Russians.

Grimm13

MartiniDaniels wrote: »

It is not software it is server load. Exactly the same software dealt with multi-bidding properly, on PC NA.

The process was altered for the NA server trader flip. Why do I say this, observation.

I was standing at my guilds trader at the flip. Saw the tabbards disappear and the info of hired trader also cleared on my home guild home page. The tabbards as easy to refresh as you just need look away then back, the home page just close and reopen.

So since I saw both were gone and knowing how EU went. I tried to hire our now vacant trader but when I clicked the hire button received a message which I will paraphrase as I did not screen shot it, "Failed to Hire as Trader resolution is still ongoing." it was a little different than that but is what it meant.

Took about another 30 seconds and the Home page updated saying I had won a different Trader so I left it to go verify in that zone. Found that it was true and looked at refund logs, saw all amounts correct.

To me it looks like they did a lock down of the Traders until the flip was complete which is as it should have been in the first place. I would say it is a combination of the software and server load, several things were overlooked.

PTS trader bidding system needs an adjustment to increase to three flips in a week instead of a single flip. The four flips was not enough to see flaws. Not enough people went to PTS to place bids, ZOS has no way to make a Guild on the fly in PTS, needs a short cut instant build. ZOS also needed to have created 15 to 30 bids on each Traders to simulate the lacking guild bids.

After the first flip it was obvious that it was not capable of testing correctly. ZOS ignored us and our concerns and after much pushing we got a single response which did nothing to address our concerns.

Hopefully this shed some light as to how it viewed to me on NA server. Maybe it next flip on EU will be better.

What I want to see from ZOS is a response as to how they are adjusting the flip process going forward. yeah I probably pissed them off again but hey I tried to get them to see they had flaws and ZOS chose to ignore all of us.

@ZOS_GinaBruno @ZOS_JessicaFolsom @ZOS_PhilipDraven

DaveMoeDee

It worked fine on PC/NA.

The reality is that software these days, more often than not, is not tested at the proper scale and in all configurations that it will be deployed to. Even worse, some features in SaaS that appear to customers to be part of the software are actually hacks put together by operations with less rigorous testing and less controlled change management.

Then there is the problem of ad hoc fixes. For example, when people say they should look at logs and roll back specific changes/transactions, that is a manual process. I don't know if the bid refunds were part of the software or if someone ran some script to refund people who were charged for bids they didn't get. If it is the latter, that will never be tested to the same degree as game updates. Sometimes it can be extremely hard to fully test such scripts because they might depend on details that only exist in a particular implementation and scale might impact how well it runs.

Here is a simple example. A script might have a bunch of SQL queries. Any query that returns a list of results could cause a problem if the data is so large that the query errors out. If the script didn't check properly that each query finished without an error, the script could interpret the empty result from the failed query as a valid data set containing 0 items. The script could continue on and really bad things can happen. That error would never occur during testing that isn't at a large enough scale for the query to fail.

And in a fast-paced SaaS environment with customers waiting for quick fixes and employees are often dealing with multiple tickets at the same time, typos can happen. You might think that people shouldn't multitask, but do you expect the employee to just sit there waiting when they kick off something that requires 10 minutes to complete? Context switching can lead to blunders. I remember dropping a table in a production database instead of on the server we were setting up to migrate to. I just lost track of which window what which. I also remember I once had some SQL I had used countless times to insert records into a table. In this instance, I was going to compare a column to a variable instead of a literal number to filter down the number of values to insert. Instead of id=@id, I typed id=id. The DBA was very busy the next morning removing hundreds of millions of records.

xenowarrior92eb17_ESO

im more curious how ZoS is dealing with the people that became billionaires over night and spit in the face of the rest of us who aren't guild owners and worked like slaves for our millions of gold...

DaveMoeDee

There are people saying it isn't the software, it is the server load. I'm sorry but server load isn't going to change the data on its own. Either the software is changing the data or someone is doing an ad hoc script to make changes. Another possibility is that the database solution itself does crazy things when it has resource issues, but it is hard to imagine that this outcome could have resulted from that.

The software needs to handle the exceptions that load can cause. Exception handling is part of software. Code that works fine with no exceptions but does crazy things when exceptions happen is bad software. It is far better for code to just crash from an exception than to start writing bad data.

The problem with dealing with databases is that if you fail to check for error codes, you can easily mistake the results of a failed query for good results. You could end up working with a partial data set that you think is complete. You could end up with an empty data set that you think is the result of your query but is the result of the query failing and giving you nothing. Decent software always checks for error codes before using results. While it can be hard to test test for errors that occur at scale, there are ways. For example, instead of querying the actual database, I can query a fake "database" created for testing that sends back an error.

Yet, even if we simulate every possibility we can think of, we are unlikely to anticipate all possible events. Someone or something needs to ensure that all possible exceptions are handled. While we don't know all possible events, we can know all possible exceptions (unless we are dealing with software that lacks documentation, which is sometimes the case).

Tan9oSuccka

Chill. Let them figure it out.

All the amateur IT people in these threads. This may be hard for you guys to fathom, but not everything can be captured in a test environment.

zsban

I work as a software architect. At my work we can not afford such hiccups as it would have major consequences. And yet, we manage it. It's not difficult, if you have the correct controls and gates in place.

There are so many things wrong in your example:

DaveMoeDee wrote: »

It worked fine on PC/NA.
Here is a simple example. A script might have a bunch of SQL queries. Any query that returns a list of results could cause a problem if the data is so large that the query errors out. If the script didn't check properly that each query finished without an error, the script could interpret the empty result from the failed query as a valid data set containing 0 items. The script could continue on and really bad things can happen. That error would never occur during testing that isn't at a large enough scale for the query to fail.

• poor design (list of results could cause a problem if the data is so large)
• inadequate testing (no simulation of what happens if a query errors out)
• inadequate code review (if the script didn't check properly that each query finished without an error)

DaveMoeDee wrote: »

"That error would never occur during testing that isn't at a large enough scale for the query to fail"

Wrong. The issues in the above example can be caught and tested with synthetic tests that do not require full integration to a QA environment.

DaveMoeDee wrote: »

Context switching can lead to blunders. I remember dropping a table in a production database instead of on the server we were setting up to migrate to. I just lost track of which window what which.

Very bad controls around segregating environments. You should never be able to run these kind of things directly in prod. You should have the script ready and validated in a lower environment (peer review, test runs, etc), and then systematically promote it to a higher environment, not allowing for any manual change in between to the script.

DaveMoeDee wrote: »

I also remember I once had some SQL I had used countless times to insert records into a table. In this instance, I was going to compare a column to a variable instead of a literal number to filter down the number of values to insert. Instead of id=@id, I typed id=id. The DBA was very busy the next morning removing hundreds of millions of records.

It's exactly the same issue as above. If correct controls would have been in place, that script would have been first peer reviewed, validated in a lower environment which you can just throw away and rebuild with a push of a button if something blows up, and once confirmed that it works, only then would have been promoted to production.

What I see a problem in the industry is that many companies are not willing to accept that these are neccessary things (unless they are enforced by regulations in their industry) because they try to cut costs everywhere. Also many couch developers are simply not thinking about supportability, maintanibilty and manageability of their work, they just botch something together, hoping "it will be fine".

This is especially true in the gaming industry where they don't really have to design any critical systems... The "worst" that can happen is some bugs in their game that they will fix in the next patch. Or so they think...

OsManiaC

@DaveMoeDee thanks for the info. I'll share our case;
- swap/flip not appeared on usual time, got delayed
- as you know, all kiosks available for free if no one bids it, and all kiosks available for 10k on their usual time.
- ı realized something was going on wrong and immediately grab one.
- ı started spamming in chat to get a trader now as my friends still waiting bids to flip,
- swap/flip or multibidding case started after several seconds, the ones that acted under 1 minute got a trader before they got winning message
- after you got the trader shown in your motd page, game not letting you grab a free one which they fixed yesterday

edit the screenshot https://i.redd.it/a7d2g82nfah31.png

DaveMoeDee

zsban wrote: »

I work as a software architect. At my work we can not afford such hiccups as it would have major consequences. And yet, we manage it. It's not difficult, if you have the correct controls and gates in place.

There are so many things wrong in your example:

DaveMoeDee wrote: »

It worked fine on PC/NA.
Here is a simple example. A script might have a bunch of SQL queries. Any query that returns a list of results could cause a problem if the data is so large that the query errors out. If the script didn't check properly that each query finished without an error, the script could interpret the empty result from the failed query as a valid data set containing 0 items. The script could continue on and really bad things can happen. That error would never occur during testing that isn't at a large enough scale for the query to fail.

• poor design (list of results could cause a problem if the data is so large)
• inadequate testing (no simulation of what happens if a query errors out)
• inadequate code review (if the script didn't check properly that each query finished without an error)

DaveMoeDee wrote: »

"That error would never occur during testing that isn't at a large enough scale for the query to fail"

Wrong. The issues in the above example can be caught and tested with synthetic tests that do not require full integration to a QA environment.

DaveMoeDee wrote: »

Context switching can lead to blunders. I remember dropping a table in a production database instead of on the server we were setting up to migrate to. I just lost track of which window what which.

Very bad controls around segregating environments. You should never be able to run these kind of things directly in prod. You should have the script ready and validated in a lower environment (peer review, test runs, etc), and then systematically promote it to a higher environment, not allowing for any manual change in between to the script.

DaveMoeDee wrote: »

I also remember I once had some SQL I had used countless times to insert records into a table. In this instance, I was going to compare a column to a variable instead of a literal number to filter down the number of values to insert. Instead of id=@id, I typed id=id. The DBA was very busy the next morning removing hundreds of millions of records.

It's exactly the same issue as above. If correct controls would have been in place, that script would have been first peer reviewed, validated in a lower environment which you can just throw away and rebuild with a push of a button if something blows up, and once confirmed that it works, only then would have been promoted to production.

What I see a problem in the industry is that many companies are not willing to accept that these are neccessary things (unless they are enforced by regulations in their industry) because they try to cut costs everywhere. Also many couch developers are simply not thinking about supportability, maintanibilty and manageability of their work, they just botch something together, hoping "it will be fine".

This is especially true in the gaming industry where they don't really have to design any critical systems... The "worst" that can happen is some bugs in their game that they will fix in the next patch. Or so they think...

I don't disagree with anything you said. I am actually giving examples of horrible practices resulting from people taking shortcuts to get things done. I am just pointing out that either (1) the testing IS responsible or (2) someone running an ad hoc script is responsible.

Thing is, it often isn't about cost cutting. It is often about customers needing things that can't wait for the dev team. In my case, the court says client A needs to provide document set X by Friday or potentially be fined $100k. Do you wait for engineering to add a feature needed to get the proper documents ready? Not going to happen. Sometimes you have to decide if the risk is worth it.

I have experienced security changes needed for compliance that made it that people could not paste commands from approved tickets into the environment where changes needed to be made for months. That meant manually retyping commands. Eventually they improved the security situation. The sad reality is that the company had no choice though but to lock down the environment with all the news about companies having internal documents dumped online.

I remember a top consulting firm licensing our product that we also used in-house. They couldn't figure out how a certain vital part of the product could possibly work for us internally. The reason is that within operations we filled in all the gaps. Our fixes did not go through rigorous testing. What our software did though was keep customers from leaving. We automated manual tasks that were far more error prone than our insufficiently tested software.

zsban

DaveMoeDee wrote: »

I don't disagree with anything you said. I am actually giving examples of horrible practices resulting from people taking shortcuts to get things done. I am just pointing out that either (1) the testing IS responsible or (2) someone running an ad hoc script is responsible.

Thing is, it often isn't about cost cutting. It is often about customers needing things that can't wait for the dev team. In my case, the court says client A needs to provide document set X by Friday or potentially be fined $100k. Do you wait for engineering to add a feature needed to get the proper documents ready? Not going to happen. Sometimes you have to decide if the risk is worth it.

I hear you. In my case it's a complicated web of policies and regulations... And most likely the firm will say better be fined for $100k by missing the deadline but still following all policies and regulations rather than making exceptions, thereby breaching the policies and regulations in place due to external pressure, and potentially introducing risk. The potential fine for doing this could be way more than $100k.

saruhrose1

Have they officially or unofficially address the situation that happened on PC EU yet?

zsban

Yes. Paraphrasing: "Investigating, stay tuned."

AlnilamE

Grimm13 wrote: »

The process was altered for the NA server trader flip. Why do I say this, observation.

I was standing at my guilds trader at the flip. Saw the tabbards disappear and the info of hired trader also cleared on my home guild home page. The tabbards as easy to refresh as you just need look away then back, the home page just close and reopen.

So since I saw both were gone and knowing how EU went. I tried to hire our now vacant trader but when I clicked the hire button received a message which I will paraphrase as I did not screen shot it, "Failed to Hire as Trader resolution is still ongoing." it was a little different than that but is what it meant.

Took about another 30 seconds and the Home page updated saying I had won a different Trader so I left it to go verify in that zone. Found that it was true and looked at refund logs, saw all amounts correct.

To me it looks like they did a lock down of the Traders until the flip was complete which is as it should have been in the first place. I would say it is a combination of the software and server load, several things were overlooked.

"Unable to hire while traders change ownership"

There is a 5-minute lock on traders when they switch, as the query takes a minute to run. Some weeks we see your result almost instantly, sometimes it takes over 60 seconds.

This has always been there since the traders came out.

Edit: Messed up quote

Wolfpaw

I thought you pc players paid zos to beta test their game.

Sincerely, console peasant😎