Servers

[Beast]

thanks for the update qan, I appreciate you. Rab stop cherry picking.

[qan]

I don't really handle rule enforcement as a general rule, Rab, but I'll send it over to the guys who do.

[afghan]

Was able to get some time free during this week from hell, hopped on and the issue was occurring.

Got five MTR reports from users and backtraced all from the server. They're juicy ... definitely have some very interesting data there that our provider's techs can use. This should be a big step forward.

They've also escalated the ticket beyond their boneheads which ask you to turn it off and then back on again and we're now talking to their development staff.

thanks for the update and thanks for all your hard work on this!!

[ogron]

Was able to get some time free during this week from hell, hopped on and the issue was occurring.

Got five MTR reports from users and backtraced all from the server. They're juicy ... definitely have some very interesting data there that our provider's techs can use. This should be a big step forward.

They've also escalated the ticket beyond their boneheads which ask you to turn it off and then back on again and we're now talking to their development staff.

thanks for the effort, hopefully this can be solved soon.

TWDT Finals are this Sunday, and Basing Cup Finals next, so those are the two biggest risk factors in the short-term.

particularly TWDT, as that affects the largest swath of the zone and 5 of the 8 teams are still in it.

[Jessup]

3.1% c2s just now huge delays after I hit player to their death. Lots of ded on eats. So odd sigh it was working well a few days ago for me.

[Rab]

yeah it's fucked again.

[Grasps]

S2C: 4.3% C2S: 27.9%

[Jessup]

S2C: 4.3% C2S: 27.9%

[Beast]

Rofl

[qan]

Good news.

Yesterday after a server recycle, we were able to completely eliminate packetloss as confirmed by two very thorough two-way traces, one before and one after the recycle. It's quite strange, because the traces themselves were dirty and then extremely clean, suggesting that something the gameserver itself is doing is causing Windows Server to drop packets.

This means that we do have a temporary fix available. Recycling should resolve the issue. Additional troubleshooting will be needed to narrow things down further, but that's a big step forward. Right now my first inclination is that this is more related to number of players rather than uptime, but it's still in the early stages of figuring out exactly what's going on, and it's not really very straightforward at all, unfortunately.

I've made sure all members of upper staff have a means at their disposal to recycle as needed. On Sunday morning I'll also be doing a recycle in order to keep things clean for DT finals. It may be worth trying to recycle right before games begin, especially if there's the smallest hint of the issue coming up.

[Iron]

I am online, still 20% of packetloss

[Edify]

Holy PACKET LOSS ServerToYou:0.1% YouToServer:15.8%

Edit: 32.3% just keeps going up

[ogron]

might be worthwhile doing a recycle before the DT-D finals, then DT-J finals, then DT-B finals... the effects seem to wear off after a few hours.

[qan]

Interesting.

Something that would be really helpful here is to try and catalog what we know about this. I'm not the guy to do this; I didn't log on for a stretch of maybe 4-5 days last week and TW unfortunately has not been able to be my focus, with work consuming massive amounts of time and mental energy, leaving me completely drained in terms of putting real thought toward anything else. Sorry, but also not sorry. That's just how it is right now and this game is nobody's job.


So, what do we know?

• Recycling helps
• Recycling helps just for a few hours (how many?)
• Usually C2S packetloss with small amounts of S2C (confirm? The distinction is important)
• Happens more seriously to some rather than others, but when it's at its worst it seems to affect everyone
• Uptime doesn't appear to matter much, but number of players seems to be a significant factor
• Packetloss can be observed outside of the game, i.e., in a trace from user to server unrelated to SS. Recycling clears the issue.
• ?

Questions:

• How long after relogging does it take for the packetloss to appear in ?lag
• Are there times when the packetloss appears high in ?lag but not on-screen?
• Does this occur more often at a certain time than others? Can this time be in any way be separated as a variable from the number of players online? For example, is there any time when it occurs but there are no players on, and there hasn't been a large number of players on since the last recycle? (We may need to recycle more to investigate this.)
• Can the issue lie dormant/not be seen for long periods of time, such as during off-time, and then suddenly occur? (Thinking particularly when the zone has a low pop count and then many log on, particularly for scheduled competitive matches)
• (other useful questions)?

Also, nobody's going to like hearing this, but if the issue is exacerbated by large numbers of players online, one solution to keep things cleaner during competitive games other than shutting down unnecessary bots is to have spectatoes log off and watch via Twitch, etc.

[Rab]

[dense info, read carefully]

There's not a chance in hell this is down to the number of players. We feel the population is high because of all the old names logging on again, but the data says the population is not high - also factor in the number of bots sat around doing nothing most of the time.


To restate the obvious, we may have never loved our server, but it has only totally shit the bed since social distancing began. So it's reasonable to suspect that something to do with everyone being at home is behind it.

We know we have 2 issues, which get conflated in our minds and we need to remember they're seperate:
1. giant ping spikes - this is the windows update issue, affects all players, and is mostly mitigated by qan manually running updates.
2. high packetloss - this is the one that keeps disrupting leagues, affects some players massively more than others, and we have no solution for it.

I'm not convinced that recycling helps. How many times have we recycled to try and fix this? Has it worked every time? How many times has it happened without us recycling it? Did it resolve itself? My experience was this happened during TWDT a couple of weeks back, we were playing javs BO3 and game 2 was wrecked by lag, but we waited a bit and played game 3 without recycling, IMO it did seem to recover by itself. I suspect the recycling thing is a false-positive.

We know that changes in behavior have affected online services from giants like Netflix lowering quality, to not being able to place an online order with your local supermarket. I have experienced something like this before - when the UK's passport service was being ddos'd, my work shared network infrastructure with them, so we were affected too. We know our hosting is shared, and we have no idea who the server+networks are shared with.

qan - are you able to collect lag information automatically? E.g. a bot which, once per 15 minutes (for example) goes into each arena and runs ?lag on everyone there and saves the results to the database? With that data (e.g. dumped into csv) I could do something to visualise it. Columns needed: timestamp, name, ping current-min-ave-max, c2s, s2c. Columns nice to haves: country, time of last recycle. This data would be useful to have anyway, similar to how I scraped resolutions from the twd site to inform the res limit discussions, having lag data would inform lag limit discussions.

Let's say we had full lag data - what could we actually do with it? Probably not much, we would probably need to take the gamble of moving server, but then we'd have delays waiting for Priitk, so we probably need to be able to set up the new server while the old one is still running which means fast-tracking a new biller and we may as well go ASSS/Linux at the same time, even if some things like full bot functionality don't work initially. At least with lag data we'd be able to compare two servers to see if they're both affected, or just one, or at different times. Perhaps it's worth comparing TW and Chaos. Perhaps it's worth speaking to other zones to see if they've had issues.

Whether we take a server move project on (maybe the key people don't even have the free time for right now), or whether we rely on our hosting company to investigate (they probably have better things to do), we should expect no progress for a month or so. So we need to compare that to doing nothing for the same period of time. If we think this is likely caused by social distancing, well.. when is social distancing expected to end or reduce? At a guess I'd say about a month or so.

This is why I come at this with "capabilities" in mind. The capability to measure lag would be useful to have at any time, and may be useful now. The capability to move server quickly would be useful to have at any time, and may be useful now. So starting these things and not finishing them in time isn't a waste.