"Let's fix the lag" Thread - data-gathering

[qan]

There has been an issue with those who have normally very clean connections still getting packetloss. This seems to occur only after the server has been running a bit (hasn't been rebooted), but this isn't absolutely confirmed. If this is the case, we need to confirm it, because if so it's an incredibly simple fix that can be automated.

Throughout this DT season, the server was rebooted before games each Sunday. As a result of this, we've had a consistent starting point to work from. However, this Sunday, the server was not rebooted.

For those of you experiencing packetloss, please think back carefully and answer these questions as accurately as you can. If we can get enough answers, we can get closer to nailing this down. The fewer answers we have, the less likely it is that we can find a solution.

• Before a DT Sunday (Friday, Saturday, day-of) did you experience packetloss? How much? S2C or C2S?
• After the reboot, would the packetloss clear for you?
• How many days (or hours) on average would it take after Sunday for the packetloss to return?
• Does it usually return very quickly/all at once, or does it slowly get worse and worse over a period of time? If it slowly gets worse, what's the period of time it takes to get worse? And, does it seem to continue to get continually worse and worse the longer the server is up, or does it top out at some point?
• Is there anything else you think would be useful to describe the problem? (For example, times when it seems to be better or worse, arenas where it's worse, latency/ping issues, etc.)

Thanks for your help.

[Rab]

There was a day, either yesterday or the day before, where everyone had 2% C2S. Just to clarify that "this sunday the server wasn't rebooted" means "this sunday the server wasn't rebooted and the server fucked up".

Q: Before a DT Sunday (Friday, Saturday, day-of) did you experience packetloss? How much? S2C or C2S?
A: Around 2% C2S. I think everyone was affected.

Q: After the reboot, would the packetloss clear for you?
A: Usually a reboot is preventative, so there's no packetloss to clear. When there is high packetloss, a reboot resolves it. There was that week where the server was rebooted early (about 2 hours before TWDT) and I wasn't convinced the lag was properly resolved (I was asking for it to be rebooted because the server seemed laggy and I didn't know it was already done before I logged on - so basically I couldn't tell it had been done). I wonder if the 2% C2S issue starts at a set time, so the early reboot was before the event.

Q: How many days (or hours) on average would it take after Sunday for the packetloss to return?
A: I think there are two issues:
a) Baseline lag is bad. I have 0.1-0.2% S2C at all times. This didn't happen with the old server. It can make it hard to define what issue we're talking about.
b) I think the 2% C2S issue occurs at a fixed time around 1pm Eastern Sunday. It feels scheduled, like windows updates, or an attack, or contention.

Q: Does it usually return very quickly/all at once, or does it slowly get worse and worse over a period of time? If it slowly gets worse, what's the period of time it takes to get worse? And, does it seem to continue to get continually worse and worse the longer the server is up, or does it top out at some point?
A: Usually the 2% C2S issue returns a long time after the last restart, but when it returns it returns instantly (so a daily reboot won't help).

Q: Is there anything else you think would be useful to describe the problem? (For example, times when it seems to be better or worse, arenas where it's worse, latency/ping issues, etc.)
A: In other games I've seen laggy players cause the whole server to lag, removing the laggiest player fixed lag for the entire server. We know there are some people with extreme lag which makes basic things like shooting/moving/dying not work properly. I'd expect an improvement for everyone by removing the laggiest ~1% of players via stricter lag limits. See BIET/RR stats which show total lagouts per player in TWD, really obvious who the offenders are.

Elim is always the laggiest arena in the zone. For me it's near unplayable while pub performs fine. Defo needs the map ripping apart to find out why. The javs map performs fine. The wbduel map performs fine. Somehow together they're a clusterfuck. I think the answer to your question though is that all maps are affected equally by this particular issue.

I think the best idea we've had so far is to move the server to a different data center. I'd rather have an extra 10ms if it gets rid of everyone's 0.1% S2C and fixes the 2% C2S issue.

[Exalt]

My lag is literally fine since I moved back to the states. These people are a bunch of whining morons. Not a single one of them have lag that matters if they live in North America. Try logging into this game from Europe or Asia though. That's when shit is literally unplayable, and a server reboot won't do a damn thing about it. People just whine about the lag because they're bad and want to blame it on something other than themselves.

[Rab]

Speaking of data. If this was something I had to deal with at work, I'd have a ton of data - so I could see when the issue was, and see what happens after I make changes. If I didn't have the data the first thing I'd do is put it in place. E.g. A bot that moves around all arenas and collects player ?lag data, say every 5 mins, then saves it to the database; then some sort of aggregation, analysis, and visualisation of that data.

Edit: It's also better to gather data yourself, because people are: shit, liars, incorrect, confused, lack expertise, etc. And because you end up with something automated you can continue to use forever.

Art to demonstrate what it could look like:


Such a bot could also be used to enforce lag limits. I know there are lag limits because people do get specced because of lag, but it's inconsistent and/or not enforced regularly enough. Evidence is how easy it is for me to go to elim and use !lag on everyone and get someone specced. Would also be a good way to enforce the same limits in all arenas.

[Nipple Nibbler]

1) Yes sometimes between 1 to 2 %, usually the first one, s2c.

2) Sometimes, if it doesn't it usually helps at least.

3) Unsure, I'll try to remember next Sunday.

4) Seems to climb, might be okay for a bit, but then it gets to the point the matchbot wont even allow me to play.

5) Problems usually start sometime between 4pm and 9pm est.

I Never had these problems with the old server. My 2 uneducated theories are, there is a hop close to the server, that is having problems of some sort. The other theory I have is, some sort of software problem with the server, maybe some sort of automatic updates or some sort of memory leak that gets worse with more players on maybe? I generally have between 10 and 20 ping normally, sometimes 30. Using a VPN solves the problem sometimes. So the hop theory is probably part of the problem.

Rab's idea to use a bot to mass collect data is a good one, it would give you a lot more data and the data would be more reliable, assuming the bot is programmed properly lol.

[Rab]

I think the issue RR has is a different issue because often he has issues when most people don't. So he's affected by 3 separate issues where I'm affected by 2.

Current issues:
1. Baseline 0.1% - 0.2% S2C
2. Sunday spikes to 2% C2S requiring server restart
3. Intermittent routing issue for some players (Canadians?) requiring VPN

Resolved issues:
1. Windows updates

[2pac]

There has been an issue with those who have normally very clean connections still getting packetloss. This seems to occur only after the server has been running a bit (hasn't been rebooted), but this isn't absolutely confirmed. If this is the case, we need to confirm it, because if so it's an incredibly simple fix that can be automated.

Throughout this DT season, the server was rebooted before games each Sunday. As a result of this, we've had a consistent starting point to work from. However, this Sunday, the server was not rebooted.

For those of you experiencing packetloss, please think back carefully and answer these questions as accurately as you can. If we can get enough answers, we can get closer to nailing this down. The fewer answers we have, the less likely it is that we can find a solution.

• Before a DT Sunday (Friday, Saturday, day-of) did you experience packetloss? How much? S2C or C2S?
• After the reboot, would the packetloss clear for you?
• How many days (or hours) on average would it take after Sunday for the packetloss to return?
• Does it usually return very quickly/all at once, or does it slowly get worse and worse over a period of time? If it slowly gets worse, what's the period of time it takes to get worse? And, does it seem to continue to get continually worse and worse the longer the server is up, or does it top out at some point?
• Is there anything else you think would be useful to describe the problem? (For example, times when it seems to be better or worse, arenas where it's worse, latency/ping issues, etc.)

Thanks for your help.

Chef's Kiss cleary on this one, it's the server LOL

https://upcloud.com/compare/vultr/

https://www.bbb.org/us/nj/matawan/pr...639/complaints

[Exalt]

There has been an issue with those who have normally very clean connections still getting packetloss. This seems to occur only after the server has been running a bit (hasn't been rebooted), but this isn't absolutely confirmed. If this is the case, we need to confirm it, because if so it's an incredibly simple fix that can be automated.

Throughout this DT season, the server was rebooted before games each Sunday. As a result of this, we've had a consistent starting point to work from. However, this Sunday, the server was not rebooted.

For those of you experiencing packetloss, please think back carefully and answer these questions as accurately as you can. If we can get enough answers, we can get closer to nailing this down. The fewer answers we have, the less likely it is that we can find a solution.

• Before a DT Sunday (Friday, Saturday, day-of) did you experience packetloss? How much? S2C or C2S?
• After the reboot, would the packetloss clear for you?
• How many days (or hours) on average would it take after Sunday for the packetloss to return?
• Does it usually return very quickly/all at once, or does it slowly get worse and worse over a period of time? If it slowly gets worse, what's the period of time it takes to get worse? And, does it seem to continue to get continually worse and worse the longer the server is up, or does it top out at some point?
• Is there anything else you think would be useful to describe the problem? (For example, times when it seems to be better or worse, arenas where it's worse, latency/ping issues, etc.)

Thanks for your help.

qan So... today I've seen anywhere between .5% to 20% S2C ploss no matter the connection I use. I lagged out twice using home wifi, so I switched to mobile hotspot with my phone internet to see if it was my wifi, and I lagged out with that too. I average 40 ms where I'm at. This server ploss is ridiculous. This all happened at 6:30 PM CST/7:30 PM EST. I know a few hours before, Riverside and a few others had to cancel a DD with us because the lag was too bad to play.