Trench Wars Repairs!

[PhyRe]

thanks stargazer

[Maverick]

I didn't want to drag out here either. I only made on sarcastic comment because of the way of how things recently were going. Pure_Luck responded in this forum. I'm more then happy to continue this discussion in the staff forum.

[Exalt]

Guys... for the sake of the uninformed public, that for the most part should STAY uninformed... can we take this to emails/in game pms/staff chat/etc?

I mean... This is all fine and dandy to most forum users, but you don't need new players coming in and seeing the bickering. It just isn't very professional. While I'm almost positive we all appreciate the publicity of the inner workings of upper staff (you guys are in general far too secretive), in this case it probably isn't needed.

If a new server and a donation fund is set up, then it is time to open the can of worms and give everyone a good reason to spend their money. If not, then this thread shouldn't be needed.


Needless to say, we all appreciate everything you guys are doing.

[Fit of Rage]

Being illiterate in java, I can't really do much to help that front. However, if a new server would help things, I'd be happy to be one of the ones donating to make that happen. When everything gets figured out, I'd appreciate it if PL/HMS or someone else posts a "summary" thread of problems and what's required to fix it.

[Maverick]

been many code changes since then.new banc stuff, new stuff added to bwjs bots.

A few extra bots can't make a sql server buckle under the load. I agree the new BWJS bots, the new elimination bots are putting extra strain on the database but I doubt they are the root of the problem here.

I looked at the exception logs for the last month, haven't seen one from matchbots. TWD Can run by itself for decades without going down. I think we both know that.

I clean up the logs almost every week now as they grow rapidly in size making them impossible to handle. If you would've have viewed the logs of last month, you would've seen the loads of errors from the MatchBot as it was trying to load the rules.
TWD ran "for decades" when it was on RoboQueen but that was on Priitk's server with code of 2000. This is not any benchmark at all. If you want to run TWD like that, I will get you a copy of the source code of back then. Fixing bugs is all up to you aswell then.

Dock's trying to help , I don't think he would make up that its connecting over 700 times at once. So I think its been proven that its happening.

I'm not saying he's making it up, I'm asking what proof YOU have. I haven't talked to Dock> myself so whatever you quote him is pointless to me (plus you still haven't given me anything to work on).


It's incredible how you think you know everything. If you want to solve the issues, please go ahead, I won't bother anymore.


PS. You are still missing the point I'm making that I want YOU TO COMMUNICATE. It seems like you don't know how to at all. Like I've said before, a simple ?message would've prevented all of this.

[Pure_Luck]

Explain to me how the bots were running fine some months ago and very recently just went very instable without any significant change to the source code?

been many code changes since then.new banc stuff, new stuff added to bwjs bots.

Get your facts straight, they have been throwing loads of errors - just like some other bots do. MatchBots might be the root of the problem but nobody can say for certain yet.

I looked at the exception logs for the last month, haven't seen one from matchbots. TWD Can run by itself for decades without going down. I think we both know that.

Dock's trying to help , I don't think he would make up that its connecting over 700 times at once. So I think its been proven that its happening.

[Maverick]

dock> null pointer exceptions mean garbage programming
dock> too many open files is because there are hundreds and hundreds of connections to mysql
dock> well it was 700 a day ago

Yea, quoting Dock> is really proving something here!
I haven't got the chance to talk to Dock> yet to dig into the issue. Like I've already said, lots of NullPointerExceptions mean garbage programming but we don't have that.

Explain to me how the bots were running fine some months ago and very recently just went very instable without any significant change to the source code?
Go look at the twcore.org/timeline if you don't believe me.

Matchbots have been around forever and still to this day don't throw any errors and work fine. They go down when the core goes down.

Get your facts straight, they have been throwing loads of errors - just like some other bots do. MatchBots might be the root of the problem but nobody can say for certain yet.

SpaceHiker, you can't expect a server with 1 GB of RAM to run a database that is more then 2 GB in size.
The bots are instensively using the database, I've never said they weren't.
The claims about the bots/cores using loads of connections still need to be proven via the source and once that's done, we can work on the solution.

You won't ever hear me say that TWCore's code is really the best there is. There are loads of problems, some things aren't completely implemented and some things aren't really efficient. Just the fact that I've never seen TWCore have anything longer then a week in uptime says enough.

[SpaceHiker]

There are 2 critical components to a decent server, especially in the situation we are in.

The first is the actual hardware, which is:
AMD Sempron 3000+ (I believe that's a 1.6 GHZ single core)
1 GB Ram
Ubuntu Server Edition.
Not sure about hard disk space.

The second component, and actually more important in many respects, is the connection. This is a dedicated server hosted at an ISP with a very fast and low latency backbone connection. This is critical because we do not have the ability to install the bots directly on the actual game server. In order for everything to work properly the game server and our bot server have to have a very stable connection to each other with very low ping/latency/ploss/all that stuff.

Anyone can build a server for a few hundred bucks that has newer hardware, but it doesn't matter without the connection to back it up. I have operated much higher demand applications on servers that were not even a fraction as powerful back in the 1990's.

Bottom line is that this server SHOULD be enough for our needs. We can certainly try a different server if someone can set it up, and in fact we might be doing that shortly. But in order to make the determination that our server is insufficient, you really need to study some data first rather than jump to conclusions. I have been staring at Top(similar to Windows Task Manager), PhpMyAdmin, and other stuff now for a long time , with various bots active, and frankly the server is not acting like it's overloaded until the bots suddenly start consuming insane MySQL resources. By "insane" I mean too much resources for the work they're doing.

[Pure_Luck]

You're trying to prove by showing some exceptions that the code is bad? Please go learn some Java first before going there. To satisfy your needs, let me explain.
Staffbot_banc module is mine, yes. A NullPointerException, an ArrayIndexOutOfBoundsException or a java.lang.NoClassDefFoundError has nothing to do with the database, performance or why bots die. Those are thrown just so the bot can continue with whatever it is doing. .

Do you have any proof on this? If so, which bot was this then?

dock> null pointer exceptions mean garbage programming
dock> too many open files is because there are hundreds and hundreds of connections to mysql
dock> well it was 700 a day ago

"All the new coding" - it seems like you're forgetting the instability of the MatchBots and that those are instable as every other bot in the zone?

Matchbots have been around forever and still to this day don't throw any errors and work fine. They go down when the core goes down.

[Maverick]

All the new coding throws java errors every day. This comes from every new bot that you and fantus and the rest of the crew has done. The other day pubbot spawned itslef 40 times in a row to an arena. I am not sure you can blame that on hardware. Heres some more examples of the new shitty code we have been using:

You're trying to prove by showing some exceptions that the code is bad? Please go learn some Java first before going there. To satisfy your needs, let me explain.
Staffbot_banc module is mine, yes. A NullPointerException, an ArrayIndexOutOfBoundsException or a java.lang.NoClassDefFoundError has nothing to do with the database, performance or why bots die. Those are thrown just so the bot can continue with whatever it is doing.

Repeated errors are, however, signs of something going completely wrong. I've already fixed some repeating errors already (go look at www.twcore.org/timeline to see for yourself) but to no avail. I already fixed the elimination bots dying at the end of every game round but by doing that, it put extra strain on the database as the bot didn't stop when the database buckled under the load.

"All the new coding" - it seems like you're forgetting the instability of the MatchBots and that those are instable as every other bot in the zone?

You never once suggested upgrading the server would fix the problem .

This is because I was simply not done working on the problems just yet. Splitting up the core into 3 bots was my latest plan and idea in an attempt of fixing the instability issues. I might have had some more plans and ideas to try until I would come to the conclusion of it being a server problem. However recently we've found out what kind of server we're running on. This kind of server easily explains why bots disconnect and why the mysql database is having such big problems performing normally.

I dont think a faster machine can handle a bot trying to login to the sql server 800 times like it did yesterday.

Do you have any proof on this? If so, which bot was this then?

After it not working for how many months now and you not being able to fix it

The severe instability problems started right after this server crash where the harddisk had to be replaced.

I apologize if you are upset about us not telling you we were disabling some bots but we feel we needed to move things along to figure out a solution. By all means if we want to try a new server we can do that. I am not against it at all. But I wouldn't blame the server for the main issues.

Only this sentence would've satisfied me yesterday and all of this ranting wouldn't have been necessary. I just need some communication from our system operators if they make such decisions - I don't think that's too much to ask.

It seems that being extremely sarcastic in a public forum is the only way of getting your attention and getting a response from you. Again, I wish this wasn't necessary but at least we're talking now.

[Pure_Luck]

Space going to post the specs, I really dont know what they are. But yes if we want to get money up to get a better server by all means lets do it.

[Jason]

what kind of hardware is in the current server, pl?

obviously, priority one is to clean up sloppy code. i'm not at all surprised to hear about this. twcore has been passed around so much since its inception and we've reached a "too many cooks spoil the pot" conclusion.

once we get that taken care of, i still think a hardware upgrade should be considered. (if, of course, the current hardware is actually outdated.)

[Pure_Luck]

1) We have had the same hardware for some time now. The bots haven't changed much until recently. All the new coding throws java errors every day. This comes from every new bot that you and fantus and the rest of the crew has done. The other day pubbot spawned itslef 40 times in a row to an arena. I am not sure you can blame that on hardware. Heres some more examples of the new shitty code we have been using:

java.lang.NullPointerException
at twcore.bots.staffbot.staffbot_banc.cmdListBan(staffbot_banc.java:681)
notice STAFFBOT_BANC - which is your coding

java.lang.ArrayIndexOutOfBoundsException
at twcore.core.util.ByteArray.addString(Unknown Source)
at twcore.core.net.GamePacketGenerator.sendChatPacket (Unknown Source)
at twcore.core.BotAction.sendChatMessage(Unknown Source)
at twcore.core.BotAction.sendChatMessage(Unknown Source)
at twcore.bots.staffbot.staffbot_serverwarningecho.handleEvent(staffbot_serverwarn ingecho.java:70)


Exception in thread "WBDuelBot-Session" java.lang.NoClassDefFoundError: twcore/bots/bwjsbot/bwjsbot$BWJSCaptain

Exception in thread "TWBot1-Session" java.lang.NoClassDefFoundError: twcore/bots/elim/elim$CompareByLosses
at twcore.bots.elim.elim.cmd_wl(elim.java:411)
at twcore.bots.elim.elim.handleCommands(elim.java:303 )
at twcore.bots.elim.elim.handleEvent(elim.java:276)
at twcore.core.net.GamePacketInterpreter.handleChatMe ssage(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateNor malPacket(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateGam ePacket(Unknown Source)
at twcore.core.net.ReliablePacketHandler.handleReliab leMessage(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateSpe cialPacket(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateGam ePacket(Unknown Source)
at twcore.core.Session.run(Unknown Source)

2) You never once suggested upgrading the server would fix the problem . I am no programmer but it sounds like a better server would still run slow with all these errors in the bots. Or would they just go away with a faster machine? I dont think a faster machine can handle a bot trying to login to the sql server 800 times like it did yesterday.

After it not working for how many months now and you not being able to fix it , we decided to start shutting down bots to see if we can stop it from getting to the 800 connections then crashing like it has been doing every day. I apologize if you are upset about us not telling you we were disabling some bots but we feel we needed to move things along to figure out a solution. By all means if we want to try a new server we can do that. I am not against it at all. But I wouldn't blame the server for the main issues.

[Jason]

by the way, just to put things in perspective for those who might think buying a new server is beyond the reach of a small donations style campaign, let me just say that all of the following can be had for right around $500 USD (plus tax/shipping).

cpu
motherboard
ram
hard drive
optical drive
power supply
case
os license

that's everything we would need, for only about $500. and i'm not even taking into account the fact that we may be able to re-use some of what we have, thus lowering the price tag.

$500 is a very small number. that's only 50 people donating 10 dollars. or 100 people donating 5 dollars. if you went to mcdonald's (or some other fast food chain) in the past week/month/year, chances are you spent at least 5 dollars. if you can afford a big mac, you can afford to go without the big mac (fatty!) and put the money to better use. if you have the money to pay for internet access (which you use to play subspace) you can afford 5 or 10 dollars. if you've only been playing a short time, let's say you've racked up a mere 200 hours of usage, a 5 dollar donation works out to two and a half cents per hour you've played. two and a half cents. what other service could you possibly enjoy for such a low fee?

okay, i think i've made my point.

as soon as i know where we stand with hardware on the current server, i would be more than happy to scope out specs/costs of a new machine. and i will most definitely be contributing my ten bucks.

[Creature]

Agreed with Jason