Trench Wars Repairs!

[H.M.S. Stargazer]

Since we have no official 'Public relations' staffer anymore, I shall inform the players the current status of the zone.

As we are all aware, the bots have been less then stable for a while now. Pure_Luck and Spacehiker have taken the initiative to find out exactly what the problem with the bots have been. They have started to systematically shut down certain bots and see how the core stability maintains.

Originally the offending bot(s) cause the TWCore to crash thus crashing all bots in the zone. Maverick then moved to 3 cores

TWCore-Events:
This includes: PurePubBots(RoboBoy, RoboGirl, etc), RoboBots(The bots the staff use to do hosting), TWBots(Elim, Baseelim), BJWSBots (Base, Wbduel, Javduel, and spidduel).

TWCore-Leagues:
This includes: DuelBot, TournyBot, MatchBots, TWDBot.

TWCore:
This includes the rest of the miscellaneous Bots the zone has.


01/26-01/27:
Pure_Luck Shut down all of TWCore-Events, leaving the rest of the bots up.
Result: Although there was a longer stability of the other cores, a seperate issue of spawning matchbots was found after Duelbot and all TWJD bots crashed suddenly.

Yesterday 01/27-01/28:
Pure_Luck has shut down All of TWCore-Events except PurePubBot. He has also shut down DuelBot.

Today 01/28-01/29
DoCk> and Maverick spent a good portion of the day working on all the bots. They traced the bot issues to a problem with the elim bots andthey have temporarily removed them while they look at the elim code.
TWCore is down for now. No word on when its being brought back.


I will update this daily, or as soon as issues arise to keep the public informed. This process may take a while, but we are working on fixing the problems once and for all. Thanks for your patience!

-H.M.S. Stargazer

[Ganon-Knight]

Keep us apprised. Bots dying randomly is a nuisance.

[Maverick]

Someone should inform Pure_Luck that his crappy server is responsible for 99% of the problems. You can't expect a lightweight server with 1 GB of RAM to run all the bots and websites smoothly.

But hey, just keep shutting down bots, see if it works! <_<
(Which is like chopping of limbs when you're stuck in a box that's too small to fit your body.)



PS. Good work on informing everybody that Pure_Luck and SpaceHiker has taken initiative to repair the bots. Seems like nobody else was working on them.

[Jason]

if maverick's sarcasm is backed by fact, and the bot server really is that skimpy on hardware, it's time to get serious about fund raising and donations. a server with a gig of ram isn't a server at all.

we (the public) need to know what hardware is currently in the server. we need to know if any parts can be re-used in the building of a new server. and once we know what parts are needed, we can set a budget and start working toward getting the money together.

everyone's tired of the (nearly) constant crashes. it's time to step up and drop down some dough. that's all there is to it.

[Goddess]

Erm...... just a thought. Isnt MAVERICK supposed to be the one FIXING BS? yeah u want to sit back criticizing GO DO YOUR WORK OR GTFO STAFF SON. The fact that we even HAVE a server is a wonder with how we were almost shut down. Or maybe u weren't around when we the staff (including Dock>) were having to ho ourselves out just to get the funds to keep the TW's up?

If you're not the person responsible for fixing bots then what exactly is it that you do?

Oh and right on Jason... way to be the constructive man. You must be a professional something or other that deals with getting things done <3 I 2nd all that which Jason has said and will help in w/e way i can.

[Creature]

Agreed with Jason

[Jason]

by the way, just to put things in perspective for those who might think buying a new server is beyond the reach of a small donations style campaign, let me just say that all of the following can be had for right around $500 USD (plus tax/shipping).

cpu
motherboard
ram
hard drive
optical drive
power supply
case
os license

that's everything we would need, for only about $500. and i'm not even taking into account the fact that we may be able to re-use some of what we have, thus lowering the price tag.

$500 is a very small number. that's only 50 people donating 10 dollars. or 100 people donating 5 dollars. if you went to mcdonald's (or some other fast food chain) in the past week/month/year, chances are you spent at least 5 dollars. if you can afford a big mac, you can afford to go without the big mac (fatty!) and put the money to better use. if you have the money to pay for internet access (which you use to play subspace) you can afford 5 or 10 dollars. if you've only been playing a short time, let's say you've racked up a mere 200 hours of usage, a 5 dollar donation works out to two and a half cents per hour you've played. two and a half cents. what other service could you possibly enjoy for such a low fee?

okay, i think i've made my point.

as soon as i know where we stand with hardware on the current server, i would be more than happy to scope out specs/costs of a new machine. and i will most definitely be contributing my ten bucks.

[Pure_Luck]

1) We have had the same hardware for some time now. The bots haven't changed much until recently. All the new coding throws java errors every day. This comes from every new bot that you and fantus and the rest of the crew has done. The other day pubbot spawned itslef 40 times in a row to an arena. I am not sure you can blame that on hardware. Heres some more examples of the new shitty code we have been using:

java.lang.NullPointerException
at twcore.bots.staffbot.staffbot_banc.cmdListBan(staffbot_banc.java:681)
notice STAFFBOT_BANC - which is your coding

java.lang.ArrayIndexOutOfBoundsException
at twcore.core.util.ByteArray.addString(Unknown Source)
at twcore.core.net.GamePacketGenerator.sendChatPacket (Unknown Source)
at twcore.core.BotAction.sendChatMessage(Unknown Source)
at twcore.core.BotAction.sendChatMessage(Unknown Source)
at twcore.bots.staffbot.staffbot_serverwarningecho.handleEvent(staffbot_serverwarn ingecho.java:70)


Exception in thread "WBDuelBot-Session" java.lang.NoClassDefFoundError: twcore/bots/bwjsbot/bwjsbot$BWJSCaptain

Exception in thread "TWBot1-Session" java.lang.NoClassDefFoundError: twcore/bots/elim/elim$CompareByLosses
at twcore.bots.elim.elim.cmd_wl(elim.java:411)
at twcore.bots.elim.elim.handleCommands(elim.java:303 )
at twcore.bots.elim.elim.handleEvent(elim.java:276)
at twcore.core.net.GamePacketInterpreter.handleChatMe ssage(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateNor malPacket(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateGam ePacket(Unknown Source)
at twcore.core.net.ReliablePacketHandler.handleReliab leMessage(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateSpe cialPacket(Unknown Source)
at twcore.core.net.GamePacketInterpreter.translateGam ePacket(Unknown Source)
at twcore.core.Session.run(Unknown Source)

2) You never once suggested upgrading the server would fix the problem . I am no programmer but it sounds like a better server would still run slow with all these errors in the bots. Or would they just go away with a faster machine? I dont think a faster machine can handle a bot trying to login to the sql server 800 times like it did yesterday.

After it not working for how many months now and you not being able to fix it , we decided to start shutting down bots to see if we can stop it from getting to the 800 connections then crashing like it has been doing every day. I apologize if you are upset about us not telling you we were disabling some bots but we feel we needed to move things along to figure out a solution. By all means if we want to try a new server we can do that. I am not against it at all. But I wouldn't blame the server for the main issues.

[Jason]

what kind of hardware is in the current server, pl?

obviously, priority one is to clean up sloppy code. i'm not at all surprised to hear about this. twcore has been passed around so much since its inception and we've reached a "too many cooks spoil the pot" conclusion.

once we get that taken care of, i still think a hardware upgrade should be considered. (if, of course, the current hardware is actually outdated.)

[Pure_Luck]

Space going to post the specs, I really dont know what they are. But yes if we want to get money up to get a better server by all means lets do it.

[Maverick]

All the new coding throws java errors every day. This comes from every new bot that you and fantus and the rest of the crew has done. The other day pubbot spawned itslef 40 times in a row to an arena. I am not sure you can blame that on hardware. Heres some more examples of the new shitty code we have been using:

You're trying to prove by showing some exceptions that the code is bad? Please go learn some Java first before going there. To satisfy your needs, let me explain.
Staffbot_banc module is mine, yes. A NullPointerException, an ArrayIndexOutOfBoundsException or a java.lang.NoClassDefFoundError has nothing to do with the database, performance or why bots die. Those are thrown just so the bot can continue with whatever it is doing.

Repeated errors are, however, signs of something going completely wrong. I've already fixed some repeating errors already (go look at www.twcore.org/timeline to see for yourself) but to no avail. I already fixed the elimination bots dying at the end of every game round but by doing that, it put extra strain on the database as the bot didn't stop when the database buckled under the load.

"All the new coding" - it seems like you're forgetting the instability of the MatchBots and that those are instable as every other bot in the zone?

You never once suggested upgrading the server would fix the problem .

This is because I was simply not done working on the problems just yet. Splitting up the core into 3 bots was my latest plan and idea in an attempt of fixing the instability issues. I might have had some more plans and ideas to try until I would come to the conclusion of it being a server problem. However recently we've found out what kind of server we're running on. This kind of server easily explains why bots disconnect and why the mysql database is having such big problems performing normally.

I dont think a faster machine can handle a bot trying to login to the sql server 800 times like it did yesterday.

Do you have any proof on this? If so, which bot was this then?

After it not working for how many months now and you not being able to fix it

The severe instability problems started right after this server crash where the harddisk had to be replaced.

I apologize if you are upset about us not telling you we were disabling some bots but we feel we needed to move things along to figure out a solution. By all means if we want to try a new server we can do that. I am not against it at all. But I wouldn't blame the server for the main issues.

Only this sentence would've satisfied me yesterday and all of this ranting wouldn't have been necessary. I just need some communication from our system operators if they make such decisions - I don't think that's too much to ask.

It seems that being extremely sarcastic in a public forum is the only way of getting your attention and getting a response from you. Again, I wish this wasn't necessary but at least we're talking now.

[Pure_Luck]

You're trying to prove by showing some exceptions that the code is bad? Please go learn some Java first before going there. To satisfy your needs, let me explain.
Staffbot_banc module is mine, yes. A NullPointerException, an ArrayIndexOutOfBoundsException or a java.lang.NoClassDefFoundError has nothing to do with the database, performance or why bots die. Those are thrown just so the bot can continue with whatever it is doing. .

Do you have any proof on this? If so, which bot was this then?

dock> null pointer exceptions mean garbage programming
dock> too many open files is because there are hundreds and hundreds of connections to mysql
dock> well it was 700 a day ago

"All the new coding" - it seems like you're forgetting the instability of the MatchBots and that those are instable as every other bot in the zone?

Matchbots have been around forever and still to this day don't throw any errors and work fine. They go down when the core goes down.

[SpaceHiker]

There are 2 critical components to a decent server, especially in the situation we are in.

The first is the actual hardware, which is:
AMD Sempron 3000+ (I believe that's a 1.6 GHZ single core)
1 GB Ram
Ubuntu Server Edition.
Not sure about hard disk space.

The second component, and actually more important in many respects, is the connection. This is a dedicated server hosted at an ISP with a very fast and low latency backbone connection. This is critical because we do not have the ability to install the bots directly on the actual game server. In order for everything to work properly the game server and our bot server have to have a very stable connection to each other with very low ping/latency/ploss/all that stuff.

Anyone can build a server for a few hundred bucks that has newer hardware, but it doesn't matter without the connection to back it up. I have operated much higher demand applications on servers that were not even a fraction as powerful back in the 1990's.

Bottom line is that this server SHOULD be enough for our needs. We can certainly try a different server if someone can set it up, and in fact we might be doing that shortly. But in order to make the determination that our server is insufficient, you really need to study some data first rather than jump to conclusions. I have been staring at Top(similar to Windows Task Manager), PhpMyAdmin, and other stuff now for a long time , with various bots active, and frankly the server is not acting like it's overloaded until the bots suddenly start consuming insane MySQL resources. By "insane" I mean too much resources for the work they're doing.

[Maverick]

dock> null pointer exceptions mean garbage programming
dock> too many open files is because there are hundreds and hundreds of connections to mysql
dock> well it was 700 a day ago

Yea, quoting Dock> is really proving something here!
I haven't got the chance to talk to Dock> yet to dig into the issue. Like I've already said, lots of NullPointerExceptions mean garbage programming but we don't have that.

Explain to me how the bots were running fine some months ago and very recently just went very instable without any significant change to the source code?
Go look at the twcore.org/timeline if you don't believe me.

Matchbots have been around forever and still to this day don't throw any errors and work fine. They go down when the core goes down.

Get your facts straight, they have been throwing loads of errors - just like some other bots do. MatchBots might be the root of the problem but nobody can say for certain yet.

SpaceHiker, you can't expect a server with 1 GB of RAM to run a database that is more then 2 GB in size.
The bots are instensively using the database, I've never said they weren't.
The claims about the bots/cores using loads of connections still need to be proven via the source and once that's done, we can work on the solution.

You won't ever hear me say that TWCore's code is really the best there is. There are loads of problems, some things aren't completely implemented and some things aren't really efficient. Just the fact that I've never seen TWCore have anything longer then a week in uptime says enough.

[Pure_Luck]

Explain to me how the bots were running fine some months ago and very recently just went very instable without any significant change to the source code?

been many code changes since then.new banc stuff, new stuff added to bwjs bots.

Get your facts straight, they have been throwing loads of errors - just like some other bots do. MatchBots might be the root of the problem but nobody can say for certain yet.

I looked at the exception logs for the last month, haven't seen one from matchbots. TWD Can run by itself for decades without going down. I think we both know that.

Dock's trying to help , I don't think he would make up that its connecting over 700 times at once. So I think its been proven that its happening.