Some updates from the Communication Department

[Shaddowknight]

Hello hello my fellow Trenchers! (Trenchatarians... Trenchians?)

First off, our recent issues with the bots and bot server are either all resolved, or should be so shortly. Hopefully we can prevent this from being an issue in the future as well.

KATC has stepped down as TWL head, so we are looking for a replacement.
As of now, the current schedule for TWL will be a full twd ladder reset at the beginning of October. TWL is currently set to officially start on the first weekend in January.

We are working on some changes to the pub map as well. We have a few different maps being worked on at the moment. When we are done we would like to try to isolate the positive and negative changes of each map, and use the feedback from those maps to make the best map from each of their parts.


On to our current staffers. You will see some changes and new additions. First off we give our congrats to Paradise for making Mod. We have a few re-hires as well, LAMO returning as a mod, and Grandias <ZH> coming back as well. Triping and Code 142 have both been hired as well, as Trip <ZH> and Code 142 <ZH> respectively.


We'll be announcing more on the pub map changes when we are closer to getting the maps done and some minor live testing in.


-9/22/03 updates:
fiS has also stepped down as head of TW Staff Training. As I was Assistant Head of Training, I have been promoted to the Head Trainer spot.

SliCk <ZH> has been promoted to SliCk <ER>, and Magnetic <ER> is now a mod (just Magnetic now, in case you wondered)

I'll update again when I make a decision on who I will be taking over the Assistant Head Trainer position.




Any questions or comments? Feel free to mention them here or ask me in-game.

[Rab]

What caused the bot issues?

What happened to Zone consolidation?

Why did Katherine step down?

LAMO = LMAO?

[Shaddowknight]

Bot issues caused by stuff consuming too much Server memory. K A N E and the rest of dev spent a lot of time in the past week to get it fixed and stable.
As for zone consolidation, I doubt it will actually happen for the time being.
I don't know for certain as to why KATC is stepping down, Ive got a message to him for an official reason for it at the moment.

LAMO = LMAO, yes

[Rab]

ty

[Shaddowknight]

Happy to answer your questions

[Ephemeral]

So if I understand correctly and reading between the lines….

TW was broken because we made changes and didn’t test them correctly; chances are that good that there were multiple changes made by multiple people and no one really knew what the cumulative effect of their changes would be.
So we worked hard to fix it but good news is that we are making more multiple changes by multiple people?

It would be best to post the root cause followed by what has changed to prevent it from happening again. It would be best if BEFORE people pull ‘good ideas’ out of their ass they are required to provide a test plan that not only supports the anticipated positive effect the change is expected to have but also demonstrate the server overhead or ‘cost’ of thoer idea.

So a few new questions.
1. If this hasn’t been done, why not?

2. What has been done to prevent this from happening again? (Not the specific issues that broke it, but the procedureal or process changes that have been made.)

3. Wasn’t here a new test/dev server established that can be used before making changes on the production server? Was this not done and if so, why not?

4. I know that there is a number of different people who all can make changes to the production server, how many poeple can do this and who are they?

5. Who oversees and coordinates the changes these folks make?

6. Can any of these folks make a small change to the production server without getting approval or without oversight? (Now or then?)
eph

[Astrogirl]

[Astrogirl]

can finally post without nazikid kthx. though im sure not because he grew up.

1/3 of my keys are stuck from spilled beer

[ThePAP]

Ephemeral,

I hope you don't mind getting an answer from a normal dev, instead of one of the lead developers, but if my answers are unsatisfactory, then I'll try to get one of them over to answer your questions. Besides that, I'll try to answer your questions to be best of my knowledge.

So if I understand correctly and reading between the lines….

TW was broken because we made changes and didn’t test them correctly; chances are that good that there were multiple changes made by multiple people and no one really knew what the cumulative effect of their changes would be.
So we worked hard to fix it but good news is that we are making more multiple changes by multiple people?

It would be best to post the root cause followed by what has changed to prevent it from happening again. It would be best if BEFORE people pull ‘good ideas’ out of their ass they are required to provide a test plan that not only supports the anticipated positive effect the change is expected to have but also demonstrate the server overhead or ‘cost’ of thoer idea.

I'm not certain if I am allowed to disclose the exact cause of the problem, but the cause wasn't incorrect testing nor recent changes to the bot. When developing bots, we can only test them to a certain extend, and we try to test them as much as we can. (In my case, the testing of new code takes on average roughly thrice as long as the actual writing of the code.) Unfortunately, it is impossible to fully test a change, because we do not have the resources to test f.e. the load of 150+ players during peak hours. (Sorry if that number is way of, but I assume you get the idea.) Besides that, we do not have access to the databases used on the live server, so our database related things cannot be fully tested either.

That being said, we do try to anticipate the impact of changes. For example, a bug that I found in one of the core files, hasn't been fixed yet on live, because it indirectly affects a huge number of bots. We want to make sure first that the fix for the bug (it's a relative simple one) won't actually break everything else.

Let's try to answer your questions now.

So a few new questions.
1. If this hasn’t been done, why not?

I hope the above will answer this one a bit. Otherwise, I don't quite know how to answer this one, sorry.

2. What has been done to prevent this from happening again? (Not the specific issues that broke it, but the procedureal or process changes that have been made.)

There haven't been any structural changes as of yet, however, the focus has been shifted towards making the system (the bots themselves, not the developing) more efficient. The current prevention methods still suffice enough. More on this in the reply to questions 4 to 6.

3. Wasn’t here a new test/dev server established that can be used before making changes on the production server? Was this not done and if so, why not?

There is indeed a dev server that is being used to test out the bots before they go to live. As mentioned before, we try to test them to the best of our capabilities, but we cannot test every aspect. What we cannot test, we do try to anticipate as much as we can.

4. I know that there is a number of different people who all can make changes to the production server, how many poeple can do this and who are they?

For the bot dev team, including the head developer of all the dev teams combined, there are eight. For the exact list, look for the Senior and higher members on this page: http://twcore.org/index.fcgi/wiki/Developers.
I don't quite know which people of the other developer teams have rights to make changes on the production server though. As far as the number from the bot dev department, from what I've been told and understood, this is so high to try to ensure at least one of them can be online at any time of the day, in case the sh*t hits the fan, botwise of course.

Of those eight mentioned, some have a more rights than others, but they all have the power to bring down the bots, alter their code, and bring them back up.

5. Who oversees and coordinates the changes these folks make?

The procedure to make bot changes live is as follows:
- Claim a ticket, if applicable.
- Code the fix/enhancement.
- Test out the changes on the developer server. If they fail, go back to the previous step, of course.
- Contact a senior dev (unless the person is one himself).
- With his knowledge, commit the changes to the repository. At this moment, the changes are available to the other developers, however, the changes are not live yet!
- A senior developer recompiles the code for the live server.
- The affected bots are restarted, while trying to minimize the impact for the players. This is the actual part where the changes go live.
- The senior developer and the developer who wrote the code stick around for a while, observe the bot's behaviour, doing further live tests, and when a problem occurs, one of the following will happen:
1) If the problem is severe, but can be fixed relatively easily, they make the fix, quickly test it and bring the fix online if it suffices. After this they observe some more.
2) If the problem is severe, but too hard to fix at that moment, the senior developer reverts to the last version of the bot that worked correctly (enough).
3) If the problem is light, but can be fixed relatively easily, the same is done as in option 1.
4) If the problem is light, but hard to fix, the bot will probably stay online, as long as it isn't game breaking. The original developer starts working on a fix as soon as possible.

An example of the fourth option would be: The hockeybot crashes/disconnects if the following happens at the same time: An own goal is made, but while it's being made, the entire opposing team leaves the arena. This was actually a real one, but it was caught on the dev server, before it went live. It was caused by a new feature that assigned an own goal to a member of the opposing team.

6. Can any of these folks make a small change to the production server without getting approval or without oversight? (Now or then?)
eph

Following the guidelines we have, no one that isn't a senior dev can make small changes to the production server themselves. Of course, there is always a slim chance that a change might be overseen and slips through, but these cases are few and far in between. (Haven't seen such a case yet in the short period that I've been on the team.)
Any chance goes through the trac system, that's the only way to bring them live eventually. So even the changes made by the higher devs is logged through the changesets. Besides that, there are automatic (by lack of a better word) unbypassable logs that keep track of any (re)compile to any bot, including who did that and when and with which changesets.


Hope these answers are a bit satisfactory to you. For more information on the system and changes and whatnot, you can always see it for yourself on the trac, which is located on http://twcore.org. If you find any errors made on their by a person named Trancid, and you want to blame him, then that's one of my nicks, so feel free to blame me
If anything is still unclear, please let us know

[Astrogirl]

shout out to TONY [where are you?] best captain in TW history

[Astrogirl]

Following the guidelines we have

[Ephemeral]

ThePap, thank you for your response. Please note that this has nothing to do with ‘blame’; it has everything to do with ‘improvement’.

I do not see dev as being the problem, dev will always ‘get away with’ as much as they can. The dev people do a thankless job and is normal that they seek the most efficient way to get the changes implemented. It is a developers job to be productive, it is not their job to put the proper procedural and process checks and balances in place that ensures quality and reduces repeated failures. This is a management issue.

That said, anyone can see the huge management issues that exist. First let’s consider the Change Request process. Note: I am not talking about the twcore system; that is simply the nuts and bolts that tracks the requests. Some Change Request is made. How are these CRs get reviewed and approved? For example; If a request is made that affects Pub 0, is Fis involved in the approval? This is a huge logistic issue, managing something like this properly would required many frequent meetings with superb communication and organization.

Even more importantly, are ALL changes made through the twcore system? It appears to me that only coding changing get tracked through twcore. Are other changes (ie map or ship configuration for example) ALL tracked through twcore? Is there some review/approval/testing process in place for these changes too? Who oversees the confluence of both these types changes and dev coding changes and how are they controlled? For example; can someone make a change that impacts pub 0 without Fis first approving it?

And of course there are serious issues even in the dev side of the room. You have dev people testing their own code on a test server that is not able to emulate the production server. Wow. Even if this was fixed there are no baseline metrics/data to test the change against because change requests are submitted and allowed with nothing more than a ‘that sounds like a good idea, let’s try it’. But again these aren’t thing that anyone would expect a dev resources to address, these are management issues.

TW staff fails are mostly due to the lack of proper management structure and oversight. There are few (if any) checks and balances accompanied by a complete diffusion of responsibility. In the example above Fis cannot be held responsible for Pub 0 when things change without her being part of the approval process of every single thing that impacts it . Dev cannot be held responsible because they do not have the tools and resources they need to do their job correctly.

Most everyone on staff and in dev is ok with doing things the ‘easy way’ because they aren’t being paid. Trouble is that the zone pays for this short-sight approach. It takes true leadership and vision to break from this current path and finally fix what is so badly broken in this zone. IMO, until and unless that happens the zone will continue its slide into cold, dark death of deep space. Frankly, if they was a serious attempt at fixing these management issues I would think that many people might step forward with renewed interest and motivation to improve the zone. But staying this current course of ‘make tweaks to ships and arenas and hope something get better to save the zone’ is a death sentence IMO.
eph


PS - someone figure out who hacked Astro account and shut them down.

[ThePAP]

I apologize if my reply sounded defensive. This was not my intent, I merely wanted to explain how it works from a bot developer's point of view. (English isn't my native tongue, so that might have been responsible for that.) In advance, this current post is neither with the intent to defend from any blaming, since there is none in my opinion. If it does sound like that, then please ignore it.

As for your current points, for most of them, I can only make assumptions, so it's probably for the better to keep those to myself and not put false rumors or information into the world. (Meh, that last part doesn't sound like the correct saying, but I hope you understand what I mean.)

What I do know is that not every change is made through the twcore system. I'm unsure though if I can expand onto which changes these are, and I'm not implying that these changes are not logged.

As for the developer server, yes, it would be nice if we would be able to emulate the production server even better. From my own experience though, I know that it is impossible to make a perfect emulation. This is probably something you already know, but players are unpredictable, as well as the Internet itself, which prevents the perfect emulation. Don't get me wrong, the current system can be improved, but there will always be flaws that sneak past it, no matter how good.

I hope someone else, more knowledgable, will come along and answer the questions that haven't gotten a suitable response yet, since I'm afraid that I'm of no further help to you.

Hope you'll get your answers/responses quickly

[Shaddowknight]

It appears to actually be Astro, unless the account has been hacked for awhile anyhow... And thanks ThePAP for taking over answering these questions about the dev side of the issue.

To answer about other changes being made to twcore. Generally no, they are not. Pub stuff is generally kept with backups. Any non-pub stuff is backed up at least temporarily when changes are being made. Beyond that it's the map's creator that usually hold any changes or backups/originals. Generally when a map needs to be changed (other than regular maps like pub/elim/etc) it's left to the original map maker to do, unless they are inactive or give permission for it to be changed by others.

Some changes to pub can be done without fiS's approval, mostly from the bot side of things. fiS still would hold final say though is she were to disapprove of anything that was done. Allmost all map changes go through fiS first, at least for uploading. Ideally the head of Event Dev would check through all the non-pub/elim/ect. changes and then forward them to fiS for uploading.

I'm not sure how much sense this post makes or how coherent it is... I've been busy and can't focus very well right now. If you have any more questions on it just ask them and if I get time later today I'll try to go through them more clearly. I am very busy this weekend though so can't promise to get to them till next week, unless someone else can answer for you.

[Ephemeral]

Don’t worry about replies or coherent answers, most of my questions are rhetorical and meant only to stimulate some thought. Not everyone thinks that making a poll and a few superficial map/ship changes are going to save the zone. Not everyone thinks that Steam or holding more events will save the zone.

Some of us (well, at least me) think that the real problem is that the zone not being run correctly due to ‘the buck never stopping’ on anyone’s desk. Additionally the staff structure is based upon being able to silence players, hold events, send a zoner, and ban people; this means it is not based upon good communication, teamwork, or being a solution based organization. But I acknowledge that eadership is very difficult with a volunteer group and demands even more than usual insight and experience.

TW reminds me of a ‘mom and pop’ restaurant that is failing after many previous years of success. The menu sucks, the food isn’t what it should be, the main dining room is not attractive, and we have far too many cooks in the kitchen. But the worse thing is that few, if any, really know what sales are, how much stuff is costing us, or if the things that have been changed are helping or hurting.

And unless Chef Ramsey (I.e. Demonic) comes in an kicks some serious ass here and makes sweeping changes, this restaurant’s outlook does not look good.
eph