Distributed testing?

[ycleption]

Okay, so much has been made about balance issue in V3 (please don't start arguing about that here). Would it be possible for future releases, (maybe even this one), to recruit a group of testers, give them each a module that includes two or three areas from the game? (this would alleviate fears of stealing the game and such). Each tester could run a number of characters of varying class and level through the areas, and report back to the team. With a sizable group, more might be able to be tested.
I have no idea if this is feasible or not technically, just throwing the idea out there...

[Acacea]

Well... just from my opinion, if you think about how hard it is to get a large number of testers (not a small handful, but really a lot in this example, because Layo has so many areas, that's all), it would seem like a lot of work and organization simply assigning areas, handing them off, etc. Also, you would need to be able to test with all the systems that are in place here, because what if one of the things that work without them, glitched with a current one? That happens a lot, too. But part of the problem is the full script set; more than the actual areas, I would think, because that's where a giant bulk of the module is - the scripts, items, whatever, right?

It would seem like it would just be simpler to host the full module, have a few people start up new characters, have others import old ones, and just run around trying to break stuff.

[Dorganath]

It's not a bad idea, but we can't really do it that way.  What would be more likely is to put up a "beta" on another server and recruiting people to join over there.

There are several reasons why it has to be this way:
1) while the hundreds of areas are a huge part of the effort in the module, no less important are the over 2200 scripts and I don't know how many items and such that are all custom to Layonara.  Selectively culling out only those things that would be needed to test bits and pieces would be prohibitive, and passing them along in bulk is just not an option. And some things will simply not work without the scripts in place.

2) There is little difference in effort to create a Layonara module with 3 areas as opposed to one with 30...or 100...areas. Unfortunately, that effort is not trivial, especially if each tester gets a few different areas.

3) Many things in our module rely on a proper connection to the database in order to function properly.  The database touches many aspects of the servers, from something as complex as the crafting system to something as mundane as resting. Testers would need to set up a copy of our database to test things properly.

These are the major things, and so you can probably see that dividing things up like that would be very technically prohibitive.

Having said that, it may be possible, if there is enough interest from players, Project Team and GMs to establish a sort of beta testing program whereby significant new or changed features could be put through the paces in a more diverse environment. This would require:

* a test server
* a separate copy of the database
* a separate server vault
* ACTIVE participation from beta testers, GMs and Project Team members

The first three exist already to some degree, though are all fairly far behind in terms of being kept up-to-date. Yet, that is something with a reasonably simple remedy.

On the last item, this is the most important cog in the machine.  Without it, the whole thing would fall apart.  I would point to the LSPM project and how much trouble Talan and DMOE had in recruiting beta testers and keeping them on task. A small number of people produced some very useful and consistent bug reports (and kudos to those who did), but overall, there was a lot of interest and little participation. It is my personal opinion, without drawing undue attention to anyone in particular (and no, I don't have specific people in mind), that most people don't want to work to have fun, and beta testing, done right, is too much like work.

So!

While your initial idea will not work for us, it may be possible to establish a group of people who will be available to assist in testing some updates before they're posted to the live servers.  I'm not saying that releases will be tested as such (not all will warrant it), but it is perhaps an idea worth considering.  As I said though, participation is the key.

With all that said, I'll step back a bit and let the idea, if it is to progress, take more shape given the initial framework I have described above.

[darkstorme]

W/r to the scripts, at least, unit testing could be done on those before they go live.  There's no guarantee they'd interact the right way once they were in game, but you could be sure of their integrity on a piece-by-piece basis...

While I realize the database is required for most everything, setting up a "beta" server just says "more money" to me, and given the rather limited income from advertising on the site and the limited donations (at least until next Donate Your Level Day), we'd probably best be limiting that.

[Dorganath]

W/r to the scripts, at least, unit testing could be done on those before they go live.  There's no guarantee they'd interact the right way once they were in game, but you could be sure of their integrity on a piece-by-piece basis...

While I realize the database is required for most everything, setting up a "beta" server just says "more money" to me, and given the rather limited income from advertising on the site and the limited donations (at least until next Donate Your Level Day), we'd probably best be limiting that.

Scripts are tested before they go live.  Anyone who is responsible for the final merge of the modules has a full runtime environment on their local systems.  In other words, I myself have a copy of the database, my own isolated server vault, NWNX and all the other goodies it takes to run and test the modules.  The only thing I lack is multiple people.  I often will ask someone to log in to my local server and ask for some assistance in testing when I cannot test solely on my own.  Any scripts submitted from a Project Team or GM Team member are expected to be tested (as much as possible) before they're even sent for inclusion, and they're tested again after they're integrated with Layonara's code base.  It is sometimes the case, however, that some things simply do not get exposed properly until put through their paces live, or at least in a more suitable multi-player situation, which is why I suggested the beta server possibility.

On the test server, there is already a server we can use for this so there's no extra expense, only time and effort to set up, maintain and administer it, which is still far less than chunking up the modules and distributing them to people.  As I said above, the hesitation is one of the level of active and consistent participation.

[Leanthar]

"....As I said above, the hesitation is one of the level of active and consistent participation...."

And that is the key. We need an active project team. Well over 1/2 the project team, and usually around 2/3s or more of it are inactive at the same time for various reasons. Some times RL, some times burn out, some times just not wanting to do anything for any number of reasons. Really we rarely have more than 3-4 active people really being active at one time on the Project team. It has been that way for years and it will always be that way in a non-paying and totally volunteer situation. We can't do much when we are expecting things and they never get submitted or things get submitted very late or extremlly buggy. This happens far more often than it should but there is little we can do about it since there is no pay and it is voluntarily. This was even true when working on a really tough schedule for the first V3 update, quite a number of the project team just didn't produce and a few that did produce we had to throw away and start from scratch due to the quality. But that is because many of them are learning how to work with the tools and/or have never worked with this sort of thing before and that is okay. We like to teach and train people, we just need those people to continue to be active after the training which doesn't happen so often. And it is also because they do this on a volunteer basis and have RL schedules as well.

I mean right now we have 100's of area reworks that we want completed and they are available to be picked up at any given time by any project member. But, other than Pankoki, we only have one person really working on creating/redefining areas, two from time to time. Not much we can do about it though, it is a volunteer project and everybody does it because they want to do it and in some way it is fun for them. We also have a huge, and I mean huge, system project but that is being ignored like the plague *chuckles* and I totally understand why. The effort to tackle that beast of a project is monumental and would take a few programmers a very long time of hard and dedicated time and effort, at least 1000 hours and I would bet about 2500 hours in the long run.

Now don't get me wrong many on the team are very active and do a very good job but fully at any given time 1/2 - 2/3 of the team might as well not be on the team because they are not active and perhaps worse yet not communicating with anybody on the team or in the team forum. But when they do get active they tend to produce some really good work, and it shows in the game world.

[Nibor21]

One of the issues of testing is ensuring the criteria to test to.

It is difficult to test every possible option, and in fact one of the keys to scripting (and programming in general) is not making a system that does what you want, but making a system that can cope when it is fed an unexpected input or is used incorrectly.

It is even more difficult to test these.

In terms of testing for balance then there is another issue. You would need a test group who each has access to PC who cover a wide variety of levels (as you need to test the entire scope of an encounter). You would ideally need to try testing with different party types and configurations etc.

Layonara is not a professional organisation with a team of full-time test engineers. It is therefore a difficult decision for people like Dorg to make against time spent testing to make sure it works against getting the new system out. There is little doubt that given a big enough team, a decent budget and very very large dollup of time V3 could have been perfect from day one. However i urge you all to look at what a mess Obsidian made with NWN2 and its release. And they had most of the above!

So when you look at v3 and what has been produced with the resources available is really rather stunning

[Falonthas]

maybe assemble a testing team
people that are just good at breaking things that can be broken

then the project people can do project stuff and the testing people can just make chars to try and break it
all sort so builds and combos
make several chars at stage increments
lvl 3 6 9 12 15 18 21
by that time being really higher then 21 wont matter as its not something you can get in a months time without playing 24/7 for 30 days grinding


i know i for one would volunteer to cast and break and see if it shatters