Advice on testing applications

[Evenstar]

Hi!

 

I usually shy away from forums, but I saw this Developer's Corner and was intrigued.

 

I've been developing software for many years now. These have been primarily government contracting jobs, but I've done a couple of commercial apps as well (one of which I am trying to market.) My question is very basic...

 

When it comes to testing software applications, what are the best approaches?

 

I ask this because, at present, the project that I'm currently working on has no real testing procedures or policies to speak of. In fact, we don't even have a test lab. It seems (with at least my organization) that testing gets put off and put off until it is just too late in the development cycle to even matter. Try as I do; I don't write bug-free code. I can't imagine any programmer who does, but obviously some bugs are harder to find than others.

 

As for testing, the company I'm with has just started seriously thinking about a test lab. What sort of computer systems does Obsidian use (i.e. Dell, Gateway, home built?) How many? Do you automate your tests? What sort of tools do you use (commercial or in-house?)

 

Thanks,

Brian

[TentamusDarkblade]

Hi!

 

I usually shy away from forums, but I saw this Developer's Corner and was intrigued.

 

I've been developing software for many years now.  These have been primarily government contracting jobs, but I've done a couple of commercial apps as well (one of which I am trying to market.)  My question is very basic...

 

When it comes to testing software applications, what are the best approaches?

 

I ask this because, at present, the project that I'm currently working on has no real testing procedures or policies to speak of.  In fact, we don't even have a test lab.  It seems (with at least my organization) that testing gets put off and put off until it is just too late in the development cycle to even matter.  Try as I do; I don't write bug-free code.  I can't imagine any programmer who does, but obviously some bugs are harder to find than others. 

 

As for testing, the company I'm with has just started seriously thinking about a test lab.  What sort of computer systems does Obsidian use (i.e. Dell, Gateway, home built?)  How many?  Do you automate your tests?  What sort of tools do you use (commercial or in-house?)

 

Thanks,

Brian

<{POST_SNAPBACK}>

check out this book and any book by Cem Kaner. All of MS test reads these books. Nearly ever successful game and app tester has read the books by this guy.

[Evenstar]

check out this book and any book by Cem Kaner.  All of MS test reads these books.  Nearly ever successful game and app tester has read the books by this guy.

<{POST_SNAPBACK}>

 

Thanks. I'll certainly look for it.

[Feargus Urquhart]

We pretty much work entirely on Dell Systems. We have a number of different models, but mostly 4600's, 8400's and a few Precision 470's.

 

As for automated testing. We are doing a certain amount of automated testing on the new NWN2 world editor. Since we decided to completely re-write the editor and wanted to be able to load a certain amount of NWN1 material into it, the programmer on it has used some data checking programs to make sure that the same stuff that goes in, is the same stuff that comes out.

 

When it comes to game testing, we don't really do any automated testing. Most game apps are hard to automate when it comes to testing, since the interfaces are fairly non-standard.

 

What ends up happening is that a bunch of people are thrown at the game and split up amoungst different areas of the game. On KotOR2, there was a d20 tester, and then a tester in charge of each of the worlds and then light side quick play through testers and dark side quick play through testers, long play through testers and then just other random testers playing through.

[Evenstar]

We pretty much work entirely on Dell Systems.  We have a number of different models, but mostly 4600's, 8400's and a few Precision 470's.

 

As for automated testing.  We are doing a certain amount of automated testing on the new NWN2 world editor.  Since we decided to completely re-write the editor and wanted to be able to load a certain amount of NWN1 material into it, the programmer on it has used some data checking programs to make sure that the same stuff that goes in, is the same stuff that comes out.

 

When it comes to game testing, we don't really do any automated testing.  Most game apps are hard to automate when it comes to testing, since the interfaces are fairly non-standard.

<{POST_SNAPBACK}>

 

This is actually what I figured would be the case. I shudder to think the kind of logic that would have to be programmed in to automate a user's actual game play. I'm sure it could be done to some degree, but it wouldn't be worth the effort (might look good on a resume, though )

 

What ends up happening is that a bunch of people are thrown at the game and split up amoungst different areas of the game.  On KotOR2, there was a d20 tester, and then a tester in charge of each of the worlds and then light side quick play through testers and dark side quick play through testers, long play through testers and then just other random testers playing through.

<{POST_SNAPBACK}>

 

Forgive my ignorance, but what is a d20 tester? My first guess would be a combat tester.

 

In a previous job, I worked in the drafting group for Solid Edge. Generally speaking, the developers and the testers worked together to solve problems; but fundamental GUI changes, etc. couldn't be introduced without jumping through 2^n hoops. The bug tracking system was ok, but it definately could have stood some improvement. There was one particularly nasty (at least in my way of thinking) bug involving corrupt data files. We would get files in from customers and rather than try to fix the corruption at the core (i.e. the code), our directions were to try and fix the actual data files. I never could understand that. 99% of the time, the file just wasn't fixable. The management was convinced that the save routines were perfect and couldn't possibly be at fault. Talk about frustrating! I mean, how many times have you saved a CPP file and found it to be corrupt? (Please don't say, "Gee, that happens all the time.")

 

In my current job and my "at home" project, however, I am the tester and the developer. Obviously, it's not the best scenario. There is a design spec for the project, but that's about it. How long is a typical design process for a game like NWN2 or KOTOR2? Once it is set, what is involved in changing it?

 

As for testing, I am proud to say that with my side project at home, none of my 4 (yep, four) customers have had any major problems with the toolkit I've written. I've yet to talk to my marketing person about those sales! " Of course, that's an entirely different topic.

 

A word to the wise:

Trying to code, test, debug, document, market, create a website, do tech support, and otherwise sell a software product BY YOURSELF is an extremely DAUNTING task.

[kirottu]

Forgive my ignorance, but what is a d20 tester?  My first guess would be a combat tester.

<{POST_SNAPBACK}>

 

Kotor and kotor 2 both used d20 rules system. It