Beta Testing and You!

[nipsen]

On the occasion of the beta-test opening for external testers soon, some of you may wonder what a beta-test actually is. Or, you may not, I do not know. There are some common misunderstandings surrounding beta-tests lately though, that I do know. And perhaps the misunderstanding is mine, that is certainly possible, as the evolution of the games-industry naturally evolves to develop one curious survival trait after another.

 

Is the beta-test an early look at how the game plays for promotion purposes, or is it a test to figure out where the breakage is? Is the beta-test a way to help fine-tune the mechanics, or is it a way to get free focus-group feedback? Perhaps a little bit of all of that? But questions that may or may not be known on beforehand, even in the most structured tests.

 

So after somehow ending up being part of some ten-fifteen beta-tests (closed and open) for games in the last few years, I'll here present a list of things that would have helped immensely for all participants to be aware of before they started, in each one of them. Perhaps this will be useful for the upcoming Pillars of Eternity beta as well. I know a lot of people who will be in the test has a lot of experience with testing, or at least are reasonable people. But this might help the rest not scoff about elitist entitlement princesses stealing their thunder - or try to compete for attention in the wrong way later.

 

1. The developers are actual human beings. 

-that the people sitting behind the curtain make mistakes, have expectations and wishes, etc., came as a tremendous shock in some of the tests. In one case when solutions they were extremely proud of got completely overlooked, this actually helped scuttle implementation variants over other less experimental and more straightforward (and boring) ones. It happened when the negative feedback was more frequent with the alternative than the safer solution. And the developers didn't have the guts to fish for feedback from people who actually enjoyed the experimental implementation, even though they favoured the alternative themselves. And the silent failure of the alternative had huge implications for the depth of the game when it came out. After the beta-test, it turned out that many people enjoyed it, and said, frequently, that they expected it would be in the final build simply because it worked so well. This feedback never reached the devs.

 

And how could it have? They wished and looked for it - but the situation really had no possibility where that feedback would actually have turned up.

 

So that the solution didn't have obvious issues to be worked out, but caused certain people to dislike it, made it fall off the map. Failure by the developers to explain what they were looking for and how much time they had actually spent polishing the solution they wanted was one problem. While the absence of a context given by the developers for the feedback was another. The developers let the most engaged feedback dictate what was focused on. And essentially believed blindly that the beta-test audience that provided the most feedback completely described the focus-group - as well as the complete audience.

 

What really happened was that each tester had their own context for their feedback. Which then the developer mistranslated into their own larger one. The breakage simply happened because the testers never were properly primed about what they were supposed to be looking for, and what the devs specifically would have liked to hear about. And I've actually seen this happen with fairly experienced developers, when they have had community managers gather feedback and put it into a compact. The developers just never had a chance to understand the context of the actual feedback, and had to extrapolate. Which they did.

 

As testers, consider that when providing general feedback. That it's allowed and maybe required to ask questions about the intention behind a certain solution. And gear the feedback into that very specifically. To avoid having your opinions misinterpreted, but also to get something useful back to the devs.

 

2. Your opinion doesn't have weight without a real explanation.

-since we're all here on the intertron in the first place, that is obviously a very unorthodox idea. Opinion typed down! Level UP! But the biggest failures in the beta-tests I've been part of have turned up because there's no discussion involved with the opinions that are stated often. And "popular" opinions are determined by how many ditto-thumbs the opinion gets.

 

We have the same tendency with video-game reviews. An explanation and a set of discernible reasoning invites criticism and uncomfortable challenges. That in turn is difficult for commenters and writers to engage with, since reasonable people naturally want to avoid intertron entertainment like that. While just a statement without reasoning is unassailable. So you like Rhianna. No comments, fifty likes from people who also like Rhianna. We can therefore deduce that Rhianna rules the world for this focus group.

 

In turn, this encourages lots of people favour simply stating their opinion and focusing on having it sound palatable (when pushed to have one), instead of explaining why they like or dislike something.

 

In beta-tests, the problems turn up in the form that people might like Rhianna - but testers fail to explain that they like it because everyone else seem to like it, and that other people say that it's really good. It might be great for all we know. But we don't really know why, or even what exactly snuck in as being obviously gratifying to the tester when seeing what they reacted to. Even in closed betas, this happened several times because testers weren't completely comfortable typing down explanations, or were very bad at being concise, or simply didn't think about what they were doing. The ones that were fond of typing down their thoughts flooded the feedback loop with bs.

 

Keep in mind that when the first inevitable, but always unexpected, situation comes up when some feature or other appears to cause a problem for several users at the same time. And therefore might very well be universally hated or loved. Why does it engage the specific person who has the opinion? That's what we want to know. Does the tester really believe it's great because they expect /other people who are not testers/ to like it? If you can find out, there's suddenly value to even the most idiotic and egotistical opinion, as well as the most utterly obvious and flat observation.

 

This also goes back to #1, in the sense that actually figuring out why people think what they do avoids filtering ambiguous feedback into the expectations the developers and their community people had on beforehand.

 

("Yes, it seems that there's a tendency here that suggests..." - it's very easy for that to happen, even for clever, critical and intelligent developers. You don't need to look further away than Broken Age to get a great example of it - focus groups allegedly uniformly reported that puzzles were difficult, and stumped players at several points. But no one could actually explain how that was technically possible in 99% of the game, or where it actually happened. Devs then overlooked finicky parts that really were there, that fresh eyes discovered very quickly. And they axed challenging elements they knew /might/ cause .. some imagined group.. of people to throw something at the screen and stop playing. Since those parts were the only elements they knew about that could cause people to pause on beforehand. Purists in turn defended finicky parts that should have been looked at again, believing they were defending something else).

 

3. Reproduce your bug before reporting it (or explain what you attempted when failing to reproduce it).

-technical stuff -- if Obsidian is interested in this, or if the beta hasn't progressed further than normal tests. I realise that people have paid for early access and don't have any obligations to do any work when playing through the test. But everyone sees the difference between: "The fighting animation looks weird". And between: "The fighting animation when clicking a second target after a triggered effect, wielding these two weapons, has a hiccup". It's also useful if you can say: "I used the soul siphon and there was a misstep in the animation that I couldn't reproduce afterwards". Or even "Is the character supposed to take a step back when triggering such and such effect?". "Why is the wizzard turning before casting the actual spell? Does he turn twice? Is he supposed to? I think this looks... because.." and so on. Since then this is the kind of thing that leads to figuring out how maybe targeting moving objects suddenly breaks a set of animations, for example.

 

I'm imagining that when adding the perks during the beta, things like that might turn up - triggers that have stalling animations but a target afterwards, etc.

 

4. A picture says more than a blog-post, at least.

-get some sort of screencap tool. Or maybe Obsidian has designed a way to dump a screencap along with the state information at the time? "I saw a glitch in the floor of the dungeon". Worst thing you can possibly read when trying to actually find bugs. 

 

5. Discuss things with the devs?

-what is supposed to happen? What actually happens? How did this ability actually end up being used? How was it imagined by the devs it would be used? Is it as self-explanatory as it looks? Does the explanation depend on knowing how it works on beforehand? Are you forced to choose something and not see the impact of it until you can't make an informed guess any longer? Does choosing an attribute over another impact this or that - can you spot the ruleset as you play, or when does it make sense? Are the explanations both technically and narratively sound? When did the explanations make sense to you? When could you use the explanations to make informed choices?

 

6. Avoid becoming starstruck.

-also see #1. One beta effectively ended when a developer started to explain what they wished to hear. But the setup to that situation only worked because everyone in the beta wanted to elevate the person to godhood on beforehand. Other betas have had invisible devs simply because they want to avoid tainting the feedback. But also because they can't really participate comfortably. Be respectful even if the other people are complete idiots. Work towards making the environment comfortable as testers. For the other testers, and for the devs in turn.

 

And a good sideeffect of that is that people who otherwise might not have participated with feedback actually will. And frankly, those guys - the guys who don't really want anything to do with the test and the feedback on the annoying forums, etc. - those are the ones you always want to get in.

 

---

[Lephys]

*applause*

 

Our mantra will be "Be Specific!"

[Ztirual]

*applause*

 

Our mantra will be "Be Specific!"

*Cheerleading* - "Be specific - be, be specific!"

[Sensuki]

I've got Bandicam set up so I can record examples of any of the bugs I find or demonstrate changes etc.

 

Hopefully they'll also expose the Unity log, so we can send those back with the try-catch errors that it logs.

[BLnoT]

Very nice post Nip, certainly some points to ponder on. I had never thought of seeking to reproduce an error before reporting. So thank you for that tip.

[DCParry]

*applause*

 

Our mantra will be "Be Specific!"

 

You sound like me in front of my freshmen writing class!

[redneckdevil]

Very good. Im looking forward to not only playing the beta but also as my first beta im doing, im looking forward to the info on where/how/what/etc to report.

 

I think they might wanna open up a whole new channel for the beta testers to discuss and bring up topics so its not drowned out or mixed up with the regular non-beta posts and discussions.

[Lephys]

Hopefully they do. It's always a lot easier (for both parties involved) when there's an established feedback system.

[Zwiebelchen]

How to reproduce an error is the most important thing when reporting bugs.

 

I absolutely hated it when people just told me that "this spell was not working" in my mods. What exactly wasn't working? What happened when you tried to cast it? What were you doing when the error occured?

 

 

To fix a bug in a large scale coding structure like a game is extremely complicated when you get no information on how to reproduce the bug. Because then you try it out in your test build and everything is working fine and you wonder what the report was actually about (because the ability only didn't work when combined with a certain talent, etc.)..

Edited August 1, 2014 by Zwiebelchen

[JFSOCC]

I hope people will report more than bugs, but also provide their opinion on balance, pacing, narrative, artwork, and design philosophy of the experience they encounter

Edited August 1, 2014 by JFSOCC

[neo6874]

I hope people will report more than bugs, but also provide their opinion on balance, pacing, narrative, artwork, a design philosophy of the experience they encounter

^this, except the Beta areas are kind of "outside" the main storyline (IIRC, just a couple of wilderness areas), so the pacing and balance may be broken.

 

I'm thinking (hoping!) it's like BG (etc) -- you're not outright blocked from going anywhere, but if you go to the wrong place, you're going to quickly find yourself in trouble (although, there were many times in BG where that got frustrating).

[IndiraLightfoot]

Alright! We're in the very month, the beta will be available, I'm so stoked and pumped that I get to try out this beta build, and I really hope it allows for quite a few levels with as much character creation options as possible (although Josh has already warned us about there being a bare-bone assortment of talents).

[redneckdevil]

Sigh why is it everytime i visit these forums "its the final countdown!" song keeps popping into my head thinking about the game and beta

[Zinnwaldite]

I would very much like to be a part of this beta testing,, but it seems i can't change my existing pledge to add beta to it. I will watch silently from the shadows though.

[NWN_babaYaga]

APOCALYPTIC EVENT is coming ...beta...oh wait it´s the beta? YES the beta dude now get ready with your survival hand book and the holy bible and prepare for the COMING of the christ!...

 

No that was just stupid and wrong it´s just the beta dude chill out and do your homework once you got it ..ehehehe

 

and dont forget. NEVER drop out of school because of a beta :D

 

HAHA... why dont you guys let the westboro church members throw "white" curses of godly destruction coming from the abbys of heaven at your game. I´m sure that would be a marceting hit!

Edited August 1, 2014 by NWN_babaYaga

[redneckdevil]

For people who are wanting to change their pledges, email the obsidian support and they will let u change ur pledge. You do NOT have to do a new pledge, just simply email obsidiansupport and they will let u change it. The option to change has not yet been implemented yet onto the backer site.

[AndreaColombo]

How to reproduce an error is the most important thing when reporting bugs.

 

I can't stress enough how fundamental this is.

 

If there is one thing I have learned in my (albeit limited) experience in the field, is that there is only one thing that makes a good bug report: crystal-clear, detailed repro steps. A screenshot of the issue may add value depending on the issue; everything else ranges from optional to time-wasting.

 

Nothing helps a developer like the ability to quickly reproduce the bug without wasting time wild-guessing what it might be.

Edited August 1, 2014 by AndreaColombo

[Monte Carlo]

Will Obsidian use meta-data to track how we play?

 

I hope so, that way I feel like I'm helping out by just... playing.

 

#lazyb@st@rd

[robertpoche]

so any date set?

[Ark Evensong]

For the Beta? Yes, August 18th.

Final release? No. "Winter this year."

Portal upgrade so you can edit your pledge and buy into the beta? No. "Soon."