三种游戏用户研究方法之优势编辑本段回目录
我将在本篇文章以及接下来的后续文章中阐述开发者能够用于游戏用户研究中的一些不同方法。
在此我将涉及游戏用户研究的某些要点，并详细讨论3种方法，焦点小组，启发式评估以及问卷调查等内容。
playtesting(from gamasutra)
什么是游戏用户研究？
开始之前我们必须先搞清楚什么是游戏用户研究？首先，让我们将其与质量保证（QA）进行比较。QA是软件开发过程中的一大组成部分，并且总是由一些开发团队中一些专业人士去执行这一任务。这些专业人士的主要目标便是寻找漏洞并确保游戏的顺利运行。
然而，由一些致力于某一项目的人员去执行QA也就意味着这些人士非常专注并熟悉这项工作。而这恰好也是问题的根源，特别是涉及游戏可用性和游戏体验的评估时。因为对于那些专注于项目中的人员来说，他们认为很明显或者很有趣的内容对于新用户来说可能是完全陌生且让人沮丧的内容。
所以这时候便需要游戏用户研究方法的介入——即完全专注于用户以及他们游戏体验的领域，并且这一领域所关注的主要问题是“玩家是否会觉得游戏有趣？”
简单地来说，QA和测试是关于软件，以及它是如何在用户面前运行的，而游戏用户研究则正好相反，它是关于用户，以及用户是如何面对软件。
所以，如何才能做到这一点？并且什么才是真正的乐趣？乐趣是一种很容易使用的元素，但是它同时也会伴随着奋斗和挑战的出现。有趣的体验，吸引人的体验或者轻松的体验都能够为玩家带来乐趣。
如此看来乐趣是一种主观的变量，即不同人在不同情境下对于乐趣的感受也不同。而因为乐趣拥有情感成份，所以可以说当玩家在你的游戏中感到乐趣时，他们便会想要告知你——但是前提是你必须知道如何去询问玩家的这种反馈。
那么应该如何询问玩家的反馈呢？
在开始阐述细节内容之前我想先列出一些重要的基本原则：
明确目标玩家
不论你选择的是何种方法，你都需要明确正确的玩家。这就意味着你需要获得富有代表性的用户，也就是那些你希望能够玩这款游戏的玩家。
如果你拥有大把的时间，你当然能够尽可能地拓宽用户群体（如果你真的认为所有人都会喜欢游戏的话），但是通常情况下你总是会受到时间（以及金钱）的限制，所以你最好能够专注于那些你所明确的目标玩家。
测试游戏而不是玩家
其次，当你在进行这类型的研究时都必须确保让玩家清楚自己并不是测试对象。研究是关于如何完善游戏而不是玩家，所以即使玩家在某一方面做得不好也不用因此感到自责。原则上来看，他们所提供的所有反馈（不管好坏）都是非常有价值的信息。
但是做到这点却不容易，因为你参与了游戏设计工作，所以如果听到别人对于游戏的批评肯定会感到不舒服。但是不管怎样你都要想办法克服这一障碍而让自己更加客观地接受玩家反馈。
明确自己要了解的信息
每当你在进行研究时，你都需要明确自己想要知道哪些内容。你在创造游戏时肯定会亲自尝试游戏，并且因为你拥有明确的设计文件，所以你知道游戏中的每个环节是如何运行的。所以不要责怪那些输掉游戏的玩家，而是仔细想好自己要问他们哪些问题。仔细思考哪些区块是问题所在，并在测试之前搞清楚自己想知道些什么。
请注意：我并不是说你们带有偏见或者已经“知道”自己想要的答案；我只是希望你们能够准备好并知道自己真正在寻找哪些内容，否则你将只会收到一些无用的信息。同时，你还需要学会处事不惊，因为你根本不知道自己会收到哪些答案。
尽早且频繁地进行测试
这是用户研究中最重要的一点，也就是你应该在自己认为合适的时间尽早进行测试。但是做到这点却很难，因为你可能在游戏还未完善之前就将其暴露在用户面前。但是你却可以因此更早地进行测试——例如测试纸上原型。
如果能够尽早进行测试，你便能够及早发现问题并即时做出修改。而当你做出修改后，你必须再次进行测试。也就是说你最好确保游戏不会漏洞百出，因为当你在测试游戏体验时，你肯定不希望因为漏洞而破坏体验。
一种有效的方法便是微软的游戏用户研究员所研发出来的快速迭代测试和评估方法（RITE）。在这一方法中，研究员需要不断运行测试（经常是通过行为观察），并在发现问题时立刻进行修改（直到面向下一名测试者）。有可能在第一名测试者完成测试时便会发现问题。
听取问题并付诸行动，但是测试者所提出的却不一定是解决方法。
当你在面对用户时，你需要足够开明地听取他们所提出的各种问题。你同样也可以听取他们所提出的解决方法——但是这一方法对你来说不一定有用。你是游戏开发者，所以你知道该使用自己所拥有的技术，时间和资源完成某些内容。但是你的玩家却不行。所以当玩家提出相关问题时你一定要正视它，并进行一定的研究，但对于玩家所提出的解决方法你可以持保留态度。
游戏用户研究是另外一种数据来源
通常情况下像我们这类文章发布到网上时，便会有人担忧这些研究是否会破坏游戏的艺术性或者是否会催生出“评委式设计”等问题。当然了，我理解这种担忧，但是与QA一样，用户研究也只是帮助你完善游戏的一种工具罢了。它不会主导你的设计或压制你的艺术才能；而如果你能够合理利用这一工具，不仅你的才能有望得到进一步扩展，你还能够以一种全新的视角去看待游戏。
细节内容
我将在之后的内容中从各种深度分析：焦点小组，启发式评估和问卷调查。
焦点小组
这一方法本身就存在一定的问题（其实它也是“评委式设计”问题中的一环）。所以我们将最先解决这一问题。
你可能已经非常熟悉焦点小组了——尽管你可能从未亲眼见识过。基本上来看，焦点小组就是你号召一群人来玩你的游戏，然后将他们汇聚在一个房间里谈论游戏。他们可以很自由地谈论自己的观点，不管喜不喜欢你的游戏；而你需要在房间里安排了一名引导者，负责询问一些特殊问题。
focus group(from techblog.dallasnews.com)
我们也可以在开发的早期阶段使用这一方法（而不是让人们单纯地玩你的游戏），即你可以阐述关于游戏的相关理念并获得相关反馈。
焦点小组的优点便是它们总是包含许多参与者，而你能够因此获得更多反馈。这是一种相对有效的方法，即所有人都聚集在一处地方，由引导者负责向他们提问。所以如果有人提及他们喜欢或不喜欢的内容时，你应该让他们更加详细地说出原因。
如果处理不当的话，焦点小组便会成为一种浪费时间的无用方法。所以为了避免这一问题你就需要安排一名优秀的引导者去组织你的焦点小组。引导者必须拥有足够的能力以有效地引导对话，同时也不会绝对地主导着小组成员的讨论。
焦点小组所存在的最大风险，也是为何研究员很少使用这一方法的最大原因便是：有1至2名的焦点小组成员会主导整个讨论。迫于小组的压力，你甚至不会听到其他人关于游戏的更有价值的想法。并且焦点小组总是倾向于讨论问题的解决方法而不是问题本身。而这正好都不是你所希望看到的。
最后，焦点小组还是一种非常主观的方法，也就是我们所获得的反馈都是来自于不同人的看法，我们还需要去判断他们所持有的不同“态度”，思考他们的可能性行为，并且我们也清楚人们不一定总是会按照自己所说的去行动。
优点
参与者越多也就意味着能够收到越多反馈
能让所有人汇聚到同一个地方
允许在讨论后提出问题
如果是讨论理念便非常有帮助
缺点
要求必须要有一名优秀的引导者
主导者总是会抢尽风头导致其他人不能发表自己的见解
一窝蜂地讨论解决方法而不是问题所在
人们的所言并不一定是其所行
启发式评估
启发式评估是指你找到游戏用户研究领域的专家，让他们帮你玩游戏，并基于一定标准（启发法）进行评估。有点像系统式游戏评估。
基本上来看，这些专家都需要使用一系列启发法——一些基本规则或心理模式，反馈你的游戏是否匹配这些启发法并提出问题所在。启发法多种多样，我将在此列出Christina Koeffel及其同事在2009年所列出的一些可能的启发法：
是否提供了明确的目标？
是否提供给玩家有意义的奖励？
玩家是否能够掌控局面？
游戏是否保持平衡？
游戏是否留给玩家一个好印象？
游戏中是否拥有一个好故事？
游戏是否能够有序地发展？
游戏是否具有一致性并且能够得到响应？
玩家是否能够清晰地知晓自己失败的原因？
游戏中是否拥有各种难度等级？
游戏以及游戏结果是否公平？
游戏是否具有重玩价值？
游戏的AI是否明显，是否一致，或者是否存在一些不可预知因素？
游戏是否会让玩家感到沮丧？
游戏中的学习曲线是否过于起伏不定或者太长？
有何情感影响？
是否存在过多无趣的重复内容？
玩家是否能够认出屏幕上的一些重要元素？
我认为启发法最让人惊喜之处便在于即使你不是专家，你也能够基于这一系列内容更好地去思考你的游戏。例如，你的游戏是否提供给玩家足够的反馈，并让他们知晓自己的行动对游戏世界会产生何种影响？游戏是否强迫玩家以一种笨拙的方式去使用控制器？当然了，这些都是一种常识，但是你常会发现这些所谓的“常识”却并不为人所正视。
而启发式评估的一个明显的优势便在于你只需要使用少数的专家（有时候甚至只需要一名），并且专家们也总是清楚自己在讨论些什么。但这一优势同时也是问题所在，因为你需要思考该去哪里才能找到这些专家？以及你是否能够找到合适的专家？不同专家可能会使用不同的启发法，而你还需要确保这些启发法能够适用于你的游戏——这一点非常重要。
但是有时候专家其实都太过专业，常常会忽略一些新手们可能会遇到的问题。这是因为当我们变得更加有经验时我们便不会下意识地考虑那些我们所察觉到的一切，然而那些仍然处于学习阶段的人却仍需要思考这些内容。这也是为何想要做好一件事需要遵循专家的做法，而想要学习一件事则需要咨询新手的道理。
优点
不需要太多人参与
能够较快速地解决问题
专家就是专业人士
缺点
你要去哪里寻找这些专家？
你是否能够找到合适的专家？
专家都太过专业
问卷调查
你肯定知道什么是问卷调查，但是你是否知道如何正确设计并使用这一方法？
首先，你需要清楚何时能够使用这一方法？问卷调查是关于玩家对于你的游戏的主观评价，特别是关于价值的判断。而这些调查的范围更是广泛，包括玩家喜欢何种武器以及一些更开放的问题如他们对于游戏体验的总体评价等。
开发者能够在游戏期间向玩家发放问卷调查，因为这时候他们正切身感受着游戏体验——但是这么做的风险则是会打断玩家的游戏进程（游戏邦注：所以开发者最好寻找一些更加合适的暂停时间去提出这些问题）。开发者也能够在游戏结束后进行问卷调查。而问卷调查的一大优点便是它是面对大众群体，也就是开发者能够因此获得更广泛的研究数据。
在详细阐述如何创造属于你自己的问卷调查之前，我想先列举出一些致力于评估游戏体验乐趣的问卷调查案例。
这类型问卷调查的例子包括“游戏体验问卷调查（GEQ）”——用于调查玩家的游戏体验以及影响规模或者针对于不同情感等级的人体模式系统。这些问卷调查都非常优秀，因为其中的问题都是设计师精心设计而成的，可以说都是可信赖的问题。但是通常情况下这些问卷调查问题更倾向于学术性，所以你需要谨慎设置这些问题以让它们能够更好地反映你的游戏，并在适当的时候做出调整。
所以，你到底该如何设计你自己的问卷调查？以下我将列出四大步骤：
步骤1：明确你想知道些什么
就像之前所阐述的，所有的这些方法都需要你事先明确自己想要了解的信息。特别在问卷调查中更加重要。因为你不可能不断追踪着每个玩家的答案，所以你肯定希望他们能够尽可能清晰地回答你的问题，并且你也需要确保你所获得的信息就是你想要知道的内容（保证这些问题不会太多也不会太少！）
你可以通过头脑风暴，制作列表等一系列方法明确自己到底想了解些什么。然后再进行压缩，留下那些自己真正需要知道的内容。切记一定要有所侧重！
新手研究员经常会犯的一个错误便是认为机会很多而一直问一些非常简单的问题。但是如此便会创造出一大堆需要分析的数据并最终促成一些没有意义的结果。除此之外，你肯定也不希望玩家需要投入超过15至20分钟的时间去回答你的问卷调查吧！
步骤2：设计内容
内容的设计是创造问卷调查的核心内容，我们可以将其分解为以下内容进行分析。
问题或陈述？你希望调查对象是回答问题还是进行陈述？这一点非常直接但同时也非常重要。一般来看，问题更能够帮助你获得想要的信息，例如：
Horrible Bog Beast具有多大的挑战性？
1 2 3 4 5 6 7
非常简单 非常困难
问题也能够以指令的形式表现出来。例如：“根据1至6个分数点为你刚刚玩过的游戏关卡评级，1代表你非常喜欢这一关卡，而6则代表你非常讨厌它。”这是一种非常有效的方法，就好象你在询问每个人对于不同关卡的评价等级一样。
Horrible Bog Beast是玩家所遭遇到的一个有趣的敌人。
1 2 3 4 5 6 7
强烈赞同 强烈反对
问题也好陈述也罢，你都应该为你的游戏选择最适合的方式，并且不要在同一份问卷中频繁地转换不同方式。
语言的使用。你必须使用那些耳熟能详的日常用语；避免使用晦涩的专业术语。如此你才能够确保你所询问的对象能够理解你的问题。很多研究员只是简单地提出问题，有时候甚至都不理解自己的问题（我敢保证很多人都这么做过），如此他们最终只能够获得一些没有用的信息。所以切记一定要直接提出那些你真正想知道的问题！
当你提供了一些问题的可选择答案时（游戏邦注：例如他们不得不从自己所拥有的控制台列表中做出选择）。你必须确保这些答案足够详尽。换句话说，你必须好好思考这些可选择答案。在此添加其它额外选项很有帮助，但是前提是不要频繁使用这种“其它”选项。
同时你还需留意不要问一些重复的问题（即使是基于不同方式）。一定要尽可能地压缩问题。避免出现任何消极语气；也就是基于肯定的语气而不是否定，如“我喜欢跳跃机制”而不是说“我不喜欢跳跃机制”。就比如说在消极语气问题中，人们便不得不逆向思考到底要选择“同意”还是“不同意”，而因此混淆了问题与自己的答案。
避免出现诱导性问题，双重问题以及含沙射影式问题。诱导性问题是指人们会被引导着而给出一个肯定的答案，如“这是一款有趣的游戏。而它的趣味性在哪？”当然了，这个例子太夸张了，但是你还是得正视这一点，因为实在有太多问卷调查中出现了这种类型的问题。所以请一定保持提问措辞的直接与中立。不要装腔作势！
双重（或多重）问题指同时影射多个问题。我将列举一个游戏外部的例子，即新西兰的公民投票进行说明：
“我们是否应该改革司法制度，更多地强调受害者的需求，提供给他们赔偿，并以最低的刑罚和较重的苦役判定任何严重的暴力犯罪？”同意/反对
我们可以看到在这个疑问句中至少出现了六个问题，包括是否需要进行改革，是否需要更加强调受害者的需求，是否应该提供赔偿，是否应该执行最低刑罚，是否应该施以苦役，是否应该面向所有严重的暴力犯罪？但是你却只能回答一次。正如我所说的，这一问题是面向整个新西兰的民众，但是却严重地破坏了他们的投票体验。
含沙射影式问题是指做出一些道德上的判断或没有根据的假设；同样以新西兰公民投票中的一个问题作为例子：
“在新西兰应该将打巴掌这一良好的管教孩子方法视为刑事犯罪吗？”同意/反对
这个问题使用了一个道德术语，即“良好”；同时它也使用了一个不够明确的表达，即“良好的管教孩子方法”（不管是什么方式），并最终造成了一种误解，即反对打巴掌的人不得不回答“同意”而赞成打巴掌的人则需要回答“反对”。
封闭式或开放式？问题可以是开放式的，也就是玩家可以随心所欲地说出自己的想法；也可以是封闭式的，即给出玩家一定的选项让他们做出选择。
开放式问题能够帮你收集到更多信息，因为它能够让玩家尽情地阐述反馈。但与此同时它也会产生局限，并且因为没有明确的方向而导致所有的问题变得更加含糊。
所以如果你选择开发式问题你就需要尽量确保拥有足够的时间阅读这些问题，以防你可能会要求回答者解释他所给出的答案。但是如果你的问卷调查是远程发放（游戏邦注：如通过电子邮件或在线发放）的，你便很难做到这一点。
而如果是封闭式问题你便可以通过不同衡量法掌控参与者的答案：
二分量表：也就是最简单的是与否的问题。通常情况下这是用于收集一些肯定与否定，正确与错误之类的信息。尽管这种方法非常直接，清晰，但是它却不能创造出多少信息。所以这种问题只能用于收集人口统计资料，或者当你必须在两个选项中做出选择时使用。
《暗黑破坏神III》中的二分选择(from gamasutra)
连续量表：也就是要求人们沿着数值的增减或基于比例增减做出选择（游戏邦注：就像你在游戏中定制角色的脸部比例）。这种方法的一大优势便是能够给出明确的衡量数值。但是通常情况下过于明确的数值也没有多大的用处（就像96.43与93.21又有多大区别呢？）
等距量表：这是我们最熟悉的一种量法类型。在此你可以基于一些相互分离的步骤而做出等级评定。
比起连续性选择，这种方法虽然会降低量表的精确度，但是我们却能够因此更好地比较结果。依我看来，1至7的量表是最合适的选择，因为这一范围的精确度也是最合理的，不会过宽。
《魔镇惊魂》中一些间等距量表(from gamasutra)
如果你选择了等距量表，你就需要进一步将其分解为以下类型：数值量表，李克特量表（游戏邦注：是一种心理反应量表，常在问卷中使用）或语义量表。
数值量表非常简单，你只要使用一些数值进行提问即可。这种方法经常用于评级中。
李克特量表是最频繁出现的一种方法，当你想要知道有多少人表示赞同或者反对时便能够使用这一量表，例如在1至7的数值范围中，1表示强烈反对而7表示强烈赞同。
语义量表常用于单纯的评级或价值判断中，例如在1至7的数值范围中，1代表最糟糕而7代表最优秀等。
你可以基于不同需要使用不同量表，但是最好不要过度混合使用；否则将会误导回答问题的群众，让他们不知道是该直接回答同意或不同意，还是需要逆向进行思考。
除此之外在等距量表中，你可以提出单级问题（以程度去区别变量），如在1至7的数值范围中，1代表不是很有趣而7代表非常有趣，或者是双极问题（即对比两个变量），如1代表非常无聊而7代表非常有趣。单级变量能够更深入地表现出同一种情感，而双极变量则给予人们更多的空间去传达他们的想法。切忌在同一份问卷中过度混合使用这些不同类型的量表。
步骤3：整合在一起
现在你已经明确了自己想知道哪些问题了，接下来你就需要将它们整合在一起。首先，思考你需要使用何种媒介，也就是你想通过电脑还是纸发放你的问卷？如果可能的话，通过电脑发放问卷更加有效，因为你不需要在之后进行额外的数据输入，但是如果从可携带性和易用性来看，纸上问卷则更胜一筹。
你应该投入更多时间去做出选择，如果你决定使用基于电脑的调查方式，你可以寻求一些专业公司的帮助（有很多专门提供问卷调查服务的公司），SurveyMonkey便是一个非常好的选择。
接下来你需要考虑问题的排列顺序。一般来讲我都会建议研究员将最简单的问题放在最前面。这样才不会让回答者感到厌烦——只要你的问卷不会过长，他们也会愿意回答你之后提出的较困难的问题。
基于一种合理的模式或者它们所涉及的内容将剩下的问题整合在一起。不要再问过大量关于武器的问题后一下子跳到boss身上，然后又转回到武器上。
确保你在问了一些问题后能够排除一些不可能的问题或相对应地衍生出一些新问题，也就是如果你问的是“你是否拥有Xbox”有/没有，而回答没有的人则不需要列举出他们所拥有的前10款Xbox游戏。
步骤4：测试
任何计划都需要与玩家进行第一次接触，所以你就需要先亲自测试你的问卷调查。如果你使用的是基于电脑的问卷调查，你应该尽可能保证你最终获得的信息是良好。然后邀请其他人（最好是那些不熟悉你的游戏的人，并且最好不是游戏设计师）去填写你的问卷调查（而此时的你不应该与他们待在同一个房间，所以不可能帮助他们回答问题），并询问他们对于该份问卷的意见。基本上来看，问卷的测试过程也是遵循用户测试，提出一些问题并且你予以改进（然后继续进行测试）的过程。我知道这是一项相对繁琐的工作，但是一旦你拥有了一份完善的问卷，你便能够在今后重复使用它（或至少利用其中一些有帮助的内容）。
优点和缺点
问卷的最大优点便在于你能够同时面向许多人提出相同的问题，也就是你可以获得一致的可量化数据，并基于这些数据去比较不同人的看法。
但是这种方法却缺少后续过程，也就是你不能够询问人们为何他们选择这一级别；这并不是一种绝对客观的方法，而如果你希望获得一些可靠的结论你便需要投入额外的努力。即当你针对于某些人进行测试时，你所提供的问卷只能帮你获得这些人的观点而已。
优点
一致性
足够量化
能够快速进行管理
可以大规模地使用
缺点
缺少后续过程
不够客观
至多只能代表接受测试者的观点
整合一份优秀的问卷需要投入不少时间
最后，我想说的是我在此所阐述的内容还不够全面，但是至少我希望它们都是有帮助的。而在之后的系列文章中，我将进一步涉及面谈，观察法，游戏参数以及生物统计学等内容。
（本文为游戏邦/gamerboom.com编译，作者：Ben Lewis-Evans ）
游戏用户研究方法种类编辑本段回目录
作者：Ben Lewis-Evans
【这是Ben Lewis-Evans关于游戏用户研究方法系列文章的第二部分，第一部分主要阐述焦点小组、启发法、问卷调查等内容，本文将讨论访谈、观察法（其中包括“自言自语”法和实境调查法）、游戏参数和生物统计学等内容。】
访谈
与问卷调查一样，它的主要目标也是搜集主观数据。但面对面的访谈意味着你可以在互动过程中搜集数据，如果执行得当，就可以获得丰富的数据。但这种方法也非常费时，很难分析和量化你最终所得到的数据。
你在访谈中所搜集到的数据质量，很大程度上取决于你自身的采访水平，以下是一些访谈技巧：
interview(from gamasutra)
采访前
这里有一些基本内容要注意。首先，要选择一个合适的环境，一般来说最好是干预因素较少并且令人舒适的环境。
另外，最好一次只采访一个人，这样可以避免受访者的看法受到他人观点的影响，但有时候也可以一次采访2个人，因为这样有助于受访者相互启发对方——这一点对讨论协作型或多人模式的游戏尤其适用。
很显然，在受访者自己所处的游戏环境中访问他们会更困难，但如果非如此不可的话，就要尽是减少周围的干扰因素（例如，有礼貌地询问他们是否可以把门关上等等）。
可以在访谈过程中录音，但并不一定要录相，最重要的是记录下访谈内容。因为你过后未必记得所有的内容，就算你很擅长作笔记，也最好能够利用录音资料回顾访谈内容。最好是持续记录，不要让中断的动作让受访者觉得你评估的是他本人而非游戏。
你得告知受访者你们会录音记录，并获得他们的同意才能执行这种操作。另外得记得向他们说明，你希望他们谈论哪些内容，并解释采访目的并非评价他们的游戏水准，而是观察他们的游戏体验，以及游戏运行表现。
你事先要拟一个采访提纲，例如写下你想提的问题，并对其进行排序。另外也要准备提出补充问题，但要注意这些问题不可带有挑衅意味。受访者可能对你的游戏存在非议，但你的表现要潇洒一点。
撰写这些问题时可以参照上篇文章提到的调查问卷的多数设计原则。换句话说，这些问题要简洁扼要，不可具有明显的诱导性，不可累赘，并且只有一个意图。另外要回避一些只能答“是”或“不是”的问题，这类问题比较合适出现在调查问卷上。
采访中
开头先说些简单的事情，这和问卷调查一样，主要是为了暖场和热身，让受访者自然过渡到接受访谈的状态。切入正题后要记住，一次只能提一个问题，确保受访者结果回答后再进入下一题。你应该掌握移情的倾听技巧，用点头或者回答“嗯”之类的动作回应对方，以示你确实在听他们说话。
如果他们离题了，也没有什么关系，因为这是访谈过程中常见的情况。但如果他们扯得太远了（例如大谈自己优秀的游戏设计想法），或者在某个问题上纠缠过久，那就得想法缓慢地让他们重回正轨。
要尽量保持中立态度。可以给受访者一点建议，以示你之前听过这些说法。不要表现出你很不耐烦的样子，不要对他们所说的内容表现出过度反应。这听起来似乎有违移情倾听技巧的做法，但是如果操作得当，就可以完美地表现出你在认真倾听而非暗示你在判断他们所说的话。
另外，在访谈结束前不妨给一点时间，让对方补充一些你没问而他们想说的事情。
采访后
整理访谈记录。注意这项工作会花掉相当一段时间。如果你执行多次访谈，要注意找到受访者所谈的共同话题。访谈的一大好处（这也是观察游戏玩法的好处）就是它可以为开发团队（甚至是管理层）带来一些富有启发的内容。
优势
*主观印象的数据来源很丰富
*可以追问其他问题
弊端
*不易量化数据
*费时
*不够客观
观察研究
观察研究（有时也称为行为观察或现场观察）就是让一些人玩你的游戏，然后进行录相以便之后进行观察，或者直接在现场观察。这可能是传统用户研究测试法中最有价值的方法，你自己亲自看用户玩游戏无疑更有说服力。因为此时你可以观察他们如何在游戏中取得进展，如何应对挑战，他们在哪受阻或失去兴趣。
你还可以注意他们的肢体动作和表情，以便探测他们的情绪。但要注意，只能记录下你所看到的内容，不要妄自揣测他们的心理活动。观察者或组织者应该保持中立和不干预态度，要是想知道玩家在哪跌倒或犯错，若非不得已的情况，就不要轻易施予援手。
如果你有一个合适的实验场地，就可以展开大规模的观察。但最好是进行一对一的观察，这样可以让玩家免受彼此干扰，而观察者也不会因测试玩家太多而分散注意力。
执行观察研究时还有一些事项需要注意，以下是我总结的一些步骤：
步骤1：设计好观察流程
要明确在哪测试，是否需要拍摄记录，要测试游戏的哪些环节。最理想的情况是全面测试，但由于现代游戏具有非线性和突发性特点，要做到这一点实非易事。另外，应该挑选游戏最具代表性的内容进行测试，并确保其核心机制足够有趣。
执行观察测试时最好是创建和使用测试脚本。测试脚本就像是电影脚本，美味糕点的食谱一样，它会安排好测试流程，包括事件的发生顺序，事件内容，甚至是供测试者使用的脚本对话。
脚本内容要尽量详细。要考虑到用户进入房间后的情形，例如是否应该向他们提供咖啡？他们要坐在哪里？是否有告诉他们卫生间和紧急出口在哪？在测试过程中你是否随时都和他们交谈，或者在特定节点才交谈？你怎样呈现游戏环节？
这听起来有点过于严谨了，也有人认为这种方法不适合展开调查研究，因为它会吓坏玩家，让他们坐立难安，觉得自己才是检测对象。这种说法不无道理，但如果你想获得更有效的数据，最好还是事先安排好测试流程。优秀的调查人员可以做到这一点，而且不会让操作过程显得太生硬。
最后要记住，如果一开始是愉快地欢迎测试用户，不停地同他们聊些无关话题，后面又发现时间透支，只能匆匆接待下一个测试者，这也并不是很好的做法。因为你后面会发现，用户评价的并非你的游戏，而是你本人。
良好的测试脚本应该包括接待参与者、管理人员，介绍流程（例如测试步骤，时间等）。记住，此时的态度可以随意和放松一点，让大家知道接受评估的是游戏而非玩家。
脚本应该清楚描述玩家需执行的任务，例如“从这一关开始玩到下一关”，重要的是明确起点和终点。因为这可以让你准确把握这一过程中所发生的特定问题。
你还应该明确自己希望看到哪类行为（可以是游戏中或游戏外的行为），这样你才会清楚应该注意哪些地方。假如许多人观察到的行为都和你所写下的观察结果一致，那么就再好不过了——双跳看起来到底应该是怎样？跳跃失败又如何呢？这种大家都认同的行为描述列表就是所谓的行为详述，它有助于减少多人一起观察并讨论结果时所产生的困扰。
或以借助一些软件解决这个问题（游戏邦注：例如Morae或Observer等软件），这类工具可以记录屏幕和参与者画面，为测试计时，还可以让你在观察过程中（或者之后查看记录视频时）进行快速的预编码。如果你无法录相或使用优质软件工具，那就要制作一个良好的行为详述表以写下观察结果（最好坐在用户看不到你的地方），这样也有助于生成有用的观察结果。
一切准备就绪之后，你应该彻底检验一下自己的安排，确保各个环节流畅运转，并拥有明确的指令。这也是出色的调查人员所应获取的观察经验。
在展开测试之前，你得招募测试者。在第一部分中我们已经讨论过这个话题，但这里还是要强调一点，最重要的是引进具有代表性的用户。如果你的游戏锁定的是5-8岁的女孩，那么找隔壁宿舍的男生来参与测试就没有什么意义了。而如果你要找孩子来测试游戏，那就有另一堆问题需要考虑（游戏邦注：例如得到家长同意，安排群组测试和单人测试，以及沟通交流等问题）。
步骤2：执行观察流程
展开观察后要做些什么？当然就是观察玩家行为。要查看这些参与测试者的反应，并记录下他们在游戏中，在游戏外的行为，以及他们所说的话。可以借助上文提到的软件完成这项工具，记录玩家的游戏过程有助于你过后重新回顾。另外，录相资料还可以作为说服他人哪些地方需要修改的证据。
要注意，不但要记录玩家的行为，还要记录他们某个操作发生的时间和情形，这种情形对诠释数据来说也很重要。
在观察过程中，要注意只能记录自己的所见所闻，例如要写“玩家看到过场动画时发笑了”而不是“玩家很开心”，因为后者的描述只是你自己对玩家心理状态的臆断。
这里看起来又有于严谨了一点，但从长期来看，你的臆断越少就越好，并且准确地描述玩家行为也有助于你同他人探讨这些问题。
除了你所看到的行为之外，你还应该记录玩家在游戏中的表现。例如，玩家通过特定步骤所费的时间，游戏中取得的分数，破坏数量和使用的弹药数量等等。如果你并非通过游戏衡量系统追踪这些变量，那就真的很有必要详细记录内容。
要注意在玩家体验游戏过程中，不要传递你的反馈意见。当你看到玩家卡在某个环节而却没找到“显而易见”的解决方法时，这确实会让你纠结，但这也正是你需要看到的结果。这意味玩家开始测试时，你就不可以再跟他们聊天，并且要礼貌地请求他们不要跟你谈天。可以让他们带一个耳机，这样可以制造一种独立空间之感，也有助于他们投入心理状态。
但这并不代表你不可以插入提问，或者真的出现问题时仍然坐视不理。无论如何，最好要记录下你感到哪里不对劲的阶段，测试结束后再问玩家他们当时的感受。
步骤3：结束测试
这时要感谢玩家参与测试，也许还可以借此让他们填写一份最终的调查问卷，甚至是对其进行简短的采访。这个简短采访可以让你借机针对自己观察到的有趣情况提出问题，以便让自己释疑解惑，得到一些有用的反馈。再次强调，一定要用非对抗性或中立的态度进行采访，不要咄咄逼人地指出他们在第三关时应该如使用喷气机脱离那个深洞。
最后，假如你进行多次测试，建议你腾出15-30分钟时间重新整理房间，在下一拨用户进来前理清头绪。长时间的观察可能让你精疲力尽，所以安排片刻休息可以让自己放松一下，也可以让你重新审视自己的观察结果。
实境调查
另一种观察研究法就是实境调查。这实际上就是一种现场堪察工作，你要在玩家的自然状态中进行观察。也就是说，你可以让他们在自己正常的游戏环境中观察他们体验游戏的习惯，并从中获得在你自己的测试过程中无法得到的结果。
在这种情况下，我之前提到的测试脚本就没有多大用处了，因为此时的关键是观察玩家如何在自己的日常生活中体验你的游戏。不过提前准备自己来访时想说的内容，自己将如何进行观察，这一点仍然很有必要。
实境调查更有助于提升现有游戏质量，你也可以通过这种方法观察玩家如何在现实生活中体验与自己的产品类似的游戏。在这种情形中，测试游戏原型或者正处于开发过程中的游戏，就没有多大意义了，因为玩家无法对这种内容形成一种游戏习惯。不过，在这种过程中观察自己的游戏如何被玩家使用（或者滥用）仍是一个不错的主意。
观察方法的优劣
观察研究法最大的优势之一就是人可以真正看到新玩家如何体验你的游戏。在我看来，这就是极有价值的信息。这种方法也有助于让你了解意想不到的情况。
不过，观察研究很显然非常耗时，如果你有闻必录的话那就更是如此。另外，成为优秀的观察者也需要投入一定精力。正如我之前所言，优秀的观察者必须保持中立态度，尽量不要干扰玩家体验游戏。另外，他们还必须清楚自己要了解哪些内容，何时补充提问等。
先进行实践，并列出你希望看到的典型行为，这有助于培养你的观察能力。然后你就会知道在展开测试前需要了解哪些内容。记住你首要观察的是他们在游戏中的行为，所以要无设想他们将执行的动作。例如，观察他们是否在同一个地方跳跃多次？或者一直墥墙？
此外，不要用自己的想法来判断你所看到的情形。不要臆断玩家心理状态，只要写下自己看到的东西即可，如果真想知道他们在想什么，过后再问他们。
优势
*可提供客观数据。你可以看到玩家真正的行为，而不是他们所说的操作。
*玩家的行为可能出乎你的意料。
*此时记录设备所搜集的信息，也许还有其他用途。
弊端
*费时，尤其是你需要详细记录内容的时候。
*要掌握足够的经验才能成为出色的观察者。
*进入测试环境可能让玩家感到不安，或者他们得知自己被监控，可能会表现出一些完全不自然的行为。
“自言自语”法（think out loud)
这是观察法的一个扩展，也就是你可以采用上述提到的观察法，然后在玩家体验游戏时询问他们当时的感受。然后你一边观察他们的行为，一边记录下这些内容。这种方法可以增加有关玩家心理状态的信息，让你了解他们对游戏的假设性看法。注意，此时你可能会发现他们所说的情况是错误的，或者并非你喜欢听到的答案，但一定要克制这种纠正他们的冲动。
这种方法可以让你了解意想不到的情况，并且这些是其他方法所无法提供的信息，至少可以让你更为了解用户在心理上与游戏的交互情况。但是，它也有一个极大的弊端，那就是让人们一边玩一边说下自己的感受，这种做法非常不自然。这可能会改变玩家实际上的游戏操作行为，影响他们的游戏乐趣。
与此同时，你可能还会发现，有些人一开口就没完了，而有些人却几乎难以启齿，结果你就会发现这种方法更多地是反映人们之间的差别而非游戏本身的问题。
优势
*允许你搜集客观（行为）数据的同时，获知玩家内心的想法和感受。
*可能获得意想不到的结果。
弊端
*耗时。
*不自然。
*人们所说的仍是主观信息。
玩法参数
还有一个观察法补充内容就是玩法参数。它实际上就是游戏本身搜集的统计资料，可以让你明白玩家所执行的操作。例如，《光晕》中的统计系统、《极品飞车》中的日志系统。
这些玩法参数可以提供有关游戏玩法的可靠、主观而统计性的事实。要在游戏中植入这些数据搜集系统，如果采用这种做法，你就会发现自己手头上掌握了大量数据来源。想知道在地图上哪种武器最常让玩家毙命？玩家使用哪些升级道具？他们最常丧命的地方？这些都可以从玩法参数中找到答案。优秀的参数触发器可以成为一种十分有效的辅助观察手段，可以减轻你记录玩家表现细节的担子。
game metric(from gamasutra)
（参数的力量：上图左是玩家在《光晕：致远星》中以某种武器杀敌时所站的地点，右图则是该武器的杀敌数。你只具备一点基本的FPS游戏知识，也可以大概推测出这是哪类武器，并了解这个关卡的提升空间。）
不过搜集玩法参数也非常费时，并且你需要一定规模的玩家参与测试才能有效搜集数据。此外，这种参数缺乏与情感，以及玩家个人情况有关的主观信息——它只是一堆数据而已。
另外，如果你无法仔细分析自己搜集到的数据，你就会面临数据超载的情况。例如，在《光晕：致远星》中载入任何一个热图，打开所有武器造成的死亡数量，将其与所有武器的杀敌数进行比较。这可能会让你了解到在该玩家区域的杀敌数与死亡数的分布情况，但却无助于分析玩家体验的细节。
优势
*可以远程和分离地收集客观数据。
*可以查看趋势，并为其他观察法提供数据支持。
*可以在不影响玩家的情况下持续搜集数据。
弊端
*费时。
*缺乏主观反馈。
*需要大量样本才能有效搜集数据。
*同时也会让你获得过多数据。
生物计量法
我之前已经探讨过这个话题，所以不在本文赘述详细内容。生物计量法是指通过记录玩家肢体信号（例如通过EEG获取脑波，通过眼球追踪技术查看玩家寻找的内容）从而获知玩家内部的生理状态。它和自言自语法以及玩法参数一样，可以做为一种观察测试的补充。
与观察和玩法参数一样，生物计量法的妙用在于它可以提供一些关于玩家状况的主观数据。它可以在不影响和打断玩家的情况下，持续搜集与玩家情感和兴奋感有关的数据。
但这种搜集方法目前仍十分具有干扰性，因为你得将用户与测量仪器连接起来，并且这些设备价格十分昂贵，而且你还得学习这些设备的使用方法，并且学会分析它们所搜集的数据。
此外，这种方法也可能让你搜集到错误的数据信号，并且行业对于这种方法所搜集到的数据可靠性尚未达到一致意见。例如，你的游戏中有些环节可能导致玩家心率提高，这可能意味着游戏令人兴奋，也可能意味着它让人感到恐怖或者不快，或者玩家实际上觉得很无聊，已经神游到其他令人兴奋的事情上了，甚至也有可能是因为玩家当时只是做了个深呼吸（这也可能提高人的心率）。
以上问题意味着你需要将生物计量数据与其他数据来源（游戏邦注：包括观察、采访、调查问卷）相结合，这样才能弄懂你所搜集到的生理信号究竟代表什么情况。这也导致有些人怀疑这个方法的可用性，他们认为这非但不能提供有效的补充信息，还可能产生许多额外工作量。
不过生物计量法的倡导者认为，生物计量数据的优势在于，它可以让你获知玩家自己不知道的情感和生理反应。最后要注意，它也只是一种补充工具，要不要使用这种方法要取决于你的判断。
优势
*提供客观的量化数据。
*可在不干扰玩家的情况下持续搜集数据。
弊端
*具有干扰性。
*耗时耗财。
*因特异性、推理和有效性等问题而难以诠译数据信息。
总结
我这两篇文章只能算是粗略的引言，所述内容也并不全面。如果你有兴趣深入研究这个话题，我推荐阅读Valve成员Mike Ambinder的有关著作，以及Mike同Bungie成员John Hopson在这一问题上的讨论内容。
最后要说的是，上述方法只是辅助开发游戏的手段，它们也许能搜集到大量有趣的数据，但这些数据并不能主导设计。要用你的创意想法引导游戏开发，要让游戏用户研究搜集到的数据帮助开发游戏，而非约束你的创造力。（本文为游戏邦/gamerboom.com编译，拒绝任何不保留版权的转载，如需转载请联系：游戏邦）
Finding Out What They Think: A Rough Primer To User Research, Part 2
by Ben Lewis-Evans
[The following is the second of two articles by college professor and researcher Ben Lewis-Evans on games user research methodology (see Part 1, which covered focus groups, heuristics, and questionnaires, as well as giving a grounding in the topic of user research in general. In this article, Lewis-Evans covers interviews, observational methods (including think out loud and contextual inquiry), game metrics, and biometrics.]
Interviews
Much like a questionnaire — a topic covered in the last installment — an interview is for collecting subjective data. However, the face-to-face nature of an interview means that you can be more interactive in your data collection, which if done correctly, can lead to very rich data. However, it is also obviously quite time-consuming, and it is harder to analyze and quantify the data you get at the end.
The quality of what you get out of an interview will also depend greatly on your own skill as an interviewer, so here are some tips.
The Setup
There are some basic things here. First, choose a good setting. Generally speaking, it should be comfortable, with as few distractions as possible.
Also, plan to generally only interview one (or perhaps two) people at a time. One is usually best, so as to avoid one person’s opinion dominating, but sometimes two people can have a nice dynamic and prompt each other — particularly if discussing a cooperative or multiplayer game.
Obviously, if you are going to be interviewing people in their own play environment, this gets harder, but still do you best to make sure extra distractions are not around (e.g. ask politely if the door can be closed, etc.).
Also it is useful to set up a way to record the interview; this doesn’t have to be video, but it is important that you at least record what is said. You will not remember everything, and even if you take great notes it is a good idea to have the recording to refer back to. This is also preferable to making notes constantly, as this tends to distract the person you are interviewing and can create a feeling that they are being assessed personally — rather than the game.
You will have to alert the person you are interviewing to the recording and get their permission. You will also have to explain to them what you are interested in asking them about, and also as with all of these methods make it clear that this interview is not about evaluating their performance, but to look at their experience of, and the performance of, the game.
You should also make an interview script; i.e. write down the questions you want to ask, and the order in which you want to ask them. However you should also be prepared to ask follow-up questions, although take care and don’t be confrontational when you do. It may be your game they are badmouthing, but you need to play it cool.
When coming up with these questions many of the same rules creating questionnaires apply (see part 1 of this series). In other words, try to be clear and precise with your questions, and check they are not leading, loaded, and only have one meaning. Also take care to avoid questions that can be answered by a simple yes or no; if you are after that kind of information use a questionnaire instead.
During the Interview
Start off with some easy stuff. Much like with a questionnaire, this warms them up, and will get them in the mood to talk to you. Once you get going also take care to only ask one question at a time, and usually only move on when you are sure the person you are talking to is finished. You should practice listening empathically and encourage responding by doing things like nodding your head and saying “mm-hmm” to acknowledge you are listening to them.
It is fine if they go off on a tangent; that is part of interviewing. However, if they go too far off (say, talking too much about a great design idea they have) or spend too long, be prepared to gently nudge them back into talking about things you are interested in.
Also, try to be as neutral as possible. One bit of advice given to interviewers is to act as if you have heard it all before. Not as in you are bored, but as in that you are not shocked or overly reactive to what they are saying. This may sound like it is against the idea of empathic listening, but if done correctly, it is perfectly possible to signal that you are listening carefully without also implying you are judging.
Finally, it can be useful to finish up the interview with a short period of time where your interviewees can add anything that you may not have asked about.
After the Interview
Make a transcript. Be aware this will take some time! Then, if you conducted multiple interviews, look for common themes and threads in what people are saying. One good thing about interviews (which is also a good thing about gameplay observation and think out loud methods) is that they can generate great sound bites or quotes that can be quite convincing when given to others on the development (or even management) team.
Pros
•Rich data source for subjective impressions
•Can ask follow up questions
Cons
•Not easy to quantify
•Time-consuming
•Not objective
Observational Studies
Observational studies (also sometimes called Behavioral Observation or Ethnographic Observation) are where you get some people to play your game and either record them doing so to watch later, or observe them directly. This technique is probably the most valuable of the traditional user research testing methods because nothing beats actually watching someone play your game. You can see how they progress, how they deal with challenges, and where they appear to become stuck or unmotivated.
You can also attempt to pay attention to their body posture and expression in order to try and gauge their emotions. Be careful, however, and only write down what you actually see — don’t try and infer any inner thought processes. Also the observer or facilitator should be as neutral and hands-off as possible. You want to see where people get stuck or make mistakes, so don’t help them unless you really, really have to.
Observation can be done in large groups if you have the correct lab space. But it is probably better done on a one-on-one basis so that players do not distract each other and the observer is also not distracted.
Again, there are several things to remember when setting up an observational study, and I will try and go through some of the steps below:
Step 1 – Designing the Observation Session
Work out where you will be testing, if you will be recording the session or not, and what part or parts of the game you will be testing. It would be ideal if you could test exhaustively, however with modern non-linear and emergent games this is obviously difficult. Still, you should plan to test at the very least large representative chunks of your game, and certainly make sure that the core mechanics are fun.
One useful thing to do when running an observational type test is to create and use test scripts. Test scripts, like a movie script, or the recipe for a delicious cake, lay out how the test will go and typically include the order of events, what the events are, and even scripted dialog for the tester to use.
Be specific with this. What happens when the user comes into the room? Do you offer them coffee? Where do they sit? Do you tell them where the bathroom and emergency exits are? Do you talk to them at any time during the test, or only at set points? What build do you load up? In what order to you present the game portions you are looking at?
This may sound overly scientific, and in fact some argue that it is a bad way to run research, as it scares the player and gets them in a mindset that makes them feel tested. This is a valid point. At the same time, however, if you want the best data, it is good to have at least some consistency, and certainly you need to have some plan for the session. Good researchers can do this and still not come across as a robot.
Ultimately, it is no good if at the start of the day you welcome a user happily, chat about their ride over to the lab as you offer them a coffee, and then later in the day you just rush the next user into the room and try and get things done as fast as possible. Then you may find that it is not your game they are rating, but you.
A good test script should include welcoming the participants, administrative stuff, and some introduction to the procedure (what will happen, how long it will take, etc.) Remember, again, at this point it is good to be casual and relaxed to make it clear that the game is being evaluated, not the player.
The script should also clearly describe the tasks they are going to be doing during the observation session, this can be as simple as “start the level and then get to the next level” but it is important that they have clear defined start and end points. This is because you want to know exactly when in this process certain issues or behaviors occurred.
You should also think about what kind of behaviors (in the game, and perhaps outside of it, depending on the game) that you expect to see, just so you know what to look out for. It can be helpful if many people are observing to agree on a clear sentence (or two) that outlines the behaviors you are observing — so what exactly does a double jump look like? What about a failed jump? A list of such agreed-upon descriptions of behaviors is called an ethogram, and it helps to cut down on confusion when discussing what was observed with others.
There are quite a few software packages out there aimed at helping out at this point (for example Morae or Observer, although there are many others) which can handle the recording of the screen and the participant, the timing of the test and also allow you to carry out quick predefined coding as you observe (or when you go back later to watch recorded video). This said, if you can’t afford to record and use fancy software then having a nice well-defined ethogram and somewhere to write down observations (sit somewhere the user can’t see you — even if that is sitting behind them) can still produce useful results.
Once you have everything ready to go, you should test out the setup you have to make sure everything runs smoothly and the instructions are clear. This is also important as the only way to become a good observer is get experience observing.
Finally before you can start you need to look into recruiting people to test. I covered this in part 1, but just to be clear, the most important thing at this point is that you want representative users. It’s no good testing it with the dudebro in the dorm next-door if your target market is young girls five to eight years of age. And if you are going to be testing with children you are opening up whole other box of issues (parental permission, testing in groups vs. alone, communication, and so on).
Step 2 – Running the Observation Sessions
What do you do once the session is running? Well, observe, of course. Watch the participants and take notes on what they do in the game, in real life, and what they say. As mentioned above, there is software that can help you do this, and certainly recording the play session will help you later if you can go back and review it. Plus the recordings can be great material to show other folks who might need convincing that something needs to change.
Be careful to note not only what the player is doing, but also the time at which they do it, and the situation in which it occurs. This context is important for interpreting the data.
Also when observing, write down only what you see and hear. So, write: “the player is laughing at the cutscene and smiling” not “the player is happy” as this latter description makes assumptions about the user’s internal states.
Again this may seems overly scientific, but in the long run the fewer assumptions you make, the better, and it helps when communicating with others what you found if you stick to clear descriptions of behavior.
In addition to the behaviors you see, you should also record down performance measures from in the game. In other words, you should note things like the time taken to clear a particular stage, the in-game score, the amount of damage taken and ammo used, and so on. This is especially important if you are not tracking these variables through a gameplay metrics system.
Finally, I should note that during this time it is good if the player is isolated from others (including you) and can concentrate on the game. This can be as advanced as an observation room at Microsoft or just you simply sitting in the same room out of the player’s line of sight.
The important thing is that you let the player play without feedback from you. This can be frustrating when you see someone getting stuck in a place where the solution is “obvious”, but this is exactly the type of thing you want to see. This means that you shouldn’t really talk to the player once the session has begun, and you should politely ask them not to talk to you either. Having them wear headphones can help as it adds to the sense of a self-contained environment, and can help put the player in the right frame of mind.
This doesn’t mean you can’t step in and ask a question, or help if things are going really wrong. However, generally speaking, it is better to note down periods where you are not really sure what was going on and then ask the player about them later, after the test is done.
Step 3 – End of the Session
This part is pretty straight forward. Thank the player for taking part, and perhaps take the opportunity to give them a final questionnaire or even conduct a quick interview with them.
This interview/debrief gives you the chance to ask about any interesting observations you made and try and get some feedback on what was going on. Again, be careful to do this in a non-confrontational and neutral manner; don’t jump down their throat about how they should have just used the jetpack to get over that hole they kept falling in during level 3.
Finally, if you are running multiple sessions, it is usually advisable to give yourself about 15 to 30 minutes at this point to reset the room, and gather your thoughts before the next player/group of players. Observing for long periods can be very draining, so these little breaks can help recharge your batteries a little, and also allow you to think about and perhaps review the observations you have made so far.
An Aside: Contextual Inquiry
Another type of observation study is a contextual inquiry. This is essentially fieldwork where you play David Attenborough and go out and observe your players in their natural habitat. The idea being that you can observe them in their normal play setting going about their normal routines when using your game, and therefore get insights that may not be available in your own test situation.
In this case the test scripts I mentioned above are somewhat less useful, as the point here is to see how players use your game in their normal routine. However being prepared in terms of what you will say when you show up and how you will carry out the observation is still important.
Contextual inquiries also tend to be more useful for improving existing games, or observing similar games to yours to learn about how people play them in the real world. As such, testing prototypes or games in development in this fashion can be less useful, as players haven’t had an opportunity to develop a routine with them. Still, it is certainly nice to see how your game will be used (and abused) in the wild.
Pros and Cons of Observational Methods
That is all I will cover as far as observations go (with the exception of think out loud, which I will discuss in a bit). The biggest advantage of observation studies is that you really get to see what new players do when playing your game. This is invaluable and makes observation studies the best methodology, in my opinion. This method can also give you unexpected insight, and reveal issues that you may have never seen without it.
However, observations are obviously quite time-consuming, especially if you are recording everything. Also, it can take quite some work to be a good observer. As I already said, a good observer has to be as neutral and non-intrusive as possible. But also they have to know what to look for, and when to ask follow-up questions.
What typically helps is first practicing, and making a list of typical behaviors you would expect to see. Then you know what to look for before you go in to a real session. Remember you want to primarily be looking for their behavior in the game, so think about the types of things they will be doing. So for example do you notice them jumping repeatedly in one place? Or running into walls?
Also, avoid coloring what you see with your own expectations. Don’t try to infer internal states; just write down what you observe, and then if you want to know what was going on in their head, ask them later.
Pros
•Provides you with objective data. You are seeing what players actually do, not what they say they would do/have done
•Players may surprise you
•When recording equipment is used, the material collected can be used for other purposes
Cons
•Time-consuming, especially if you record and go back through everything carefully
•It takes experience to be a good observer
•Going into a test environment can make people nervous or at least aware that they are being monitored, and therefore they may not produce completely natural behavior
Think Out Loud Protocol
An expansion of the observation method, this is where you carry out observations as detailed above, but ask the players to talk about what they are thinking as they go along. You then record this while observing what they are doing. This adds information about the internal states of the player, and gives you an idea of what assumptions they are making based on the game. Again, you may hear things that are wrong, or that you don’t like, but resist the urge to correct the player.
This method can lead to unexpected insights that wouldn’t be gained otherwise, or at the very least gives you a little more insight into how people are mentally interacting with your game. However, it does have one big downside, and that is that it is quite unnatural to talk about what you are doing and feeling as you do it. This might change how people will actually play the game and affect how much or little fun they are having.
Related to this, you may find that some people talk non-stop, whereas others hardly say a word, a finding that may reflect more the differences in people than in your game.
Pros
•Allows for the collection of objective (behavioral) data while also getting access what players are thinking and feeling
•Can result in unexpected insights
Cons
•Time-consuming
•Unnatural
•What people say is still subjective
Gameplay Metrics
Similar to, and a great addition to, observational methods are gameplay metrics. This is basically is statistics generated by the game itself. which tell you about what the players did. For example think of the statistics systems that Bungie and 343 Studios have for Halo, or the Autolog system in Need for Speed.
Such gameplay metrics give you hard, objective, statistical facts about how the game plays. The collection of this data has to be built into the game, but once it is, you have a very large data source at your fingertips. Want to know what weapons take down players most on a map? What power-ups they use? Where they tend to die? Gameplay metrics make that possible. Good metric triggers and hooks can also really compliment observations as they take some of the load off you in terms of recording the in-game details about the player’s performance.
The power of metrics: on the left is where people are standing when they make kills with a weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of FPS games, you can still probably work exactly what kind of weapon this is and where the elevated and the open spaces are in the level.
However, metrics can be time-consuming to collect and you need a good number of players to really take advantage of them. Plus metrics lack subjective information about emotion and what is going on with the players — it is just a mass of data.
Also, you can end up with data overload if you don’t analyze what you collect carefully. For example, load up any heat map in Halo Reach and turn on deaths by all weapons, and compare it to kills by all weapons. The result will generally give you a little information in that the kills and deaths seem pretty well spread out over the player area, but is not particularly useful for working out finer details of the player experience.
Pros
•Objective data that can be collected remotely and discretely
•Can be used to see trends and provide data that supports other methods
•Allows for continuous data recording without interrupting the player
Cons
•Time-consuming
•No subjective feedback
•Needs large sample sizes to get meaningful data
•However, at the same time, you can get too much data
Biometrics
I will finish up by covering biometrics. I have already done a primer on this topic, so I am not going to go into too much detail here. Biometrics are basically when you record signals from the body, such as brain waves via EEG, or where people are looking via eye tracking, to try and give us a clue as to your player’s internal body states. In this way they are again like think out loud is, and like metrics can be: an add-on to observation-based testing.
Much like observation and gameplay metrics, biometric measures can be useful because they give you objective data on what is going on. They also allow for data on emotions and excitement to be collected continuously during play without having to stop and ask questions about how people are feeling.
However, the ways you can collect biometrics are currently quite invasive in that you have to wire people up, and they cost quite a bit of money and time to use, not to mention you have to be trained in how to use them — as well as how to analyze the data that you get.
Also they have problems in that you can get artifacts or fake signals in your data, and that there is no real solid agreement on what you are measuring actually means. For example, heart rate may go up in a certain part of your game, which could mean it is exciting, but it could also mean it is scary and unpleasant, or that your player is so bored they are now thinking about last night in bed with their partner, or even just that the person you are recording took a deep breath (which also increases your heart rate).
The above issues mean that you have to combine biometric data with other sources of data (observations, interviews, questionnaires) in order to work out what is going on with the physiological signals you are collecting. This has led some people to doubt its usefulness, arguing that it doesn’t add anything that you couldn’t already get but does add a bunch of complications and extra work (see an excellent discussion on this point, and on games user research in general, by Bungie’s John Hopson and Valve’s Mike Ambinder here).
Proponents of biometrics, on the other hand, suggest that the strength of biometric data is that it gives you access to emotions and reactions that the players may not know they are having, and therefore helps you look at other data sources in a way you haven’t before. Ultimately, however, it is just another tool and you will have to judge if it is appropriate for you or not.
Pros
•Gives objective quantifiable data
•Allows for continuous data recording without interrupting the player
Cons
•Invasive
•Costs a lot of time and money
•Problems with specificity, artifacts, inference and validity can make it difficult to interpret
Final Words
As I stated in the introduction these two articles are intended as rough primers, and are in no way comprehensive. I am sure that others will disagree with what I say, and perhaps others will be happy to add more or provide material of their own. However, I do hope though that this document can be of some use.
If you are interested in further reading on this topic then I can recommend checking out this excellent document from Mike Ambinder at Valve and watching this discussion between Mike and John Hopson from Bungie. As mentioned in the last article, you may also consider checking out the IGDA Special Interest Group for Games User Research on LinkedIn.
Furthermore a wiki-based games user research primer is in the works. This primer will arise out of a workshop at the CHI 2012 conference and is planned to be an evolving wiki based site that can provide the community with up to date information on game research methodologies. So watch that space.
Finally, please remember that the above methods are here you help you develop your games. The data they give can be rich and interesting, but it should not be domineering. In the end, it is your creative vision that guides the game; the data that can be collected via games user research is there to help you, NOT limit you.（source:gamasutra）
Finding Out What They Think: A Rough Primer To User Research, Part 1
by Ben Lewis-Evans
This article, and its forthcoming followup, is intended to give a rough idea to developers of several different methods that can be used in games user research.
However, many, many books have been written on research methodology and I cannot cover everything. Therefore these two articles cannot be taken as completely comprehensive.
In the first of the articles I will be covering a few general points about Games User Research and then discussing three methods, focus groups, heuristic evaluation and questionnaires in some detail.
What is Games User Research?
So, before getting really started, what is games user research in the first place? Well, let’s start by comparing it with Quality Assurance (QA). QA is a well-established part of software development, and is often carried out by professionals within a development team. These folks are, generally speaking, aimed at finding bugs and making sure the game runs smoothly.
However, because those working on a project usually carry out QA, it means they have an investment in it, and are familiar with it. This can cause problems when it comes to evaluating the usability and experience of a game. What seems obvious and fun to someone that has been working on a project may be completely alien and frustrating to a new user.
This is where games user research methods come in, a field that is all about the user and their experience of the game — in particular, the big question, “is it fun?”
To really (over) simplify things, it could be said that QA and test is about the software, and how it functions when dealing with users, and game user research is about the user and how they function when dealing with the software. Notice I say just it is about how the user functions when dealing with the software, and that it is NOT about testing the user (more on that later).
So how is this done? And what is fun, anyway? Well, there has been plenty written on fun already, so let us just say that fun has quite a few dimensions. Fun can be something that is easy to use, but it can also come along with a struggle and a challenge. It can arise from an engaging experience, a compelling experience, or a relaxing one.
All this means fun is a subjective variable, which changes from person to person, and situation to situation. However, due to the emotional component of fun, what can be said is that if your players are having fun, then it is likely they can tell you about it — if only you know how to ask.
So How Do We Ask?
Okay — I’ll get to that. But before I get into the fine details, please bear with me as I outline some general principles to keep in mind:
Get the right players
Whatever methods you choose, make sure you get the right sample. For most methods, this means getting representative users, aka the people that you expect will play your game.
If you have the time, perhaps there is some advantage to getting as wide and large a user group as possible (if you really think that everyone will want to play your game), but given that you are likely to be constrained by time (and money) it is generally best to concentrate on getting as close a sample of the target type of player you are after as possible.
The game is being tested, NOT the user
Secondly whenever doing this type of research make sure it is clear to the user that they are not being tested. The research is about improving the game, not the user, so the user shouldn’t be made to feel inadequate if they can or cannot do something. In principle it is all valuable information.
This can be hard to do sometimes, as you are the ones designing the game, and it can be uncomfortable to hear others criticising it. But try your best to not be defensive or judgemental (i.e. avoid thinking the problem exists between the keyboard and chair.)
What do you want to know?
Whenever doing research, you should be clear about what you want to know. You will be playing the game yourselves as you work on it, and you should have design documents, so you know how things are supposed to work. So don’t just plonk people down with the game and come up with questions on the fly. Work out what areas you think might be problems and know what you want to ask about before it’s time to test.
Please note: I am not saying you go out there with preconceptions and already “know” the answers you want; rather I am just making the point that you should be at prepared and know what you’re looking for. Otherwise, you get a mass of data that may not be of any use to you at all. At the same time, be open to surprises. You never know what might pop up.
Test early and test often
The next point is considered one of the most important by user researchers, and that is to test as early as you feel it is possible to do so. This can be difficult, as it feels bad letting your baby out into the hands of users before it is 100 percent compete. But really, the sooner you can test the better — test with paper prototypes, for instance!
The primary reason for this is that it is much easier to change the game if you find an issue early in development rather than late. Once you have made the changes, you should test again. That said, it is also a good idea to make sure that the product isn’t too buggy; you want to test the experience of playing the game, not the experience of crashing due to bugs.
One extreme example of this approach is the Rapid Iterative Testing and Evaluation (RITE) method, developed by games user researchers at Microsoft. In this method a test is run (usually via behavioural observation — which will be covered in the second article in this series), and changes are made to the game as soon as problems or issues are detected, before the next participant arrives. This can occur perhaps even after just one user has been tested.
Listen to and act on problems, but not necessarily the solutions
When dealing with your users, you should be open, and listen to the problems they are raising. You can also listen to the solutions they give to those problems — but they are likely to be less useful to you. You are the game developers, and you know what is possible with the technology, time, and resources you have. The users won’t. So observe, do the research, and treat it seriously when it reveals issues, but take suggestions from users as to how to solve the issues with a grain of salt.
Games user research is just another source of data
Often when articles such as this appear online, there is much gashing of teeth and angst about taking the art out of game design and instituting “design by committee”. I can understand this worry; however, much like QA, user research is just another tool to improve your game. It shouldn’t dominate your design or suppress your artistic talent; rather, if done correctly, it should augment your talent and give you new insight.
Time for the Crunchy Stuff
Are you still with me? Great. Here, then, are the research methods that I will cover, in varying degrees of depth, in the rest of this document: focus groups, heuristic evaluation, and questionnaires/surveys.
Focus Groups
This method can be something of a dirty word (and is definitely part of that whole “design by committee” problem). So let us deal with it first and get it out of the way.
You are probably familiar with focus groups, even if you’ve never seen them used in person. Basically this is where you get a bunch of people, have them play your game, and then put them in a room to talk about it. They can be free to talk about what they like and what they didn’t like, but you also have a facilitator in the room who can ask specific questions of interest.
This process can also be used quite early on in development, where instead of getting people to play the game, you give a presentation or talk about ideas for the game and get feedback on that.
The advantages of focus groups are that they involve a lot of people, so you can get more feedback. They can also be somewhat efficient, as everyone is together in one place, and the facilitator can ask follow-up questions. So if someone mentions they like or dislike something in particular, you can gain a bit more detail on why.
Although at the same time if you are not careful, focus groups can get away from you, and end up wasting far too much time. To avoid this last problem you really need a good facilitator to run a focus group. The facilitator has to be strong enough to guide the conversation to areas that are useful, but not dominate the discussion.
Probably the biggest risk with a focus group, and one of the reasons why focus groups are not often used, is that just one or two members of the focus group can dominate the discussion. Due to group pressure, you may not hear from other people who have valuable insight into the game. Focus groups also have a tendency to get more into discussing solutions for issues rather than just the issues themselves. This is not what you want.
Finally, focus groups are a subjective method, and all you have to go on is what people say — and as much as we like to judge people on the “attitudes” they hold, and think that they predict behaviour (they usually don’t), we all know that what people say is not always what they actually do.
Pros
More people can mean more feedback (although see cons…)
Gets everyone together in one place
Allows for follow-up questions
Can be useful when discussing concepts
Cons
A good facilitator is required
Strong voices may take over and reduce feedback overall
Too many solutions, not enough issues
What people say is not always (or even often) what they do
Heuristic Evaluation
Heuristic evaluation is where you get an expert (or experts) in games user research, get them to play your game, and then they evaluate it on a set of criteria (heuristics). Kind of like a scientific game review… Kind of.
Basically, to do this, the expert(s) will use a list of heuristics, which are basically rules or mental models, and give you feedback on whether your game fits these heuristics, and where problems might come up. These heuristics can vary, but here is a selection of some possible heuristics listed in a 2009 article [PDF link] by Christina Koeffel and colleagues:
Are clear goals provided?
Are the player rewards meaningful?
Does the player feel in control?
Is the game balanced?
Is the first playthrough and first impression good?
Is there a good story?
Does the game continue to progress well?
Is the game consistent and responsive?
Is it clear why a player failed?
Are there variable difficulty levels?
Are the game and the outcome fair?
Is the game replayable?
Is the AI visible, consistent, yet somewhat unpredictable?
Is the game too frustrating?
Is the learning curve too steep or too long?
Emotional impact?
Not too much boring repetition?
Can players recognize important elements on screen?
The article itself lists over 29 of these heuristics, and goes into much more detail than I have provided here, so I recommend reading it if you have the time.
The good thing about heuristics, in my opinion, is that even if you aren’t an expert, they can provide a list of things for you to think of when you look at your game. For instance, does it provide enough feedback to players that their actions are affecting the world? Does it force players to hold the controller in an awkward fashion? And so on. Again, this may seem like common sense stuff, but it really is amazing how often so-called “common sense” is not common at all.
Now, one obvious advantage to heuristic evaluation is that you only need to use a small number of experts (just one in some cases), and being experts they know what they are talking about. This also leads to the problem that you do need experts, and where do you find those? Plus have you found the right one(s)? The types of heuristics used can vary by expert, and of course should fit your type of game — so this is obviously important.
Also sometimes experts can be a bit too expert and miss stuff that might be a problem for novices. This is because as we become more experienced at something we no longer need to consciously consider everything that we perceive and do, whereas someone who is still learning is still thinking about what they are doing all the time. This is why generally if you want to see something done well you watch an expert, but if you want to learn how to do something it is often best to ask a novice.
Pros
Smaller number of people needed
Relatively fast turn around
Experts are expert
Cons
Where do you find experts?
Did you find the right one?
Experts can be too expert
Questionnaires & Surveys
I am sure you know what a questionnaire is, but do you know how to design and use one correctly? Mountains of books have been written on this, but hopefully I can make at least some important points clear.
First of all, when do you use questionnaires? Well, they are usually used to evaluate subjective views about your game, particularly value judgements. This may vary from specifically asking players about their favorite weapon to open-ended questions asking for general comments on the experience.
Questionnaires can be given to players during the game, which means the experience is fresh, but this risks interrupting the flow of the game (if possible, find natural down times to ask). They can also be used after a gameplay session is over. The big advantage of questionnaires is that they can be given to many people, and as such you can end up with lots of nice data to examine (in theory).
Before I go into the detail of constructing your own questionnaire, there are some pre-existing questionnaires out there aimed at evaluating the fun of gameplay experiences.
Examples of such questionnaire are the Game Experience Questionnaire (GEQ) for examining gameplay experiences and the affect grid [PDF link] or the manikin system [PDF link] for rating emotions. These pre-existing questionnaires can be great, because they are usually well-written and reliable. However, they also have a tendency to be more academic in nature, and care should also be taken that they fit your game, so modify them if necessary.
So how do you go about designing your own questionnaire? Here are four steps that I hope can help you.
Step 1. Work out what you want to know
As stated already, all of these methods require that you know what information you are after. But this is extra important when it comes to questionnaire design. You usually don’t get any chance to follow up on people’s answers, so you want them to be as clear as possible, and for the information you gain to be what you are after (and neither too little nor too much!)
So, brainstorm, make lists, do whatever is best for you in terms of getting down what you want to know. Then cut it down to only what you really, really need to know. Be focused!
A mistake that novice researchers often make is to ask for everything simply because the opportunity is there. However what this results in is a mess of data that will take forever to be analyzed, and may not produce a meaningful result. You don’t really want your questionnaire taking more than 15 to 20 minutes to answer (this is not a target, by the way, but a maximum).
Step 2. Design the content
The design part is the meat of the process and can be further broken down into a few things to consider.
Questions or statements? Do you want people to answer questions, or rate statements? This is pretty straightforward but still important. Basically, questions are good for gaining information, for example:
How challenging was the Horrible Bog Beast?
1 2 3 4 5 6 7
Very easy Very Hard
Questions can also be worded in the form of an instruction. For example, “Rank the levels you just played in order from 1-6, where 1 is the one you enjoyed the most, and 6 is the one you enjoyed the least.” It’s effectively just like asking a question about the ratings for each level individually.
On the other hand, getting people to rate statements is more typically used to assess value judgments or agreement with ideas, so an example would be something like:
The Horrible Bog Beast is an interesting enemy to encounter.
1 2 3 4 5 6 7
Strongly Agree Strongly Disagree
Either questions or statements are fine; however, use them where they are best, and don’t switch between the two types too often.
Language use. It is incredibly important that you use clear, everyday language. Avoid jargon. This is vital because you want to be sure that the people you are testing understand what you are asking. Many people will just answer anyway, even if they don’t understand a question (be honest — I am sure you have done it) and the data you get may not be useful at all. So be blunt, be direct, and ask for exactly what you want to know.
When you offer alternative answers to your questions (for example if they have to select from a list of consoles they own), these should be relatively exhaustive. So in other words there shouldn’t be any alternatives that you haven’t thought of. Adding an “other” where additional options can be filled in can help here, but it is best if this “other” option isn’t used too often.
You should also be careful not to ask what is essentially the same question, but in a different way. Remember the goal is to keep the number of questions down. Also avoid asking questions that are phrased negatively. So “I like the jumping mechanic” and a gradient from agree to disagree rather than “I don’t like the jumping mechanic”. In the case of negatively-phrased questions, people filling your questionnaire have to select “agree” to disagree, and “disagree” to agree. It seems silly, but this can confuse people.
Also avoid leading, double barreled, and loaded questions. A leading question is one where the person filling it in is lead or biased to give a certain answer, e.g. “This game was fun. How fun was it?” This example is, of course, over the top, but you would be surprised how often leading questions can slip into questionnaires, so just try and keep your wording direct, and neutral. Don’t assume. Ask!
Double (or multiple) barreled questions ask about more than one thing at a time. Here’s an example from outside the world of games — from a national referendum held in New Zealand:
“Should there be a reform of our justice system placing greater emphasis on the needs of victims, providing restitution and compensation for them and imposing minimum sentences and hard labour for all serious violent offences?” YES/NO
As you can see, there are at least six questions here; should there be a reform, should it place greater emphasis on the needs of victims, should it provide restitution and compensation, should it impose minimum sentences, should it impose hard labour, and should it be for all serious violent offences? But you only get to say yes or no once. As I say, this question was put to the whole of New Zealand, and really ruined my first ever experience of voting.
Loaded questions are ones that make moral judgments or make assumptions that are unfounded; an example of this type of question also comes from a New Zealand referendum where it was asked:
“Should a smack as part of good parental correction be a criminal offence in New Zealand?” YES/NO
This question is loaded in that it uses a moral term like “good”. It is also ill-defined, as it uses the term “good parental correction” (whatever that is), and finally it is misleading in that if you are anti-smacking you have to say yes (agree) and if you are pro-smacking you have to say no (disagree).
Closed or open? Questions can be either open, as in the player can say whatever they like, or closed, where there are only a few limited options to select.
Open questions allow for richer data to be collected, as they let players give as much feedback as they want. However they can also give as little as they want, and often without direction, the answers may be vague.
One way to get around this if you do use open questions is to make sure you have time to read them before the end of the session, in case you want to ask for clarification on any points people bring up. However this is difficult if the questionnaire is being given remotely, such as when it is being used via email or online.
Closed questions on the other hand let you have much tighter control on the answers your participants give, and come in several flavors (scales):
Dichotomous: This is for simple yes/no questions. So usually it used to collect stuff like yes/no, true/false. It is nice, direct, and precise. However, it is also not very data rich. So generally speaking such questions are used just to collect demographics, or when you really want to force a choice between two options.
A dichotomous choice in Diablo III
Continuous: This is where you ask people to give ratings along a continuum or sliding scale (like when you are doing face customization in games). The big advantage of a continuous scale is that it gives a lot of resolution. However that much resolution is hardly ever needed (is there really a difference between a rating of 96.43 and a 93.21?).
A (semi) continuous scale in Saints Row: The Third
Interval: This is probably the type of scale we are most familiar with. Here, you give a rating along some sort of scale with each step being separate/discrete from the other (rather than continuous).
This does lower the resolution of the scale when compared to a continuous scale, but it is much clearer in terms of allowing for comparison of results. In my opinion, a 1-7 scale is probably the best way to go (over say a 1-5, or something larger) as it allows for some good resolution, while not being too broad. Others may disagree
Several interval scales in Arkham Horror
If an interval scale is used, then it can also be broken down further into several types: numeric/categorical, Likert, or semantic.
A numeric scale is simple; it just asks for a number, and is used often to rank things.
Likert scales you have probably seen many, many times and are usually for when you want to see how much someone agrees or disagrees with a certain statement, e.g. 1-7 where 1 is strongly disagree and 7 is strongly agree
Semantic scales are when you want to simply ask for a rating related to something or a value judgment, e.g. 1-7 where 1 is poor and 7 is good.
Each of these types of scales can be used where necessary, although generally speaking, it is best not to mix scale types too much. Otherwise people filling in the questionnaire might get confused and provide a rating that they think is using an agree to disagree dimension but is actually asking for poor to good.
Finally, with interval scales, you can either make them unipolar, where you ask for varying levels of one variable e.g. 1-7 where 1 is not very exciting and 7 is very exciting, or they can be bipolar where you contrast two different variables, e.g. 1-7 where 1 is very boring and 7 is very exciting. Unipolar scales zoom in on one area a bit better, but bipolar scales give people filling in the questionnaire more room to express their opinions. As with before, try to not mix these styles up within the same questionnaire too often, or at all.
Step 3. Put it together
You have your questions, so now’s the time to put it together. First, work out what medium you are going to use. Will it be collected via a computer (say a through a web interface), or via paper? Collecting via a computer is preferable if possible as it means that there will be less data entry later on, however sometimes the portability and ease of paper can win out.
Give yourself plenty of time to do this, and if you are planning on using a computer-based survey, there are many companies online that offer this kind of service. In particular I hear that SurveyMonkey is a popular choice, although I have not used it myself.
You should next consider what order to put your questions in. Generally speaking I advise putting the easy questions at the start. This gets people rolling, and as long as your questionnaire isn’t too long, they should be more prepared to tackle the bigger questions you want to ask later.
Then try and cluster the remaining questions in a sensible fashion, such as by subject, or what they refer to within the game. Don’t ask a bunch of questions about the weapons, then move on to the boss, and then go back to the weapons.
Also check to see if answering some questions excludes other questions or could cause new questions to be asked later on. If so, make sure this occurs. In other words, if you ask “Do you own an Xbox” yes/no, make sure later on the people who answered “no” aren’t asked to list the top ten Xbox games that they own.
Step 4. Test it
No plan survives its first contact with the players, so first test your questionnaire yourself. If you are using a computer-based questionnaire, make sure the data that comes out the end looks okay. Give it to a few other people (preferably people who are not too familiar with the game, or are not designers) and ask them to fill it in (with you outside of the room so you can’t help them), and then ask for their comments on the clarity of the questions. Essentially, user test it, and then change anything that might be wrong (and then test again…) This is a great deal of work, I know, but once you have a good questionnaire, you can possibly use it again in the future (or at least cannibalize it for juicy parts).
Pros and Cons
There is much more that could be said, but perhaps this is already too much for a “primer” — so I will move on. The best thing about a questionnaire is that you are asking the same questions to everyone, so you get consistent quantifiable data that you can compare between people.
However they lack follow-up, in that you can’t ask people why they selected a certain rating, they are not objective, and you do need quite a few people if you want to draw completely solid conclusions from them. That said, even when testing with just a few people, providing a questionnaire can give you some ideas about what those individuals thought.
Pros
Consistent
Quantifiable
Relatively quick to administer
Can be used on a large scale
Cons
Can lack follow-up
Not objective
Really at their best with large sample sizes
It can take a while to put together a good questionnaire
Hopefully that is more than enough for this first article. While again this article is in no way comprehensive, I hope it has proven to be useful. In the upcoming part of this rough primer, I will be covering interviews, observational methods, game metrics and biometrics.
Finally, if you are interested in games user research or work in the area, then please consider checking out the IGDA Special Interest Group for Games User Research (GUR-SIG) on LinkedIn (just search for the group). It is a great place to get together and discuss GUR with others working in or around the industry.(source:GAMASUTRA)
