Can machine learning techniques analyze qualitative focus group data effectively?

挑战

比以往任何时候都多, researchers have access to a variety of broad, 丰富的, educationally relevant text data from several sources, such as literature databases (e.g., ERIC [教育 Resources Information Center]), open-ended responses from online courses/surveys, online discussion forums, transcribed audio of face-to-face classes or focus groups, 数字的文章, 还有社交媒体.

数据可用性方面的这种进步(加上新兴的分析技术)可以极大地增加发现新模式的可能性, efficiencies in research, and testing of new theories in educational contexts. 例如, can emerging analytic techniques based in machine learning, 比如主题建模, be used to analyze qualitative focus group data effectively, and what limitations and or recommendations can be identified?

解决方案

我们为伍斯特理工学院提供了支持，将主题建模的使用扩展到从焦点小组收集的数据上，这些小组的教师对学生实施了一些数学技术干预:, a game-based perceptual learning intervention; DragonBox 12+ (DragonBox), a widely used game-based technology application; and 2 versions of ASSISTments (Immediate Feedback and Active Control).

We compared the results of qualitative coding and topic modeling. Unlike the textual data typically used in text mining, 如上所述, the focus groups in this study involved dynamic human communications—that is, teachers sharing thoughts and experiences in a meaningful way while taking in, 处理, and responding to both the facilitator and other participants. 在这个过程中, 演员(在这种情况下), 教师)不断地轮流(并同时聆听)扮演演讲者和听众的角色(Watzlawick等)., 1967*; DeVito, 2016*). 像这样, the patterns of communication constantly evolve, 在焦点小组中交换信息的方向和深度可能会因教师群体和调解人的技能而大不相同.

这项调查使用的信息是与伍斯特理工学院合作进行的一项更大的随机对照试验(RCT)的一部分, 缅因大学, and Indiana University during the COVID-19 pandemic.

The larger study was an RCT across 10 middle schools (9 in-person and 1 virtual), 包括3,600+ 7年级学生. 本研究测试了三种教育技术干预对七年级学生代数理解的影响, across 4 conditions: (a) FH2T, (b) DragonBox, (c)即时反馈, (d)主动控制. The FH2T and DragonBox conditions represent use of game-based applications. Immediate Feedback entails problem sets using an online homework system, ASSISTments. For purposes of this study, 主动控制条件模仿传统的家庭作业，但仍然使用技术.

As part of the larger study, 教师参与焦点小组讨论(a)教师对学生对数学技术的反应的看法, (b)教师和学生在使用数学技术时遇到的挑战, (c) impacts of the various applications on student learning, (d) the unique impact of the pandemic on instructing students, and (e) suggestions for changes or improvements to the mathematics technologies. Out of the 34 teachers who implemented the study, 16 (47%) participated in 1 of the 4 focus group sessions.

在这个调查中, 我们检验了主题建模是否能够从教师焦点小组数据中提取出与更多定性分析方法一致的模式. Specific questions we explored in this study were:

• What themes emerged from the qualitative coding approach and the topic modeling approach?
• What limitations does topic modeling have regarding the analysis of the focus group data?
• 对于其他可能尝试使用从焦点小组收集的数据进行主题建模的研究人员，我们有什么建议?

与其他形式的文本数据(如文学、书籍或论文)不同，在这些文本数据中，一种类型的信息(如.g., information about actors, information about data source, information about findings) is organized within a section, data generated from focus groups reflected dynamic human communications (i.e., participants share thoughts and experiences in a meaningful way while taking in, 处理, and responding to the moderator and other participants). 引导者还可以引导参与者回到焦点小组的问题上，或者跟随焦点小组讨论的方向, depending on the research questions posed.

结果

主题建模结果与定性编码结果高度吻合. 主题建模和定性编码的最大区别是在更高的层次上对主题的组织/分类.g.， 5个主题vs. 3主题). 在较低的层次, both approaches identified many of the same sub themes and, 因此, 结果表明，学生们对使用不同的数学技术的反应是相似的. 在检查两种不同方法(主题建模和定性编码)的编码和叙事结果时, 两位研究人员发现，主题建模方法在捕获定性编码能够识别的细微信息方面效果较差. 例如, 定性编码能够更好地提供学生对每种学习工具的具体反应的细节, 因此, was able to provide more nuanced findings than topic modeling. 类似的, 定性编码能够更好地识别学生群体的差异和特定信息.e.特殊教育学生和速成学生)比主题建模方法.

本研究成功地将话题建模的应用扩展到焦点小组数据上. 一起使用, 研究结果表明，主题建模是对焦点群体数据进行多种编码的可行方法. It can easily be used prior to qualitative analysis to identify nodes (i.e., 主题)或同时进行或在定性分析之后，以确定定性分析中可能固有的优势和劣势. 定性编码员的好处是技术的快速特性，它允许使用主题建模进行更快的编码.

*注:

DeVito J.A. (2016). The interpersonal communication book (14日艾德.). London, England: Pearson 教育 Limited.

罗瓦兹拉威克,P.巴维拉斯，J.B.和杰克逊，D.D. (1967). 人类交际的语用学:对互动模式、病态和悖论的研究. 纽约，纽约州:W.W. 诺顿 & 公司.