[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-580652-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-580652-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","superficially-plausible-outputs-from-a-black-box-problematising-genai-tools-for-analysing-qualitative-sotl-data","Superficially Plausible Outputs from a Black Box - Problematising GenAI Tools for Analysing Qualitative SoTL Data","","Generative AI tools are increasingly used for academic tasks, including qualitative analysis within Scholarship of Teaching and Learning (SoTL). As academic developers, the authors are asked whether using GenAI is reliable, valid, and ethical, yet the literature provides both optimistic and cautionary accounts. The paper experiments with chatbot-style GenAI tools to support qualitative analysis of survey and interview data previously analysed via thematic analysis. Outputs may look plausible but are incomplete and not reproducible, with undocumented reasoning and context-related limitations.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/superficially-plausible-outputs-from-a-black-box-problematising-genai-tools-for-analysing-qualitative-sotl-data/580652/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/superficially-plausible-outputs-from-a-black-box-problematising-genai-tools-for-analysing-qualitative-sotl-data/580652.png","ImageObject",300,407,{"name":42,"@type":43},"Miles","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-15","2026-10-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",13,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper identify with using GenAI for qualitative SoTL data analysis?","Question",{"text":63,"@type":64},"GenAI outputs can appear plausible but are incomplete and not reproducible, with interpretations and extrapolations that may not follow the dataset-only constraints in prompts. The lack of transparency about how results are generated prevents describing a reproducible method.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"Which GenAI tools are experimented with in the SoTL analysis workflow?",{"text":68,"@type":64},"The experiments use chatbot-style GenAI tools, specifically ChatGPT 4, ChatGPT 4o, and Microsoft Copilot, to support or conduct qualitative analysis of survey and interview data.",{"name":70,"@type":61,"acceptedAnswer":71},"Why is transparency and context important for qualitative analysis in this paper?",{"text":72,"@type":64},"Both algorithm and training data are undisclosed, making it impossible to know how outputs are derived. Additionally, the tools mainly analyze text transcripts and miss context such as interviewer tone, body language, and broader historical or organisational factors.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},580652,1792061843,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":116,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145,"preview_page_count":4},13056703019404,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","Mirjam Sophia Glessmer, LUND UNIVERSITY, [mirjam.glessmer@lth.lu.se](mirjam.glessmer@lth.lu.se)[ ](mirjam.glessmer@lth.lu.se)Rachel Forsyth, LUND UNIVERSITY, [rachel.forsyth@edusupport.lu.se](rachel.forsyth@edusupport.lu.se)  \nSoTL in Process  \nSuperficially Plausible Outputs from a Black Box: Problematising GenAI Tools for Analysing Qualitative SoTL Data  \nABSTRACT  \nGenerative AI tools (GenAI) are increasingly used for academic tasks, including qualitative data analysis for the Scholarship of Teaching and Learning (SoTL) . In our practice as academic developers, we are frequently asked for advice on whether this use for GenAI is reliable, valid, and ethical. Since this is a new field, we have not been able to answer this confidently based on published literature, which depicts both very positive as well as highly cautionary accounts. To fill this gap, we experiment with the use of chatbot style GenAI (namely ChatGPT 4, ChatGPT 4o, and Microsoft Copilot) to support or conduct qualitative analysis of survey and interview data from a SoTL project, which had previously been analysed by experienced researchers using thematic analysis. At first sight, the output looked plausible, but the results were incomplete and not reproducible. In some instances, interpretationsand extrapolations of data happened when it was clearly stated in the prompt that the tool should only analyse a specified dataset based on explicit instructions. Since both algorithm and training data of the GenAI tools are undisclosed, it is impossible to know how the outputs had been arrived at. We conclude that while results may look plausible initially, digging deeper soon reveals serious problems; the lack of transparency about how analyses are conducted and results are generated means that no reproducible method can be described. We therefore warn against an uncritical use of GenAI in qualitative analysis of SoTL data.  \nKEYWORDS  \nGenAI, generative AI, qualitative analysis, research methods, academic development  \nQUALITATIVE ANALYSIS OF SOTL DATA AND GENERATIVE ARTIFICIAL INTELLIGENCE  \nA lot of SoTL activity relies on analysing qualitative data sets which have been collected in a specific educational context. Traditionally, qualitative analysis means coding of the data based on how people familiar with the context read and interpreted it. This is done as transparently as possible, but it is also labour-intensive and prone to omissions or differences in interpretation, like any complex human judgements. As Generative Artificial Intelligence (GenAI) tools are becoming available, they are increasingly being explored as tools to support SoTL by mitigating human error and speeding up the analysis of large datasets.  \nGenAI tools are technologies that are able to generate new content, such as text, images, or audio, based on patterns learned from existing data. These technologies are based on Large Language Models, which have been trained using large sets of data to provide outputs, and then use human feedback to validate those outputs. In relation to research, it has been suggested that the tools might  \nbe useful for generating ideas, summarising and synthesising existing content, research and analysis, and asking questions (McCormack 2023) . This is despite the issue of “hallucinations,” described as“text that is nonsensical, or unfaithful to the provided source input”(Ji et al. 2023, 248) .  \nHannigan, McCarthy, and Spicer (2024) explain that GenAI tools are “‘predicting’ responses rather than ‘knowing’ the meaning of their responses” and coin the term “botshit” for what is produced by uncritically using GenAI output: Something that can happen to be right or wrong, but that is used without regard for its veracity. Lindebaum and Fleming (2023) argue that the use of GenAI tools in qualitative analysis would undermine responsible research and change our understanding of what research should be; they point out that GenAI tools have no stake in the outcome of the re","cbCaibgfIGaZxrI6","https://ap.wps.com/l/cbCaibgfIGaZxrI6","pdf",256099,"English","# ABSTRACT\n## Qualitative analysis of SoTL data and generative artificial intelligence\n## GenAI tool capabilities and limitations\n## Ethical issues in qualitative GenAI use\n## Interpretive sufficiency, context, and researcher agency\n## Perspectives on benefits and accuracy","[{\"question\":\"What problem does the paper identify with using GenAI for qualitative SoTL data analysis?\",\"answer\":\"GenAI outputs can appear plausible but are incomplete and not reproducible, with interpretations and extrapolations that may not follow the dataset-only constraints in prompts. The lack of transparency about how results are generated prevents describing a reproducible method.\"},{\"question\":\"Which GenAI tools are experimented with in the SoTL analysis workflow?\",\"answer\":\"The experiments use chatbot-style GenAI tools, specifically ChatGPT 4, ChatGPT 4o, and Microsoft Copilot, to support or conduct qualitative analysis of survey and interview data.\"},{\"question\":\"Why is transparency and context important for qualitative analysis in this paper?\",\"answer\":\"Both algorithm and training data are undisclosed, making it impossible to know how outputs are derived. Additionally, the tools mainly analyze text transcripts and miss context such as interviewer tone, body language, and broader historical or organisational factors.\"}]","Superficially Plausible Outputs from a Black Box - Problematising GenAI Tools for Analysing Qualitative SoTL Data | PDF",1792038025,23]