Multi-Modal Machine Learning for Assessing Gaming Skills in Online Streaming - A Case Study with CS:GO

Aug 5, 2026

Upload by Patrick

Online streaming is an emerging market where evaluating gaming skills from video content helps streaming service providers discover talented players and deliver customized recommendations and promotions. The study treats skill assessment as a multi-modal machine learning problem combining vision, audio, and text. It first audits and manually cleans the dataset, then trains end-to-end model variants to learn joint representations across modalities. Extensive experiments validate the proposals, while analysis highlights a risk of user-identification bias and motivates future work.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Introduction

Dataset issues and cleaning

Learning joint representations

Analysis of model behavior and dataset structure

Related Work

Multimodal Approaches

Audio-visual speech recognition (AVSR)

Multi-Modal Machine Learning for Assessing Gaming Skills in Online Streaming: A Case Study with CS:GO Longxiang Zhang Wenping Wang arXiv :2307 . 12236v 1 [ cs .LG] 23 Jul 2023 Abstract Online streaming is an emerging market that address much attention. Assessing gaming skills from videos is an important task for streaming service providers to discover talented gamers. Service providers require the information to offer customized recommendation and service promotion to their customers. Meanwhile, this is also an important multi-modal machine learning tasks since online streaming combines vision, audio and text modalities. In this study we begin by identifying ﬂaws in the dataset and proceed to clean it manually. Then we propose several variants of latest end-to-end models to learn joint representation of multiple modalities. Through our extensive experimentation, we demonstrate the efﬁcacy of our proposals. Moreover, we identify that our proposed models is prone to identifying users instead of learning meaningful representations. We purpose future work to address the issue in the end. 1. Introduction [Twitch.tv](Twitch.tv) is an online video streaming platform that has witnessed fast growth in recent years 1. It offers an open and live platform not only for people to have simple fun recording their gaming process, but also for potentially skilled gamers to showcase their gaming skills for possible talent scouting opportunities. This is especially true for popular e-sports that offer monetary tournaments and professional league careers. From an academic research point of view, Twitch.tv also presents a natural source of publicly available multimodal data (video, audio and chat) that is suitable fora gamut of deep learning tasks such as sentiment and emotion classiﬁcation (Ringer et al., 2019 ; Ameer et al., 2022 ; Zhang et al., 2022), video highlights (Fu et al., 2017), and spatial/temporal text grounding in video (Anne Hendrickset al., 2017 ; Yang et al., 2022a) . 1 [https://www.newyorker](https://www.newyorker) . com/magazine/2017/11/20/ how-to-get-rich-playing-video-games-online Figure 1 . Twitch.tv graphical user interface Figure 1 shows a streaming for the game Counter Strike: Global Offensive (CS:GO)2 CS:GO. The left hand side of the screen shows the player's view and the player himself. The right hand side of the screen is a chat room where other twitch users can chat. Besides, viewers can hear sound effects of the game and the player's voice. Recently, Belova et al. (2019) created a multimodal dataset [from Twitch.tv](from Twitch.tv)[ ](from Twitch.tv)streams of the game CS:GO. The authors investigated evaluating gaming skills of players using Bayesian networks. However, they only carried out coarse grained analysis and their methodology was limited to shallow statistical models. In this study, we seek to go beyond the approaches proposed in (Belova et al., 2019) . Recent end-to-end models (Assaelet al., 2016) (Tsai et al., 2019) have demonstrated success on multimodal classiﬁcation tasks. Based on these models we investigated learning join/coordinated representation from multiple modalities that lead to better performance on evaluating gaming skills. Moreover, we carry out in-depth analysis of this dataset. Firstly, during data exploration we discover that many of the videos in the dataset are not for the CS:GO game. This issue is neglected by previous work. Secondly, we investigate utilizing numeric ranking in evaluating gaming skills. Last but not least, we empirically analyze how our deep models learn from the data. On the one hand, there are only 1620 videos from 268 users. We empirically when making predictions our models tempt to utilize user iden- 2 [https://blog.counter-strike.net/](https://blog.counter-strike.net/) tity instead of meaningful features. This is an interesting problem and worth investigating in future work. On the other hand, CS:GO videos are semi-structured. As shown

FAQs

Why is assessing gaming skills from streaming videos important for providers?

It enables streaming service providers to identify talented gamers and support customized recommendations and service promotion for their customers.

What problem does the study find in the dataset before model training?

Many videos in the dataset are not actually about CS:GO, which was overlooked by previous work, so the authors manually clean and label the data by game.

What key risk do the proposed models exhibit according to the authors?

The models are prone to identifying users instead of learning meaningful representations, so the authors call for future work to address this issue.

Aug 5, 2026

Upload by Patrick