What Do Customers Say About My Products? - Benchmarking Machine Learning Models for Need Identification

Aug 5, 2026

Upload by Jiven

Needmining extracts customer needs from user-generated content by classifying text as informative or uninformative for need content. Prior work cannot be compared directly because training and test data are private. This study benchmarks widely proposed needmining models, including CNN, SVM, RNN, and RoBERTa, using a sampled and annotated Amazon review dataset covering four product categories. The dataset is released publicly as a gold set. Results show RoBERTa outperforming other classifiers, and the benchmark establishes a different model hierarchy than previous, dataset-dependent evaluations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Customer needs and economic value

Limitations of traditional need elicitation

Needmining with machine learning and UGC

Research gap and study contributions

Related research

Customer needs

Proceedings of the 56th Hawaii International Conference on System Sciences | 2023 What Do Customers Say About My Products? Benchmarking Machine Learning Models for Need Identification Sven Stahlmann University of Cologne stahlmann@wim.uni[koeln.de](koeln.de) Oliver Ettrich Kühne Logistics University oliver.ettrich@the[klu.org](klu.org) Marco Kurka University of Cologne kurka@wim.uni[koeln.de](koeln.de) Detlef Schoder University of Cologne schoder@wim.uni[koeln.de](koeln.de) Abstract Needmining is the process of extracting customer needs from user-generated content by classifying it as either informative or uninformative regarding need content. Contemporary studies achieve this by utilizing machine learning. However, models found in the literature cannot be compared to each other because they use private data for training and testing. This study benchmarks all previously suggested needmining models including CNN, SVM, RNN, and RoBERTa. To ensure an unbiased comparison, this study samples and annotates a dataset of customer reviews for products from 4 different categories from amazon. Henceforth, the dataset is publicly available and serves as a gold-set for future needmining benchmarks. RoBERTa outperformed other classifiers and seems to be best suited for needmining. The relevance of this study is reinforced by the fact that this benchmark creates a different hierarchy between models than otherwise suggested by comparing the results of previous studies. Keywords: machine learning, natural language processing, customer needs, product innovation 1. Introduction Knowing and understanding customer needs is an important tool to increase customer satisfaction and the quality of products and services (Matzler & Hinterhuber, 1998) . For marketing departments, this can help to segment the market, identify strategic decisions (Park et al., 1986), and lead to better channel management decisions (Timoshenko & Hauser, 2019) . For research and development departments, customer needs help to identify new product opportunities (Eppinger & Ulrich, 2015; Herrmann et al., 2000) or improve existing ones (Matzler & Hinterhuber, 1998) . Therefore, knowing and understanding these needs is of great economic value. To exploit this economic value, methods such as observations, surveys, and interviews are traditionally used to identify such needs (Edvardsson et al., 2012; Griffin & Hauser, 1993) . However, these methods do not scale for large amounts of data, since a major part of the work is manual labor (Kühl et al., 2020) . Additionally, these methods are cost-intensive (Fisher et al., 2014), time-consuming (Griffin & Hauser, 1993) and can result in a delay of time to market. User-generated content (UGC) such as Amazon reviews and Twitter microblogs can be a low-cost source of customer needs (Kuehl et al., 2016) and are readily available in large quantities. Using traditional methods to identify needs in UGC is not feasible since the bulk of the content is either uninformative (i.e. it does not contain any needs) or repetitive. This results in high labor costs when analyzing UGC manually (Timoshenko & Hauser, 2019) because a lot of time is wasted on reading content that adds no value to market researchers. Consequently, a new, efficient approach to extract customer needs from UGC is needed. Machine learning looks to be a promising method to filter this vast amount of available UGC for need-containing content. Studies tackling this problem already exist (e.g. Christensen et al., 2017; Kuehl et al., 2016; Stahlmannet al., 2022; Timoshenko & Hauser, 2019; Zhang et al., 2021) . These studies apply various supervised machine learning classifiers to separate UGC into informative (defined as containing customer needs) and uninformative content. This is referred to as‘needmining’ (Kuehl et al., 2016) . Experts process the informative content further to extract insights for purposes such as innovation and product development. Howev

FAQs

What problem does needmining address?

Needmining extracts customer needs from user-generated content by classifying text as informative or uninformative regarding need content.

Why can’t earlier classifiers be directly compared?

Earlier studies use private training and testing data, so the models are evaluated under different conditions, preventing fair direct comparison.

Which model performs best in the benchmarking study?

RoBERTa outperforms the other evaluated classifiers and appears best suited for needmining.

Aug 5, 2026

Upload by Jiven