Text Classification - Neural Networks VS Machine Learning Models VS Pre-trained Models - Final Project Report

Sep 17, 2026

Uploaded by Theodora

Text classification is addressed through a direct performance comparison among seven pre-trained language models, three standard neural network architectures, and three classic machine learning models. The study evaluates two embedding strategies—TF-IDF and GloVe—and shows that GloVe consistently outperforms TF-IDF. Experiments demonstrate that transformer-based pre-trained models such as BERT and DistilBERT achieve stronger results than standard and traditional algorithms, supported by the role of transfer learning and contextual embeddings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

1 Introduction

2 Related Work

3 Embedding Techniques

4 Dataset and Pre-processing

6 Neural Network Architectures

7 Machine Learning Models

8 Transfer Learning Results

Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models Christos Petridis [christos.petridis@temple.edu](christos.petridis@temple.edu) Temple University Philadelphia, Pennsylvania, USA arXiv :2412 .21022v1 [ cs .LG] 30 Dec 2024 Abstract Text classification is a very common task nowadays and there are many efficient methods and algorithms that we can employ to accomplish it. Transformers have revolutionized the field of deep learning, particularly in Natural Language Processing (NLP) and have rapidly expanded to other domains such as computer vision [6] [19], time-series analysis [1], and more. The transformer model was firstly introduced in the context of machine translation and its architecture relies on self-attention mechanisms [22] to capture complex relationships within data sequences. It is able to handle long-range dependencies more effectively than traditional neural networks (such as Recurrent Neural Networks and Multilayer Perceptrons) . In this work, we present a comparison between different techniques to perform text classification. We take into consideration seven pre-trained models, three standard neural networks and three machine learning models. For standard neural networks and machine learning models we also compare two embedding techniques: TF-IDF and GloVe, with the latter consistently outperforming the former. Finally, we demonstrate the results from our experiments where pre-trained models such as BERT and DistilBERT always perform better than standard models/algorithms. Keywords Text Classification, Transformers, Pre-Trained Models, Natural Language Processing, Embeddings Reference Format: Christos Petridis. 2024. Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models. CIS 5525 Neural Computation @ Temple University, Philadelphia, PA, USA 1 Introduction NLP has been a rapidly growing area due to its impact on how humans interact with technology. We have seen many applications ranging from voice-activated assistants and chatbots to information retrieval, text summarization, and sentiment analysis. Its relevance continues to expand with advances in GenAI (Generative AI), making it a field with vast research potential and numerous practical applications. NLP continues to expand due to its great capabilities in understanding, generating, and transforming human language. Some popular examples are the great advances in Large Language Models (LLMs) like ChatGPT (models: GPT-3.5, GPT-4o, GPT-4o mini, GPT-4 Turbo, GPT-4 etc.), LLaMA (models: Llama 3.1, Llama 3. 2 etc.) and others. These models have revolutionized the field This paper is NOT published. It is the Final Project Report for the CIS 5525 Neural Computation @ Temple University. Instructor: Prof. Hongchang Gao CIS 5525 Neural Computation, Fall 2024, Philadelphia, PA, USA © 2024 Copyright held by the owner/author. by achieving levels of fluency, comprehension, and contextual understanding like never before. LLMs have transformed the field of NLP with their ability to learn patterns in text from massive datasets. We have seen results that were not possible with earlier rule-based models. LLMs can generate human-like text, understand complex queries, and even perform tasks on which they were not explicitly trained, thanks to their pre-trained and transfer learning capabilities. Computers and, of course, AI models understand and generate human language through the use of embeddings. Embeddingsin NLP refer to dense vector representations of words, phrases, or even entire sentences that capture semantic meaning and contextual relationships within a numerical format that models can understand. Unlike traditional one-hot encoding methods, which are very sparse and fail to convey any information about how words relate to each other, embeddings encode words into continuous, low-dimensional spaces. This allows models to map semantically similar words close to each other

FAQs

What models are compared for the text classification task?

The comparison includes seven pre-trained models, three standard neural networks (MLP, RNN, TransformerEncoder), and three machine learning models (SVM, Random Forest, Logistic Regression).

How are embeddings handled in the experiments?

TF-IDF and GloVe embeddings are used for standard neural networks and machine learning models, while each pre-trained model uses its corresponding predefined embeddings.

Which embedding method performs better and why?

GloVe consistently outperforms TF-IDF, since dense embeddings better capture semantic and contextual relationships between words than sparse frequency-based representations.

Sep 17, 2026

Uploaded by Theodora