Pixel VQ-VAEs for Improved Pixel Art Representation

Aug 20, 2026

Machine learning image models often prioritize photorealism, leaving pixel art underexplored despite its discrete, visible structure and limited palette. Traditional pixel-focused learning methods can also struggle because many architectures aggregate neighboring pixels. This work introduces the Pixel VQ-VAE, a specialized VQ-VAE approach for learning pixel-art representations. Results show improved embedding quality and stronger downstream task performance compared with existing models.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Representation learning for pixel art

Background on embeddings and VAEs

Background

Pixel VQ-VAEs for Improved Pixel Art Representation Akash Saravanan1 , Matthew Guzdial1 1 University of Alberta, 116 St & 85 Ave, Edmonton, AB T6G 2R3, Canada Abstract Machine learning has had a great deal of success in image processing. However, the focus of this work has largely been on realistic images, ignoring more niche art styles such as pixel art. Additionally, many traditional machine learning models that focus on groups of pixels do not work well with pixel art, where individual pixels are important. We propose the Pixel VQ-VAE, a specialized VQ-VAE model that learns representations of pixel art. We show that it outperforms other models in both the quality of embeddings as well as performance on downstream tasks. Keywords Procedural Content Generation, Computational Creativity, Deep Learning, Machine Learning, Computer Vision, Pixel Art, Image Representation, Image Embedding 1. Introduction Deep neural networks have been used for a variety of image-related tasks including image generation [1], transformation [2], and translation [3] . However, the majority of this work focuses on photo-realism while avoiding other, more unrealistic art styles despite their usage in popular media. Pixel art is one such art style, characterized by a restricted color palette and discrete visible blocks of pixels (e.g. far-left of Table 1) . Originally created for 8- & 16-bit games, pixel art has remained popular, appearing in games like Minecraft, Pokémon, and Stardew Valley, as well as animations and webcomics [4, 5] . Improving our ability to work with pixel art in ML models could thus impact several domains. More specifically, this will lead to an improvement in any task that involves pixel art including generation and transformation of images in and outside of games. Prior work on pixel art has focused on specific tasks, primarily that of image generation [6, 7] . A shared representation for pixel art offers value as a common starting point for different tasks by saving human effort. In machine learning such representations, known as embeddings, are information-rich, multi-dimensional vectors learned by models for use in downstream tasks. Understandably, the vast majority of prior work on learning embeddings focuses on photorealism, leaving other art styles, including pixel art, largely unexplored. In terms of learning embeddings, Variational Auto Encoders (VAEs) [8] dominate the field due to their excellent representational capabilities [9, 2] . However, the EXAG 2022: AIIDE Workshop on Experimental AI in Games, October 25–25, 2022, Cal Poly Pomona, California [Envelope-Open](Envelope-Open akash.saravanan@ualberta.ca)[ akash.saravanan@ualberta.ca](Envelope-Open akash.saravanan@ualberta.ca) (A. Saravanan); [guzdial@ualberta.ca](guzdial@ualberta.ca) (M. Guzdial) Orcid 0000-0003-0720-1967 (A. Saravanan); 0000-0001-8673-9962 (M. Guzdial) © 2022 Copyright © 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4 .0 International (CC BY 4 .0) . CEURWorkshopProceedings CEUR Workshop Proceedings ([CEUR-WS.org](CEUR-WS.org)) [http://ceur-ws.org](http://ceur-ws.org) [ISSN 1613-0073](ISSN 1613-0073) generated images can appear blurry and lack detail [10], although recent work addresses this [11, 12] . Pixel art, characterized by discrete blocks of pixels, loses this defining property when the image is blurred. However, for pixel art, the problem may not be the VAEs themselves. Rather, Convolutional Neural Networks (CNNs), the backbone of many image processing models, process clumps of neighboring pixels together. This makes it difficult for such models to precisely target individual pixels in pixel art. One further complication is that pixel art is hand-authored and does not use natural images, thus severely limiting the available data. We identify two drawbacks with current approaches for pixel art. First, prior work has approached individual tasks separately when a c

FAQs

Why do many common ML image models struggle with pixel art?

Pixel art relies on discrete, visible pixels. Many models use convolutional processing that groups neighboring pixels together, and blurry outputs can destroy the defining pixel structure.

What does the Pixel VQ-VAE aim to achieve?

It learns representations specifically for pixel art using VQ-VAE principles, mapping discrete embeddings to groups of individual pixels while improving suitability for downstream tasks.

What enhancements are proposed to improve performance on pixel art?

The paper introduces the PixelSight block and an Adapter layer to enhance CNN performance for pixel art.

Aug 20, 2026

Upload by Callum