A Gold Standard Study of What Makes a Lightweight Game-Playing Agent Strong

Jul 16, 2026

Upload by Sage

Reinforcement learning agents for imperfect-information card games are limited by the quality of the opponents used for training, and by the lack of stable evaluation yardsticks. This work builds a strong fixed rule-based expert for Gin Rummy and uses it only for grading, never training. Over 100 controlled runs, trust-region updates, reward design, tougher-opponent curriculum, warm starting, and best-checkpoint selection raise performance from ~30% to ~36%. Capacity increases and multiple reward/state/imitation/LLM ideas do not remove the information bottleneck. The approach is validated on Leduc Hold’em with computable optimum, yielding a reusable lightweight recipe with robust statistics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Opponent bottleneck and evaluation yardsticks

Prior work limitations

Paper approach: fixed expert and controlled experiments

Contributions and findings

A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong Nima Kelidari, Mohammadsaeed Haghi, Mahdi Salmani University of Southern California {kelidari, haghim, [salmanis](salmanis}@usc.edu)[}](salmanis}@usc.edu)[@usc.edu](salmanis}@usc.edu) arXiv :2607 .06854v 1 [ cs .LG] 7 Jul 2026 Abstract Reinforcement learning agents for imperfect-information card games are only as strong as the opponents they train against, and they are hard to grade, since they beat a random opponent over 99 percent of the time and only tie copies of themselves. So we build a strong, fixed, rule-based expert for Gin Rummy and use it only as a yardstick, never for training. It beats every agent we trained 70 to 99 percent of the time. Across more than a hundred runs, we isolate what makes a lightweight agent stronger. Trust region updates, a well-aimed reward, a curriculum of tougher opponents, warm starting, and keeping the best checkpoint all help, and stacking them lifts a self-play champion from about 30 to 36 percent against the expert. Several ideas did not pay off. Shortterm and longer-term reward shaping, learned state embeddings, imitation and DAgger, and a live large language model opponent were each unhelpful, too slow, or too heavy to train at scale. Comparing MLP, convolutional, set-based, attention, and recurrent encoders shows that extra capacity does little to break the ceiling, suggesting the limit is information rather than network size. We add standard baselines (neural fictitious self-play and information set Monte Carlo search) and confirm the approach carries over to Leduc Hold’em, where the optimum is computable. The result is a lightweight, gameagnostic recipe that trains competitive agents without training on the expert, for any game a small model can handle, reported with robust statistics and released as a reusable package. Introduction Learning to play imperfect-information card games well is a long-standing testbed for game AI. Games such as poker, DouDizhu, Mahjong, and Gin Rummy hide the opponent’shand and the order of the deck, so a player must act under uncertainty, plan over a long horizon, and adapt to whoever sits across the table. The dominant recipe for building agents in these games is deep reinforcement learning (RL) with self-play, which has produced superhuman play in several of them (Heinrich and Silver 2016 ; Li et al. 2020 ; Zha et al. 2021) . Behind those headline results, though, a practitioner who wants a small and fast agent runs into two practical problems that the headline results do not solve. Preprint. Submitted to the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2026) . This is the authors’ version and not the version of record. The first is the opponent bottleneck. An RL agent is only as good as the opponents it practices against. Training against a weak fixed opponent teaches a weak ceiling, and pure self-play can chase its own tail, cycling through strategies that beat the last version without getting stronger in any absolute sense. The second problem is that, for most of these games, the practitioner has no cheap absolute yardstick. Strength is usually measured against random play or against the agent’s own past checkpoints, both of which can flatter a mediocre agent. As a result, the design choices that actually build a strong lightweight agent (which RL algorithm, how to shape the reward, how to schedule opponents, how to represent the state, which checkpoint to ship) are tuned against references that are either too weak or constantly moving. The field has many strong single agents but few clean, controlled accounts of which of these choices matter and why. If we had a fixed, strong, and cheap reference opponent, we could measure these choices cleanly and turn them into a reusable recipe. Existing lines of work do not fill this gap directly. The famous self-play systems (Silver et al. 2018 ; Vinyals et al. 2019 ; Be

FAQs

Why does the quality of training opponents matter for imperfect-information card-game agents?

An RL agent’s strength is bounded by the opponents it practices against. Training against weak or moving references can create a low ceiling or misleading improvements without absolute progress.

What is the role of the Gin Rummy gold-standard expert in this study?

The authors build a strong, fixed, deterministic Gin Rummy expert and use it only as a yardstick for evaluation. The RL agents are never trained against this expert.

Which lightweight training choices were found to improve performance most?

Trust-region updates, a well-aimed reward design, an opponent curriculum that increases difficulty, warm starting, and keeping the best checkpoint each contribute to stronger play and together raise win rates against the expert.

Jul 16, 2026

Upload by Sage