Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning

Jul 16, 2026

Upload by Ophelia

As autonomous agents are deployed across diverse operational contexts, aligning behavior with human intent requires reward functions that stay robust under environmental changes instead of overfitting to a single setting. Inverse reinforcement learning (IRL) infers objectives from human feedback, yet prior teaching analyses emphasize single-environment, demonstration-only scenarios and leave open how multiple feedback modalities and environment dynamics jointly shape cross-environment generalization. This work analyzes modality-specific constraint strength and proposes hierarchical multi-MDP teaching.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages30

Reading Time1 h 16 min

Summary

Contribution(s)

Abstract

Introduction

arXiv :2607 .08647v 1 [ cs .LG] 9 Jul 2026 Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning Ali Larian, Qian Lin, Chang Zong Wu, Daniel S. Brown Keywords: Learning from Human Feedback, Machine Teaching, Reward Learning Summary As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single environment. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing analyses of optimal teaching approaches for IRL focus on single-environment, demonstrationonly settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments. Because demonstrations in one MDP entangle reward information with that environment’s specific structure, the resulting rewards frequently fail to generalize when the agent is deployed in a new setting. We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions. Contribution(s) 1. Analysis and insights into different feedback types for machine teaching of reward functions in both unlimited-data and limited-budget regimes, showing that comparisons impose the strongest global constraints while demonstrations are more constraint-efficient per query under tight budgets. Context: Prior work on teaching Inverse RL agents (Büning et al., 2022 ; Brown and Niekum, 2019) has primarily focused on demonstrations and did not systematically analyze how different feedback types constrain reward recovery. 2. A formal characterization of environment-dependent reward identifiability, showing that even unlimited feedback in a single MDP leaves residual reward ambiguity. Context: Existing machine teaching approaches for IRL (Büning et al., 2022 ; Brown and Niekum, 2019) typically assume a fixed environment and do not analyze how environment dynamics affect reward identifiability. 3. Hierarchical Set Cover Optimal Teaching (HSCOT), a framework that selects informative environments and feedback queries to efficiently constrain rewards across multiple MDPs. Context: Prior teaching methods (Büning et al., 2022 ; Brown and Niekum, 2019) operate within a single environment and cannot exploit variation in environment dynamics to reveal complementary reward constraints. 4. Empirical validation showing that HSCOT achieves higher constraint coverage and lower regret on held-out environments compared to uniform teaching under identical budgets. Context: Existing evaluations of machine teaching for IRL (Büning et al., 2022 ; Brown and Niekum, 2019) only consider single-environment teaching settings. Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning Ali Larian 1 , Qian Lin 1 , Chang Zong Wu 1 , Daniel S. Brown 1 {ali.larian, qian.lin, u1440478, [daniel.s.brown}@utah.edu](daniel.s.brown}@utah.edu)[ ](daniel.s.brown}@utah.edu)[1](1 Kahlert School of Computing)[ Kahlert School of Computing](1 Kahlert School of Computing), [University of Utah](University of Utah). Abstract As autonomous agents

FAQs

Why do reward functions learned from a single MDP often fail to generalize to new environments?

Because demonstrations collected within one MDP entangle reward information with that environment’s specific structure, the recovered rewards can be overly tied to that structure and break when the agent is deployed elsewhere.

How do different feedback modalities compare in constraining rewards in the unlimited-data regime?

In the unlimited-data regime, comparison feedback imposes strictly stronger global constraints than other modalities.

What is the hierarchical teaching approach proposed for reward learning across multiple MDPs?

The method first greedily selects informative environments that reveal complementary reward constraints, then strategically queries low-cost feedback within those environments, yielding lower regret and stronger generalization under the same feedback budget.

Jul 16, 2026

Upload by Ophelia