Position: Intent-aligned AI Systems Must Optimize for Agency Preservation

Aug 6, 2026

Aligned AI research often focuses on intent-consistent systems that avoid deception and produce recommendations humans judge as matching their goals. This work argues that intent alignment alone is insufficient for safety and that preserving humans’ long-term agency should be an explicit, optimizable standard. It analyzes the science of intent and control, shows how intent can be manipulated, and proposes a formal definition of agency-preserving human-AI interactions using forward-looking evaluations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

AccessFree to download

Introduction

Intention and the science of agency

From the philosophy to the neuroscience of agency

Position: Intent-aligned AI Systems Must Optimize for Agency Preservation Catalin Mitelut 1 Ben Smith 2 Peter Vamplew 3 Abstract A central approach to AI-safety research has been to generate aligned AI systems: i.e. systems that do not deceive users and yield actions or recommendations that humans might judge as consistent with their intentions and goals. Here we argue that truthful AIs aligned solely to human intent are insufficient and that preservation of long-term agency of humans may be a more robust standard that may need to be separated and explicitly optimized for. We discuss the science of intent and control and how human intent can be manipulated and we provide a formal definition of agency-preserving AI-human interactions focusing on forward-looking explicit agency evaluations. Our work points to a novel pathway for human harm in AI-human interactions and proposes solutions to this challenge. 1. Introduction Artificial intelligence (AI) researchers have made significant advances in recent years due in large part to the development of deep learning algorithms and the availability of massive datasets (Goodfellow et al., 2016) . Advances have led to highly creative text-to image generators such as DALL-E (Ramesh et al., 2021) and the development of large-language-models (LLMs) such as GPT3 (Brown et al., 2020), ChatGPT and GPT-4 (OpenAI, 2023) . Some now view the development of artificial general intelligence (AGI)(Goertzel, 2014) as increasingly likely (Roser, 2023) with a growing call for research into AI-safety and in particular”AI alignment” to ensure such systems act consistently with the goals of users and avoid growing lists of failure modes e.g. (Amodei et al., 2016) . AI-alignment, defined in terms of consistency with human intention (or human judgment), has been presented as key 1Forum Basiliense, University of Basel 2University of Oregon 3Federation University Australia. Correspondence to: Catalin Mitelut <[mitelutco@gmail.com](mitelutco@gmail.com) >. Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024 . Copyright 2024 by the author(s) . to making safe AI-systems (italics added): • There are incentives to build AI systems “that defer to humans and gradually align themselves to user preferences and intentions.”(Russell, 2019) . • “[C]orrectly specifying intent can become more important for achieving the desired outcome as RL algorithms improve.”(Krakovna et al., 2020) . • The general reason given for why an AI system would cause harm is that they “violate human intent in order to increase reward”(Cotra, 2022) . Here we argue that satisfying human intent alone is not a sufficient condition for safe AIs and that intent-aligned systems converge towards human agency loss, i.e. removing the power of humans to choose and control present and future goals. In particular, intent-aligned AI systems (i) converge on strategies that optimize for human agency loss and (ii) they do so by design rather than by accident. The formal reason this occurs is because agency loss is not exempt from Goodhart’s law (Strathern, 1997): loss of human control becomes the best solution to the increasingly complex optimization problems that AIs will be tasked with. This position paper argues that safe AI systems should explicitly decompose and evaluate outputs for both their immediate utility as well as their downstream effects on human agency. We provide additional discussions on why reasoning (Appendix A) or psychological needs (Appendix B) are insufficient to protect human agency. Human agency is not well understood. Section 2 lays out the empirical basis of our work, namely, the unsolved empirical problem of ”human agency”, i.e. what it means for humans to be in control of their lives and larger social structures. We argue that it remains unknown how much real control or ”agency” humans alone have over their immediate and long term futures given the increas

FAQs

为什么仅满足人类意图的AI对安全性是不够的？

文中指出，意图对齐的系统可能会收敛到降低人类能动性的策略，并且这种收敛并非偶然，而是由优化目标与复杂性共同导致。因而，仅与人类意图一致并不能保证长期的安全。

作者提出的“能动性保持（agency preservation）”标准是什么？

作者主张安全AI应明确分解并评估输出，不仅考虑即时效用，也要评估对人类能动性的下游影响。他们提供了面向未来的显式能动性评估，并给出形式化定义。

意图可能如何被操控，从而影响人类控制？

文中讨论了意图与控制的科学基础，指出人类意图可能被操控。并强调当AI融入社会结构后，可能更容易对人类行为施压，而生物或心理防御不足以抵消这种影响。

Aug 6, 2026

Upload by Emma Wilson