BeyondSight: Object Permanence for End-to-End Autonomous Driving

Jul 16, 2026

Upload by Eden

Autonomous driving often operates under partial observability, where vehicles or infrastructure can fully occlude nearby actors for extended periods. Many end-to-end driving systems implicitly tie actor existence to instantaneous observations, causing actor hypotheses to degrade or vanish during occlusion and depriving downstream prediction and planning of critical agents. BeyondSight decouples existence from observability by maintaining persistent, permanence-aware actor hypotheses over time. It propagates actor queries temporally and updates them using observation-conditioned evidence, enabling joint perception, prediction, and planning for temporarily unobservable actors, supported by the nuScenes-Permanence benchmark with supervision and evaluation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Introduction

Motivation: Partial Observability and Occlusion

Object Permanence Concept

Limitations of Existing End-to-End Driving Approaches

Benchmark Extension: nuScenes-Permanence

Proposed Framework: BeyondSight

arXiv :2607 .09138v1 [ cs .RO] 10 Jul 2026 BeyondSight: Object Permanence for End-to-End Autonomous Driving Sandro Papais 1 , Letian Wang 1 , Mudit Jain2 , Behnaz Rezaei2 , and Steven L. Waslander 1 1 University of Toronto {sandro.papais, [letian.wang](letian.wang) , [steven.waslander}@robotics.utias.utoronto.ca](steven.waslander}@robotics.utias.utoronto.ca) 2 Automated Driving, Qualcomm Technologies, Inc. {mudijain, brezaei}@qti.qualcomm.com Project page: [https://beyondsight-eccv.github.io](https://beyondsight-eccv.github.io) Abstract. Autonomous driving operates in partially observable environments where actors may become fully occluded by other vehicles or infrastructure. Most end-to-end driving systems implicitly couple actor existence to instantaneous observations, causing actor hypotheses to degrade or disappear during prolonged occlusion and removing potentially critical agents from downstream prediction and planning. We introduce BeyondSight, a permanence-aware end-to-end driving framework that decouples actor existence from observability by maintaining persistent actor hypotheses over time. BeyondSight propagates actor queries temporally and updates them with observation-conditioned evidence, enabling joint perception, prediction, and planning to reason about actors even when they are temporarily unobservable. To enable principled training and evaluation of persistence-aware models, we further introduce nuScenes-Permanence, an extension of nuScenes that provides supervision and observability-conditioned evaluation for unobservable actors. Experiments show that BeyondSight substantially improves reasoning under occlusion, increasing detection performance for unobservable actors from 0 to 0.249 mAP while reducing planning error from 0.61 to 0.54 L2avg . These results highlight object permanence as an important modeling principle for robust end-to-end autonomous driving. Keywords: Object Permanence · End-to-End Autonomous Driving · Spatiotemporal Reasoning · Partial Observability 1 Introduction Autonomous driving operates in partially observable environments where actors frequently become fully unobservable due to occlusion or sensor limits. Our analysis of nuScenes shows that approximately 30% of actors are fully unobservable at each timestep, including many within close proximity to the ego vehicle. Therefore, safe driving requires maintaining persistent hypotheses about actorseven when they are temporarily unobservable. This capability corresponds to 2 S. Papais et al. object permanence: the ability to reason about objects that continue to exist despite missing observations. Recent end-to-end autonomous driving (E2E-AD) systems jointly model perception, prediction, and planning within a unified architecture. However, most existing approaches implicitly couple actor existence to instantaneous observations. When an actor becomes fully unobservable, its representation often degrades or disappears entirely (Fig. 1), removing it from the scene representation used by downstream prediction and planning modules. As a result, the planner must reason only over currently visible agents, leading to reactive or brittle behavior in occlusion-heavy scenarios such as intersections and crosswalks. While temporal aggregation improves short-term stability, it does not explicitly model actor persistence during extended observability gaps. (a) Existing observable driving stacks. (b) BeyondSight. Fig. 1: Object permanence in prolonged occlusion. A pedestrian near a crosswalk becomes occluded. Top: End-to-end driving models (ex. SparseDrive) drop the actor hypothesis. Bottom: BeyondSight maintains a persistent representation during occlusion. Current perception benchmarks implicitly equate observability with existence: when an actor becomes fully occluded, it typically disappears from both supervision and evaluation. This prevents models from learning persistent representations across observability gap

FAQs

Why do current end-to-end autonomous driving systems struggle during prolonged occlusion?

They often couple actor existence to instantaneous observations, so actor representations degrade or disappear when an actor becomes fully unobservable. This removes the actor from downstream perception context used for prediction and planning.

What problem does BeyondSight address?

BeyondSight targets persistent reasoning about actors that remain present even when temporarily unobservable. It decouples actor existence from observability by maintaining persistent actor hypotheses over time.

How does nuScenes-Permanence support training and evaluation for object permanence?

nuScenes-Permanence extends nuScenes with annotations for unobservable actors and an observability-conditioned evaluation protocol. This enables systematic training and evaluation of persistence-aware models.

Jul 16, 2026

Upload by Eden