Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration

Jul 16, 2026

Earth system science is generating increasingly large, high-dimensional datasets from physics-based and AI-driven models. Embedding representations make such data searchable, yet nearest neighbors in latent space may be scientifically meaningful or may instead encode artifacts from preprocessing, geography, seasonality, or model bias. A provenance-aware open-source visual analytics workbench links embedding experiments to shared source data, metadata, spatial context, and model configurations. It supports interactive, multi-constraint image-level and patch-level retrieval inspection, enabling scalable out-of-core search over tens of millions of embeddings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Abstract

Introduction

Motivation and challenges of scientific search in latent space

arXiv:2605.00972v2 [[physics.data-an](physics.data-an)] 13 Jul 2026 Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration Nihanth W. Cherukuru* Matt Rehme† Kirsten J. Mayer‡ David John Gagne§ John Schreck¶ John Clyne|| Charlie Becker** NSF National Center for Atmospheric Research, Boulder, Colorado USA Figure 1: Overview of the workbench and example retrieval workflow with vision transformer based embedding models. (a) The visual analytics workbench augments conventional weather and climate analysis (shown in gray) by supporting a workflow for generating image and patch level embeddings and subsequent interactive exploration. (b) The system separates a shared source reference (In this example- RGB inputs) containing metadata, source imagery, and meteorological variables, from model-specific embedding experiments (shown in green) . Each experiment stores its model configuration and searchable embedding representations, enabling multiple models to be examined in the visual interface over the same source data ABSTRACT Earth system science is producing increasingly large, highdimensional datasets from both physics-based and AI-driven models. While embedding-based representations make these data searchable and serve as foundational building blocks for AI-driven discovery engines, nearest neighbors in latent spaces are not automatically scientifically meaningful. They may reflect real meteorological structures, or simply artifacts of preprocessing, geography, or model bias. Researchers therefore need visual tools to inspect latent space organization, trace search results back to physical evi- * e-mail: [ncheruku@ucar.edu](ncheruku@ucar.edu)[ ](ncheruku@ucar.edu)†[e-mail: mrehme@ucar.edu](e-mail: mrehme@ucar.edu)[ ](e-mail: mrehme@ucar.edu)‡[e-mail: kjmayer@ucar.edu](e-mail: kjmayer@ucar.edu) § e-mail: [dgagne@ucar.edu](dgagne@ucar.edu)[ ](dgagne@ucar.edu)e-mail: [schreck@ucar.edu](schreck@ucar.edu) e-mail: [clyne@ucar.edu](clyne@ucar.edu) ** e-mail: [cbecker@ucar.edu](cbecker@ucar.edu) dence, and evaluate candidate representations against one another. We present an open source visual analytics workbench designed to support this provenance-aware scientific retrieval workflow. The system links distinct embedding experiments to shared source data, metadata, spatial contexts, and model configurations. It enables interactive retrieval strategy design by allowing users to issue imagelevel and localized patch-level queries, apply multi-constraint filters, and inspect analogs through familiar meteorological views. This facilitates a discovery loop where scientists characterize a phenomenon in a well-understood dataset and use its latent signature to probe larger archives. While we demonstrate the workbench through a tropical cyclone retrieval scenario using a vision foundation model (DINOv3) on ERA5 data, the framework is modelagnostic and designed to integrate with other embedding architectures in the future. Finally, we evaluate its out-of-core retrieval backend, demonstrating that interactive visual search over tens of millions of embeddings is highly scalable on commodity hardware. Index Terms: Embeddings, Latent Space Visualization, Ensembles, Meteorology, Weather, Climate 1 INTRODUCTION Numerical weather prediction and Earth system models already produce petabyte-scale, multivariate, spatiotemporal datasets. AIbased weather and climate models compound this: they enable far larger ensembles and cheaper experimentation, opening the door to broader exploration of rare and extreme events. As datasetsize grows and the field moves toward combining machine-learning methods with physics-based simulation, the bottleneck increasingly shifts from technical feasibility of producing forecasts to searching, interpreting, and validating the resulting data. Embedding-based representations offer a promising way to make these datasets searchable

FAQs

Why are nearest neighbors in embedding latent space not automatically scientifically meaningful?

Nearest neighbors can reflect real meteorological structures, but they can also arise from artifacts in preprocessing, geography, seasonality, representation choices, or model bias.

What does the proposed visual analytics workbench enable?

It provides provenance-aware linking between embedding experiments and shared source data/metadata/spatial context, enabling interactive design of retrieval strategies with image-level and patch-level queries plus multi-constraint filters and familiar meteorological views.

How is scalability addressed for large numbers of embeddings?

The workbench evaluates an out-of-core retrieval backend and demonstrates highly scalable interactive visual search over tens of millions of embeddings on commodity hardware.

Jul 16, 2026

Upload by Ava Thompson