ALERT-Transformer - Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data

Aug 4, 2026

Upload by Rowan

This work enables classic processing of continuous ultra-sparse spatiotemporal streams from event-based sensors using dense machine learning. The proposed hybrid pipeline combines asynchronous sensing with synchronous feature extraction, including an ALERT PointNet-style embedding that continuously updates via a leakage mechanism, a flexible readout that provides always up-to-date features at any sampling rate, and a patch-based Vision Transformer inspired method to exploit sparsity efficiently. A transformer trained for object and gesture recognition attains state-of-the-art performance with lower latency, while the asynchronous model operates at arbitrary desired sampling rates.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

Related Works

Event-Based Data Representations

ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data Carmen Martin-Turrero * 1 2 Maxence Bouvier * 1 Manuel Breitenstein 1 Pietro Zanuttigh 2 Vincent Parret 1 arXiv :2402 .01393v3 [ cs .CV] 30 Jul 2024 Abstract We seek to enable classic processing of continuous ultra-sparse spatiotemporal data generated by event-based sensors with dense machine learning models. We propose a novel hybrid pipeline composed of asynchronous sensing and synchronous processing that combines several ideas: (1) an embedding based on PointNet models – the ALERT module – that can continuously integrate new and dismiss old events thanks to a leakage mechanism,(2) a flexible readout of the embedded data that allows to feed any downstream model with always up-to-date features at any sampling rate,(3) exploiting the input sparsity in a patch-based approach inspired by Vision Transformer to optimize the efficiency of the method. These embeddings are then processed by a transformer model trained for object and gesture recognition. Using this approach, we achieve performances at the state-of-the-art with a lower latency than competitors. We also demonstrate that our asynchronous model can operate at any desired sampling rate. 1. Introduction Event-based sensors capture visual information in an eventdriven, asynchronous manner (Finateu et al., 2020 ; Gallego et al., 2020) . Efficiently exploiting their data has proven challenging as the vast majority of approaches published in the literature consist of either converting event-based data to dense representations, or deploying spiking neural networks (SNNs) on streams of events. The former allows to exploit standard machine learning (ML) frameworks such as PyTorch and Tensorflow, but does not leverage the inherent sparsity and other properties of event-based data *Equal contribution 1 Sony Semiconductor Solutions Europe, Sony Europe B.V, Stuttgart Laboratory 1, Zurich, Switzerland 2University of Padova, MEDIA Lab, Veneto, Italy. Correspondence to: Vincent Parret <[vincent.parret@sony.com](vincent.parret@sony.com) >. Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024 . Copyright 2024 by the author(s) . (Gehrig et al., 2019) . The latter relies on SNNs, which are hard to train and usually exhibit lower accuracy than an equivalent dense neural network. Furthermore, while the neuromorphic community has argued in favor of their higher energy efficiency for decades, recent research and breakthroughs in edge AI accelerators indicate this is still an open question (Dampfhoffer et al., 2023 ; Garrett et al., 2023 ; Moosmann et al., 2023 ; Caccavella et al., 2023) . Nevertheless, considering the inherent advantages of eventbased vision sensors, namely high dynamic range (HDR) and high temporal resolution – simultaneously, without any tradeoffs between the two –, we aim to find a way to leverage this sparse and low-latency data for real-world situations. Standard ML relies on tensor-based processing. Converting the stream of events – represented as tuples of values (x andy pixel coordinates, polarities and timestamp)– to a multidimensional tensor is thus a crucial step. The challenge involves (1) representing time in a reliable and continuous manner, allowing it to be processed similarly to the finite spatial and polarity dimensions,(2) continuously incorporating new events in the feature tensors which also requires forgetting previous events,(3) using limited computational resources to allow real-time processing. Our main contributions towards Event-Based ML are the following: • The ALERT module, an embedding based on PointNet which continuously integrates new events dismissing old ones via a leakage mechanism. This module introduces novel asynchronous embedding updates. • A flexible readout of the embedded data that can feed any downstream model with up-to-date features at diffe

FAQs

What problem does ALERT-Transformer address for event-based sensors?

It targets efficient processing of continuous ultra-sparse spatiotemporal data produced asynchronously by event-based sensors, avoiding limitations of converting to dense tensors or relying only on spiking neural networks.

What are the key components of the proposed hybrid pipeline?

The pipeline includes an ALERT embedding with continuous asynchronous updates via a leakage mechanism, a flexible readout for up-to-date features at any sampling rate, and a patch-based approach inspired by Vision Transformer to leverage input sparsity efficiently.

How does the method support both low latency and high accuracy?

It uses an ALERT-Transformer framework that can run synchronously for high-accuracy gesture recognition or asynchronously for ultra-low-latency decision-making, with a seamless interface between the two regimes.

Aug 4, 2026

Upload by Rowan