arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2412.17921

VITRO: Vocabulary Inversion for Time-series Representation Optimization

Filippos Bellos·Nam H. Nguyen·Jason J. Corso
View original source ↗Read source document ↗Export metadata ↓

Abstract

Although LLMs have demonstrated remarkable capabilities in processing and generating textual data, their pre-trained vocabularies are ill-suited for capturing the nuanced temporal dynamics and patterns inherent in time series. The discrete, symbolic nature of natural language tokens, which these vocabularies are designed to represent, does not align well with the continuous, numerical nature of time series data. To address this fundamental limitation, we propose VITRO. Our method adapts textual inversion optimization from the vision-language domain in order to learn a new time series per-dataset vocabulary that bridges the gap between the discrete, semantic nature of natural language and the continuous, numerical nature of time series data. We show that learnable time series-specific pseudo-word embeddings represent time series data better than existing general language model vocabularies, with VITRO-enhanced methods achieving state-of-the-art performance in long-term forecasting across most datasets.

Explore related subjects

cs.LGcs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Filippos Bellos, Nam H. Nguyen, Jason J. Corso. 2024-12-23. VITRO: Vocabulary Inversion for Time-series Representation Optimization. https://arxiv.org/abs/2412.17921

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability

Transfer-based adversarial attacks rely on surrogate models to craft perturbations, yet often overfit the surrogate's decision boundary. To address this problem, we propose Inverse Knowledge Distillation (IKD), a simple and attack-agnostic mechanism that maximizes the prediction-distribution discrepancy between benign and adversarial samples on the surrogate model. IKD uses a CE/KL-equivalent soft-label objective to push adversarial predictions away from a fixed benign prediction anchor and enrich the attack with Fisher-sensitive surrogate directions. We prove that, under a matched fixed-anchor implementation, soft-label cross-entropy and KL divergence differ only by a constant entropy term and therefore induce identical gradients, Hessians, and adversarial optimization trajectories. Our information-geometric analysis further derives a quantitative lower bound on dominant Fisher-subspace overlap between surrogate and target models from local same-task stability and a Fisher eigengap, and establishes a sufficient target-margin crossing condition under oriented gradient coherence and target smoothness. This analysis connects IKD's surrogate Fisher sensitivity to cross-model transfer. In contrast, mean squared error uses a different Euclidean pullback in output probability space. IKD integrates seamlessly with standard gradient-based attacks without modifying their optimization pipelines. Extensive ImageNet experiments demonstrate consistent black-box gains across CNN, ViT, and defended models, while ablations confirm CE and KL equivalence and the pronounced disadvantage of MSE. These results establish IKD as an effective and lightweight component for improving adversarial transferability. Code is available at https://github.com/ImmortalTing/IKD.

cs.LG↗
arXiv2026

A Generative-AI Modeling Framework for Explainable Decision Support in Complex Geosteering Scenarios

The real-time process of directional changes while drilling, known as geosteering, is crucial for hydrocarbon extraction and emerging directional drilling applications such as geothermal energy, civil infrastructure, and CO2 storage. The geo-energy industry seeks an automatic geosteering workflow that continually updates subsurface uncertainties and captures the latest geological understanding, informed by real-time observations. We propose a real-time, AI-driven geosteering workflow that integrates Generative Adversarial Networks (GANs) for geological parameterization, ensemble methods for model updating, and global discrete dynamic programming (DDP) optimization for complex decision-making during directional drilling operations. Our framework relies on offline training of a GAN model to reproduce relevant geology realizations and a Forward Neural Network (FNN) to model the response of Logging-While-Drilling (LWD) tools for a given geomodel. This paper introduces a first-of-its-kind workflow that progressively reduces GAN-geomodel uncertainty around and ahead of the drilling bit and adjusts the well plan accordingly. The workflow automatically integrates real-time around-bit LWD, which, through learned geological correlations, reduces uncertainty in predicted geology ahead of drilling. A DDP-based decision support system leverages probabilistic look-ahead predictions to suggest better steering strategies. We test the workflow prototype on a small yet challenging low-net-to-gross drilling scenario with several possible targets. The results show that the workflow produces meaningful steering recommendations and, through its probabilistic updates, automatically maps formation boundaries along the drilled well.

cs.LG↗
arXiv2026

R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks

This paper presents the Robust Recurrent Deep Network (R2DN), a scalable parameterization of stable and robust recurrent neural networks for machine learning and data-driven control. We construct R2DNs as the feedback interconnection of a linear time-invariant system and a 1-Lipschitz deep feedforward network, and directly parameterize the weights so that our models are stable (contracting) and robust to input perturbations (Lipschitz) by design. Our parameterization uses a structure similar to the recurrent equilibrium network (REN), but without having to iteratively solve an equilibrium layer at each time-step. This speeds up model inference and training on GPUs, and makes it computationally feasible to scale up the network size and input sequence length in comparison to RENs. We compare R2DNs to RENs on representative problems in nonlinear system identification, observer design, learning-based feedback control, and sequential image classification. We find that training and inference are up to an order of magnitude faster with similar performance, and that they scale more favorably with respect to model expressivity.

cs.LG↗