Long-term Forecasting using Tensor-Train RNNs

We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions. Furthermore, we decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs. We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.

• Publication year

  2017

• Venue

  arXiv.org

• Publication date

  2017-10-31

• Fields of study

  Computer Science

• Identifiers
  arXiv 1711.00073
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Deep State Space Models for Time Series Forecasting

  2018cited by this paper
• Practical Learning of Predictive State Representations

  2017cited by this paper
• Duality of Graphical Models and Tensor Networks

  2017cited by this paper
• Tensor-Train Recurrent Neural Networks for Video Classification

  2017cited by this paper
• Expressive power of recurrent neural networks

  2017cited by this paper
• Higher Order Recurrent Neural Networks

  2016cited by this paper
• Supervised Learning with Tensor Networks

  2016cited by this paper
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition

  2016cited by this paper
• Hierarchical Multiscale Recurrent Neural Networks

  2016cited by this paper
• Deep Learning

  2016cited by this paper
• Generating Long-term Trajectories Using Deep Hierarchical Networks

  2016cited by this paper
• Tensorizing Neural Networks

  2015cited by this paper
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

  2015cited by this paper
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

  2015cited by this paper
• On the Expressive Power of Deep Learning: A Tensor Analysis

  2015cited by this paper
• Deep learning in neural networks: An overview

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

  2014cited by this paper
• Spectral Tensor-Train Decomposition

  2014influential reference
• Nonparametric Estimation of Multi-View Latent Variable Models

  2013cited by this paper
• A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States

  2013cited by this paper
• A Method of Moments for Mixture Models and Hidden Markov Models

  2012cited by this paper
• Generating Text with Recurrent Neural Networks

  2011cited by this paper
• Tensor-Train Decomposition

  2011cited by this paper
• Exponential Families for Conditional Random Fields

  2004cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Universal approximation bounds for superpositions of a sigmoidal function

  1993influential reference
• Higher Order Recurrent Networks and Grammatical Inference

  1989influential reference
• Testing multidimensional integration routines

  1984cited by this paper
• Time series analysis, forecasting and control

  1972influential reference

• WEITS: A wavelet-enhanced residual framework for interpretable time series forecasting

  2026cites this paper
• CANet: ChronoAdaptive Network for Enhanced Long-Term Time Series Forecasting under Non-Stationarity

  2025cites this paper
• Trajectory prediction algorithm based on maneuver analysis for hypersonic glide vehicles in a deep neural network

  2025cites this paper
• Short-term forecast of ionospheric sporadic E intensity fusing physical parameters via deep learning

  2025cites this paper
• LETS Forecast: Learning Embedology for Time Series Forecasting

  2025cites this paper
• LLGformer: Learnable Long-range Graph Transformer for Traffic Flow Prediction

  2025cites this paper
• Foresee: ML-Driven, Communication-Efficient Time-Series Forecasting

  2025cites this paper
• A Scalable Factorization Approach for High-Order Structured Tensor Recovery

  2025cites this paper
• Beyond point forecasts: Uncertainty quantification in tensor extrapolation for relational time series data

  2024cites this paper
• Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers

  2024cites this paper
• conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks

  2024cites this paper
• Guaranteed Nonconvex Factorization Approach for Tensor Train Recovery

  2024cites this paper
• GCINet: global convolution interaction network with a pre-trained reversible normalization method for long-term time series forecasting

  2024cites this paper
• Compact lossy compression of tensors via neural tensor-train decomposition

  2024cites this paper
• Self-transformers: long-term series forecasting based on smooth series

  2024cites this paper
• ATT-LSTM: An Interpretable Deep Learning Framework for COVID Outbreak Prediction

  2024cites this paper
• WEITS: A Wavelet-enhanced residual framework for interpretable time series forecasting

  2024cites this paper
• Quantum State Tomography for Matrix Product Density Operators

  2023cites this paper
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture

  2023cites this paper
• Expanding the prediction capacity in long sequence time-series forecasting

  2023cites this paper
• Anomaly Detection in Battery Charging Systems: A Deep Sequence Model Approach

  2023cites this paper
• GNN and Encoder Integrated Model for Distributed Solar and Wind Power Forecasting

  2023cites this paper
• Efficient Time Series Classification Based on Learning Similar Trend Features in the Same Class Sequences

  2023cites this paper
• Tensor-Train Decomposition in the Presence of Interval-Valued Data

  2023cites this paper
• Boosting Defect Detection in Manufacturing using Tensor Convolutional Neural Networks

  2023cites this paper
• MLGN: multi-scale local-global feature learning network for long-term series forecasting

  2023cites this paper
• Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing

  2023cites this paper
• TS-Fastformer: Fast Transformer for Time-series Forecasting

  2023cites this paper
• Hierarchical Joint Graph Learning and Multivariate Time Series Forecasting

  2023cites this paper
• CTFNet: Long-Sequence Time-Series Forecasting Based on Convolution and Time–Frequency Analysis

  2023cites this paper
• Fast and highly coupled model for time series forecasting

  2023cites this paper
• Ultra-short-term wind speed and wind power forecast via selective Hankelization and low-rank tensor learning-based predictor

  2022cites this paper
• Put Attention to Temporal Saliency Patterns of Multi-Horizon Time Series

  2022cites this paper
• Long Sequence Time-Series Forecasting via Gated Convolution and Temporal Attention Mechanism

  2022cites this paper
• Applications of machine learning to finite volume methods

  2022cites this paper
• Energy Consumption Analysis and Prediction of New Energy Buses

  2022cites this paper
• Evaluating BERT on cloud-edge time series forecasting and sentiment analysis via prompt learning

  2022cites this paper
• SCIF-ARF: Container Anomaly Prediction for Container Cloud Platforms

  2022cites this paper
• Kernel-based Hybrid Interpretable Transformer for High-frequency Stock Movement Prediction

  2022cites this paper
• Inductive Attention for Video Action Anticipation

  2022cites this paper
• Temporal Saliency Detection Towards Explainable Transformer-Based Timeseries Forecasting

  2022cites this paper
• FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecasting

  2022cites this paper
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting

  2022cites this paper
• Tensor Neural Controlled Differential Equations

  2022cites this paper
• Foreformer: an enhanced transformer-based framework for multivariate time series forecasting

  2022cites this paper
• Low Complexity Rate-Adaptive Deep Joint Source Channel Coding for Wireless Image Transmission Using Tensor-Train Decomposition

  2022cites this paper
• Granger Causality using Neural Networks

  2022cites this paper
• Error Analysis of Tensor-Train Cross Approximation

  2022cites this paper
• NVIDIA-UNIBZ Submission for EPIC-KITCHENS-100 Action Anticipation Challenge 2022

  2022cites this paper
• Video Anomaly Detection via Prediction Network with Enhanced Spatio-Temporal Memory Exchange

  2022cites this paper
• Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting

  2022cites this paper
• Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing

  2022cites this paper
• Tensor-Train Recurrent Neural Networks for Interpretable Multi-Way Financial Forecasting

  2021cites this paper
• Network of Tensor Time Series

  2021cites this paper
• Neural Granger Causality

  2021cites this paper
• Designing Tensor-Train Deep Neural Networks For Time-Varying MIMO Channel Estimation

  2021cites this paper
• Spatiotemporal Deep Learning for Bridge Response Forecasting

  2021cites this paper
• Granger Causality: A Review and Recent Advances

  2021cites this paper
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

  2021cites this paper
• Traffic Flow Forecasting with Spatial-Temporal Graph Diffusion Network

  2021cites this paper
• Purchase Intent Forecasting with Convolutional Hierarchical Transformer Networks

  2021cites this paper
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

  2021cites this paper
• Recurrent Neural Networks for Edge Intelligence

  2021cites this paper
• Multi-scale tensor network architecture for machine learning

  2021cites this paper
• Accuracy and Efficiency Comparison of Object Detection Open-Source Models

  2021cites this paper
• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting

  2021cites this paper
• Higher Order Recurrent Space-Time Transformer

  2021cites this paper
• QTN-VQC: an end-to-end learning framework for quantum neural networks

  2021cites this paper
• Chaos as an interpretable benchmark for forecasting and data-driven modelling

  2021cites this paper
• Towards Better Long-range Time Series Forecasting using Generative Forecasting

  2021cites this paper
• MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data

  2021cites this paper
• Block Access Pattern Discovery via Compressed Full Tensor Transformer

  2021influential citation
• Probabilistic Transformer For Time Series Analysis

  2021cites this paper
• Industry Kinetic Energy Index Prediction via Stacked Transformer

  2021cites this paper
• Towards Better Long-range Time Series Forecasting using Generative Adversarial Networks

  2021cites this paper
• FiniteNet: A Fully Convolutional LSTM Network Architecture for Time-Dependent Partial Differential Equations

  2020cites this paper
• Block-term tensor neural networks

  2020cites this paper
• Generating Realistic Stock Market Order Streams

  2020cites this paper
• A Multi-Scale Tensor Network Architecture for Classification and Regression

  2020cites this paper
• A parallel sparse tensor benchmark suite on CPUs and GPUs

  2020cites this paper
• A Sparse Tensor Benchmark Suite for CPUs and GPUs

  2020cites this paper
• Tensorized Embedding Layers

  2020cites this paper
• Connecting weighted automata, tensor networks and recurrent neural networks through spectral learning

  2020cites this paper
• Probabilistic Time Series Forecasting with Structured Shape and Temporal Diversity

  2020cites this paper
• A Deep Learning Framework for COVID Outbreak Prediction

  2020cites this paper
• Physics-Informed Tensor-Train ConvLSTM for Volumetric Velocity Forecasting of the Loop Current

  2020cites this paper
• Improving Recurrent Neural Network Responsiveness to Acute Clinical Events

  2020cites this paper
• Physics-incorporated convolutional recurrent neural networks for source identification and forecasting of dynamical systems

  2020cites this paper
• Low-Rank Autoregressive Tensor Completion for Multivariate Time Series Forecasting

  2020cites this paper
• Do RNN and LSTM have Long Memory?

  2020cites this paper
• MARS: Masked Automatic Ranks Selection in Tensor Decompositions

  2020cites this paper
• An Overview of Neural Network Compression

  2020influential citation
• Hybrid time-aligned and context attention for time series prediction

  2020cites this paper
• Bridging the Gap Between Training and Inference for Spatio-Temporal Forecasting

  2020cites this paper
• DEEPEYE

  2020cites this paper
• Explainable Unsupervised Change-point Detection via Graph Neural Networks

  2020cites this paper
• Masters Thesis: Tensor Network B-splines

  2020cites this paper
• PhICNet: Physics-Incorporated Convolutional Recurrent Neural Networks for Modeling Dynamical Systems

  2020cites this paper
• Real Time Probabilistic Models for Robot Trajectories

  2020cites this paper
• Recurrent Graph Tensor Networks: A Low-Complexity Framework for Modelling High-Dimensional Multi-Way Sequences

  2020cites this paper