On the Statistical Challenges of Echo State Networks and Some Potential Remedies

Echo state networks are powerful recurrent neural networks. However, they are often unstable and shaky, making the process of finding an good ESN for a specific dataset quite hard. Obtaining a superb accuracy by using the Echo State Network is a challenging task. We create, develop and implement a family of predictably optimal robust and stable ensemble of Echo State Networks via regularizing the training and perturbing the input. Furthermore, several distributions of weights have been tried based on the shape to see if the shape of the distribution has the impact for reducing the error. We found ESN can track in short term for most dataset, but it collapses in the long run. Short-term tracking with large size reservoir enables ESN to perform strikingly with superior prediction. Based on this scenario, we go a further step to aggregate many of ESNs into an ensemble to lower the variance and stabilize the system by stochastic replications and bootstrapping of input data.

• Publication year

  2018

• Venue

  arXiv.org

• Publication date

  2018-02-20

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1802.07369
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• An ensemble quadratic echo state network for non‐linear spatio‐temporal forecasting

  2017cited by this paper
• Comparison of echo state network output layer classification methods on noisy data

  2017cited by this paper
• Echo State Network

  2017cited by this paper
• Echo State Networks-Based Reservoir Computing for MNIST Handwritten Digits Recognition

  2016cited by this paper
• Echo State Network Ensemble for Human Motion Data Temporal Phasing: A Case Study on Tennis Forehands

  2016cited by this paper
• Design and Analysis of a Neuromemristive Reservoir Computing Architecture for Biosignal Processing

  2016cited by this paper
• Functional echo state network for time series classification

  2016cited by this paper
• Optimizing the echo state network with a binary particle swarm optimization algorithm

  2015cited by this paper
• Real-Time Reservoir Computing Network-Based Systems for Detection Tasks on Visual Contents

  2015influential reference
• A Practical Guide to Applying Echo State Networks

  2012cited by this paper
• Connectivity, Dynamics, and Memory in Reservoir Computing with Binary and Analog Neurons

  2010cited by this paper
• Temporal-Kernel Recurrent Neural Networks

  2010cited by this paper
• Real-Time Epileptic Seizure Detection on Intra-cranial Rat Data Using Reservoir Computing

  2008cited by this paper
• On Estimating the Size and Confidence of a Statistical Audit

  2007cited by this paper
• Short-term electricity demand forecasting using double seasonal exponential smoothing

  2003cited by this paper
• Random Forests

  2001cited by this paper
• Bagging Predictors

  1996cited by this paper
• Model Building and Forecasting with ARIMA Processes

  1991cited by this paper
• Data: A Collection of Problems From Many Fields for the Student and Research Worker

  1987cited by this paper
• Data : a collection of problems from many fields for the student and research worker

  1985cited by this paper
• A Survey of statistical work on the Mackenzie River series of annual Canadian lynx trappings for the

  1977influential reference

• A Random Matrix Perspective of Echo State Networks: From Precise Bias-Variance Characterization to Optimal Regularization

  2025cites this paper
• Empirical Investigation into Configuring Echo State Networks for Representative Benchmark Problem Domains

  2025cites this paper
• Connectomes inform function: from time-varying dynamics to animal behaviour

  2025influential citation
• Computational Complexity Optimization of Neural Network-Based Equalizers in Digital Signal Processing: A Comprehensive Approach

  2024cites this paper
• Using echo state networks to inform physical models for fire front propagation

  2023cites this paper
• Initializing hyper-parameter tuning with a metaheuristic-ensemble method: a case study using time-series weather data

  2022influential citation
• Imposing Connectome-Derived Topology on an Echo State Network

  2022influential citation
• On the Post Hoc Explainability of Optimized Self-Organizing Reservoir Network for Action Recognition

  2022influential citation
• Bidirectional parallel echo state network for speech emotion recognition

  2022influential citation
• Using Connectome Features to Constrain Echo State Networks

  2022influential citation
• RESERVOIR COMPUTING WITH TRUNCATED NORMAL DISTRIBUTION FOR SPEECH EMOTION RECOGNITION

  2022cites this paper
• Computational Complexity Evaluation of Neural Network Applications in Signal Processing

  2022cites this paper
• A Pattern Recognition Framework for Signal Processing in Metaverse

  2022cites this paper
• Computational Complexity Evaluation of Neural Network Applications in Signal Processing

  2022cites this paper
• On the post-hoc explainability of deep echo state networks for time series forecasting, image and video classification

  2021cites this paper
• Performance Versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

  2021cites this paper
• Computational Modeling of Structural Synaptic Plasticity in Echo State Networks

  2021cites this paper
• Universal set of Observables for the Koopman Operator through Causal Embedding

  2021cites this paper
• Speech Emotion Recognition by Late Fusion for Bidirectional Reservoir Computing With Random Projection

  2021cites this paper
• The Application of Echo State Networks to Atypical Speech Recognition

  2021cites this paper
• Online Sequential Extreme Learning Machine based Instinct Plasticity for Classification

  2020cites this paper
• Self-Organizing Kernel-based Convolutional Echo State Network for Human Actions Recognition

  2020cites this paper
• Grammatical Structure Detection with Intrinsic Plasticity based Echo State Networks for Cognitive Robot

  2019cites this paper
• An Ensemble Learning Approach to the Predictive Stability of Echo State Networks

  2018cites this paper