Efficient implementations of echo state network cross-validation

by   Mantas Lukoševičius, et al.

Background/introduction: Cross-validation is still uncommon in time series modeling. Echo State Networks (ESNs), as a prime example of Reservoir Computing (RC) models, are known for their fast and precise one-shot learning, that often benefit from good hyper-parameter tuning. This makes them ideal to change the status quo. Methods: We suggest several schemes for cross-validating ESNs and introduce an efficient algorithm for implementing them. This algorithm is presented as two levels of optimizations of doing k-fold cross-validation. Training an RC model typically consists of two stages: (i) running the reservoir with the data and (ii) computing the optimal readouts. The first level of our proposed optimization addresses the most computationally expensive part (i) and makes it remain constant irrespective of k. It dramatically reduces reservoir computations in any type of RC system and is enough if k is small. The second level of optimization also makes the (ii) part remain constant irrespective of large k, as long as the dimension of the output is low. We discuss when the proposed validation schemes for ESNs could be beneficial, three options for producing the final model and empirically investigate them on six different real-world datasets, as well as do empirical computation time experiments. We provide the code in an online repository. Results: Proposed cross-validation schemes give better and more stable test performance in all the six different real-world datasets, three task types. Empirical run times confirm our complexity analysis. Conclusions: In most situations k-fold cross-validation of ESNs and many other RC models can be done for virtually the same time complexity as a simple single-split validation. Space complexity can also remain the same in all the cases. This enables cross-validation to become a standard practice in reservoir computing.


page 5

page 6


Efficient Cross-Validation of Echo State Networks

Echo State Networks (ESNs) are known for their fast and precise one-shot...

Towards new cross-validation-based estimators for Gaussian process regression: efficient adjoint computation of gradients

We consider the problem of estimating the parameters of the covariance f...

Fast Cross-Validation for Incremental Learning

Cross-validation (CV) is one of the main tools for performance estimatio...

Approximate Cross-Validation for Structured Models

Many modern data analyses benefit from explicitly modeling dependence st...

Approximate Cross-validated Mean Estimates for Bayesian Hierarchical Regression Models

We introduce a novel procedure for obtaining cross-validated predictive ...

On the discriminative power of Hyper-parameters in Cross-Validation and how to choose them

Hyper-parameters tuning is a crucial task to make a model perform at its...

Euler State Networks

Inspired by the numerical solution of ordinary differential equations, i...

Please sign up or login with your details

Forgot password? Click here to reset