Automatic hyperparameter tuning in on-line learning: Classic Momentum and ADAM
Published in 2020 International Joint Conference on Neural Networks (IJCNN), 2020
We propose a method that adapts hyperparameters, namely step-sizes and momentum decay factors, in on-line learning with classic momentum and ADAM. The approach is based on the estimation of the short- and long-term influence of these hyperparameters on the loss value. In the experimental study, our approach is applied to on-line learning in small neural networks and deep autoencoders. Automatically tuned coefficients surpass or roughly match the best ones selected manually in terms of learning speed. As a result, on-line learning can be a fully automatic process, producing results from the first run, without preliminary experiments aimed at manual hyperparameter tuning.
Recommended citation: Wawrzyński, P., Zawistowski, P., & Lepak, Ł. (2020). "Automatic hyperparameter tuning in on-line learning: Classic Momentum and ADAM." 2020 International Joint Conference on Neural Networks (IJCNN), 1–8.
Download Paper