文献类型：期刊文献

英文题名：A zeroing neural dynamics based acceleration optimization approach for optimizers in deep neural networks

作者：Liao, Shan[1];Li, Shubin[1];Liu, Jiayong[1];Huang, Haoen[2];Xiao, Xiuchun[2]

机构：[1]Sichuan Univ, Sch Cyber Sci & Engn, Chengdu 610065, Peoples R China;[2]Guangdong Ocean Univ, Coll Elect & Informat Engn, Zhanjiang 524088, Peoples R China

年份：2022

卷号：150

起止页码：440

外文期刊名：NEURAL NETWORKS

收录：SCI-EXPANDED(收录号：WOS:000793324700013)、、EI(收录号：20221411879609)、Scopus(收录号：2-s2.0-85127361388)、WOS

基金：This work was supported in part by the Key Research and Development Program of Sichuan province, China under Grant 2020YFG0076, in part by the Sichuan Science and Technology Program, China under Grant 2021YFG0159, in part by the Sichuan Science and Technology Program, China under Grant 2021YFG0156, in part by the Fundamental Research Funds for the Central Universities, China.

语种：英文

外文关键词：Zeroing neural dynamics (ZND); Optimizer; Optimization; Deep neural networks (DNN)

外文摘要：The first-order optimizers in deep neural networks (DNN) are of pivotal essence for a concrete loss function to reach the local minimum or global one on the loss surface within convergence time. However, each optimizer possesses its own superiority and virtue when encountering a specific application scene and environment. In addition, the existing modified optimizers mostly emphasize a given optimizer without any transfer property. In this paper, a zeroing neural dynamics (ZND) based optimization approach for the first-order optimizers is proposed, which can assist ZND via the activation function to expedite the process of solving gradient information, with lower loss and higher accuracy. To the best of our knowledge, it is the first work to integrate the ZND in control domain with the first-order optimizers in DNN. This generic work is an optimization method for the most commonly-used first-order optimizers to handle different application scenes, rather than developing a brand-new algorithm besides the existing optimizers or their modifications. Furthermore, mathematic derivations concerning the gradient information transformation of the ZND are systematically provided. Finally, comparison experiments are implemented, which demonstrates the effectiveness of the proposed approach with different loss functions and network frameworks on the Reuters, CIFAR, and MNIST data sets. (C)& nbsp;2022 Elsevier Ltd. All rights reserved.