WO2022141704A1 - 神经元网络的反向传播算法的学习方法 - Google Patents

神经元网络的反向传播算法的学习方法 Download PDF

Info

Publication number
WO2022141704A1
WO2022141704A1 PCT/CN2021/073578 CN2021073578W WO2022141704A1 WO 2022141704 A1 WO2022141704 A1 WO 2022141704A1 CN 2021073578 W CN2021073578 W CN 2021073578W WO 2022141704 A1 WO2022141704 A1 WO 2022141704A1
Authority
WO
WIPO (PCT)
Prior art keywords
layer
hidden layer
hidden
expected value
neural network
Prior art date
Application number
PCT/CN2021/073578
Other languages
English (en)
French (fr)
Inventor
孔繁生
周华
Original Assignee
光华临港工程应用技术研发(上海)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 光华临港工程应用技术研发(上海)有限公司 filed Critical 光华临港工程应用技术研发(上海)有限公司
Publication of WO2022141704A1 publication Critical patent/WO2022141704A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • the invention relates to the field of neuron networks, in particular to a learning method of a back-propagation algorithm of neuron networks.
  • Backpropagation algorithm (that is, backpropagation algorithm) is a learning algorithm suitable for multi-layer neuron network, which is based on gradient descent method.
  • the input-output relationship of BP network is essentially a mapping relationship: the function completed by a BP neural network with n input and m output is a continuous mapping from n-dimensional Euclidean space to a finite field in m-dimensional Euclidean space. Mapping is highly nonlinear. Its information processing ability comes from the multiple composition of simple nonlinear functions, so it has strong function recurrence ability. This is the basis for the application of the BP algorithm.
  • the basic flow of the backpropagation algorithm is shown in Figure 1, and the flow diagram is shown in Figure 2, including the input layer, the hidden layer and the output layer.
  • the hidden layer is set to be multi-layered. The more layers there are, the smaller the error of the entire network will be, but it will complicate the entire network, increase the training time of the network, and may also cause "overfitting". Therefore, how to balance the number of layers and the network training time is a problem that the existing technology needs to solve.
  • the technical problem to be solved by the present invention is to provide a learning method of the back-propagation algorithm of the neuron network, which can reduce the learning time.
  • the present invention provides a method for learning a back-propagation algorithm of a neuron network, including the following steps: providing a neuron network, the neuron network includes an input layer, a hidden layer, and an output layer, wherein
  • the hidden layer is a multi-layer structure, including a first hidden layer and a second hidden layer; the first expected value and the second expected value are respectively configured for the first and second hidden layers; the first hidden layer is trained according to the first expected value. , optimize the first hidden layer; keep the optimization result of the first hidden layer, perform training on the second hidden layer according to the second expected value, and optimize the second hidden layer.
  • the present invention can perform separate training for each layer in the hidden layer, and on the premise of maintaining the optimization result of the previous hidden layer, then perform training on the following hidden layer, and optimize layer by layer.
  • This optimization method does not need to repeatedly perform operations on other layers while learning a certain layer, so it can save the overall learning time.
  • Fig. 1 shows the basic flow of the back-propagation algorithm in the prior art.
  • FIG. 2 is a flow chart of the back-propagation algorithm in the prior art.
  • FIG. 3 is a schematic diagram of the implementation steps of the method according to a specific embodiment of the present invention.
  • FIG. 4 is a schematic diagram of the structure of a neuron network corresponding to the method according to a specific embodiment of the present invention.
  • step S20 providing a neuron network, the neuron network includes an input layer, a hidden layer, and an output layer, wherein the hidden layer is a multi-layer Layer structure, including the first hidden layer and the second hidden layer;
  • Step S21 respectively configure the first expected value and the second expected value for the first and second hidden layer;
  • Step S22 according to the first expected value to the first hidden layer Perform training to optimize the first hidden layer;
  • step S23 keep the optimization result of the first hidden layer, and perform training on the second hidden layer according to the second expected value to optimize the second hidden layer.
  • the neuron network includes an input layer X, hidden layers W1, W2, ... Wn, output layers O1, O2, ... On, and an expected value D1 , D2, ... Dn.
  • an expected value Dn can be configured separately for each hidden layer Wn.
  • each hidden layer can be trained separately, and on the premise of maintaining the optimization result of the previous hidden layer, the following hidden layers are trained and optimized layer by layer.
  • the execution flow of the back-propagation algorithm shown in Figures 1 and 2 is still executed. This optimization method does not need to repeatedly perform operations on other layers without learning a certain layer, so it can save the overall learning time.
  • a global learning strategy that is, a complete loop iteration, is first performed. After the response error is smaller than a preset range, the step of layer-by-layer learning is started to balance the learning time and accuracy to achieve the optimal effect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Feedback Control In General (AREA)

Abstract

本发明提供了一种神经元网络的反向传播算法的学习方法，包括如下步骤：提供一神经元网络，所述神经元网络包括输入层、隐藏层、和输出层，其中隐藏层为多层结构，包括第一隐藏层和第二隐藏层；为第一和第二隐藏层分别配置第一期待值和第二期待值；根据第一期待值对第一隐藏层执行训练，使第一隐藏层优化；保持第一隐藏层的优化结果，根据第二期待值对第二隐藏层执行训练，使第二隐藏层优化。本发明可以针对隐藏层中的每层进行单独的训练，保持上一隐藏层的优化结果的前提下，再对下面的隐藏层执行训练，逐层优化。这种优化方法可以不必在学习某一层的情况下，反复对其他层进行运算，因此可以节约总体的学习时间。

Description

神经元网络的反向传播算法的学习方法 技术领域
本发明涉及神经元网络领域，尤其涉及一种神经元网络的反向传播算法的学习方法。
背景技术
反向传播算法(即backpropagation算法)适合于多层神经元网络的一种学习算法，它建立在梯度下降法的基础上。BP网络的输入输出关系实质上是一种映射关系：一个n输入m输出的BP神经网络所完成的功能是从n维欧氏空间向m维欧氏空间中一有限域的连续映射，这一映射具有高度非线性。它的信息处理能力来源于简单非线性函数的多次复合，因此具有很强的函数复现能力。这是BP算法得以应用的基础。
反向传播算法的基本流程如图1所示，流程框图如图2所示，包括输入层(input layer)，隐藏层(hidden layer)和输出层(output layer)，输出层值与期待结果(desired value)，并将比较结果反向反馈给隐藏层，实现闭环的学习过程。为了提高神经元网络的计算能力，会对隐藏层设置为多层。层数越多整个网络的误差也就越小，但是会是整个网络复杂化，增加网络的训练时间，也有可能出现“过拟合”的情况。因此如何平衡层数和网络训练时间，是现有技术需要解决问题。
发明内容
本发明所要解决的技术问题是，提供一种神经元网络的反向传播算法的学习方法，可以降低学习时间。
为了解决上述问题，本发明提供了一种神经元网络的反向传播算法的学习方法，包括如下步骤：提供一神经元网络，所述神经元网络包括输入层、隐藏层、和输出层，其中隐藏层为多层结构，包括第一隐藏层和第二隐藏层；为第一和第二隐藏层分别配置第一期待值和第二期待值；根据第一期待值对第一隐藏层执行训练，使第一隐藏层优化；保持第一隐藏层的优化结果，根据第二期待值对第二隐藏层执行训练，使第二隐藏层优化。
本发明可以针对隐藏层中的每层进行单独的训练，保持上一隐藏层的优化结果的前提下，再对下面的隐藏层执行训练，逐层优化。这种优化方法可以不 必在学习某一层的情况下，反复对其他层进行运算，因此可以节约总体的学习时间。
附图说明
附图1所示是现有技术中反向传播算法的基本流程。
附图2所示是现有技术中反向传播算法的流程框图。
附图3所示是本发明一具体实施方式所述方法的实施步骤示意图。
附图4所示是本发明一具体实施方式所述方法对应的神经元网络结构示意图。
具体实施方式
下面结合附图对本发明提供的一种神经元网络的反向传播算法的学习方法的具体实施方式做详细说明。
附图3所示是本具体实施方式所述方法的实施步骤示意图，包括：步骤S20，提供一神经元网络，所述神经元网络包括输入层、隐藏层、和输出层，其中隐藏层为多层结构，包括第一隐藏层和第二隐藏层；步骤S21，为第一和第二隐藏层分别配置第一期待值和第二期待值；步骤S22，根据第一期待值对第一隐藏层执行训练，使第一隐藏层优化；步骤S23，保持第一隐藏层的优化结果，根据第二期待值对第二隐藏层执行训练，使第二隐藏层优化。
附图4所示是上述方法对应的神经元网络结构示意图，所述神经元网络包括输入层X，隐藏层W1、W2、……Wn，输出层O1、O2、……On，以及期待值D1、D2、……Dn。对应于有n个隐藏层的网络，可以为每个隐藏层Wn都单独配置一个期待值Dn。
上述神经元网络进行学习训练时，可以针对隐藏层中的每层进行单独的训练，保持上一隐藏层的优化结果的前提下，再对下面的隐藏层执行训练，逐层优化。在每一层的学习过程中，仍然执行附图1和2所示的反向传播算法的执行流程。这种优化方法可以不必在学习某一层的情况下，反复对其他层进行运算，因此可以节约总体的学习时间。
作为一种优选的具体实施方式，在上述步骤执行之前，首先执行全局学习策略，即完整的循环迭代。至响应误差小于一预设范围后，开始执行所述逐层学习的步骤，以平衡学习时间和精度，达到最优效果。
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。

Claims (3)

  1. 一种神经元网络的反向传播算法的学习方法，其特征在于，包括如下步骤：
    提供一神经元网络，所述神经元网络包括输入层、隐藏层、和输出层，其中隐藏层为多层结构，包括第一隐藏层和第二隐藏层；
    为第一和第二隐藏层分别配置第一期待值和第二期待值；
    根据第一期待值对第一隐藏层执行训练，使第一隐藏层优化；
    保持第一隐藏层的优化结果，根据第二期待值对第二隐藏层执行训练，使第二隐藏层优化。
  2. 根据权利要求1所述的方法，其特征在于，所述神经元网络还包括第三隐藏层；
    为第三隐藏层配置第三期待值；
    保持第一和第二隐藏层的优化结果，根据第三期待值对第三隐藏层执行训练，使第三隐藏层优化。
  3. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    执行全局学习策略，至相应误差小于一预设范围后，开始执行所述逐层学习的步骤。
PCT/CN2021/073578 2020-12-31 2021-01-25 神经元网络的反向传播算法的学习方法 WO2022141704A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011631825.5A CN113159311A (zh) 2020-12-31 2020-12-31 神经元网络的反向传播算法的学习方法
CN202011631825.5 2020-12-31

Publications (1)

Publication Number Publication Date
WO2022141704A1 true WO2022141704A1 (zh) 2022-07-07

Family

ID=76878572

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/073578 WO2022141704A1 (zh) 2020-12-31 2021-01-25 神经元网络的反向传播算法的学习方法

Country Status (2)

Country Link
CN (1) CN113159311A (zh)
WO (1) WO2022141704A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105404865A (zh) * 2015-11-16 2016-03-16 杭州电子科技大学 基于概率态受限玻尔兹曼机级联的人脸检测方法
CN106778921A (zh) * 2017-02-15 2017-05-31 张烜 基于深度学习编码模型的人员再识别方法
US20190196943A1 (en) * 2017-12-26 2019-06-27 Hitachi, Ltd. Coverage test support device and coverage test support method
CN110995365A (zh) * 2019-03-29 2020-04-10 国家无线电监测中心检测中心 射频基因库建立方法及违法违规无线电波检测方法
CN111814878A (zh) * 2020-07-09 2020-10-23 仲恺农业工程学院 基于ssda-helm-softmax的农业投入品实时分类预测方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105404865A (zh) * 2015-11-16 2016-03-16 杭州电子科技大学 基于概率态受限玻尔兹曼机级联的人脸检测方法
CN106778921A (zh) * 2017-02-15 2017-05-31 张烜 基于深度学习编码模型的人员再识别方法
US20190196943A1 (en) * 2017-12-26 2019-06-27 Hitachi, Ltd. Coverage test support device and coverage test support method
CN110995365A (zh) * 2019-03-29 2020-04-10 国家无线电监测中心检测中心 射频基因库建立方法及违法违规无线电波检测方法
CN111814878A (zh) * 2020-07-09 2020-10-23 仲恺农业工程学院 基于ssda-helm-softmax的农业投入品实时分类预测方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIN SHAOFEI, SHENG HUIXING, LI QINGWU: "Handwritten Digital Classification Based on the Stacked Sparse Autoencoders", WEICHULIJI = MICROPROCESSORS, DIANZI GONGYEBU, DONGBEI WEIDIANZI YANJIUSUO, CN, no. 1, 28 February 2015 (2015-02-28), CN , pages 47 - 51, XP055949020, ISSN: 1002-2279 *

Also Published As

Publication number Publication date
CN113159311A (zh) 2021-07-23

Similar Documents

Publication Publication Date Title
Li et al. Adaptive neural network finite-time control for multi-input and multi-output nonlinear systems with positive powers of odd rational numbers
CN109934336B (zh) 基于最优结构搜索的神经网络动态加速平台设计方法及神经网络动态加速平台
WO2020134828A1 (zh) 一种神经网络推理结构优化方法及装置
CN107909142A (zh) 一种神经网络的参数优化方法、系统及电子设备
CN110531614B (zh) 新颖的无刷直流电机模糊神经网络pi控制器
CN109766996A (zh) 卷积神经网络的优化方法、装置、存储介质和系统
CN114610897A (zh) 基于图注意力机制的医学知识图谱关系预测方法
US20210056427A1 (en) Apparatus and method for training deep neural network
WO2022141704A1 (zh) 神经元网络的反向传播算法的学习方法
CN113469891A (zh) 一种神经网络架构搜索方法、训练方法、图像补全方法
KR20210070262A (ko) 파이프라인 구조를 가지는 인공신경망용 연산 가속 장치
CN114861980A (zh) 一种基于bp-lstm模型的碳预测方法
JP3374476B2 (ja) ニューラルネットワーク構築方法
JP2019200657A (ja) 演算装置及び演算装置の制御方法
Lian et al. Performance enhancement for T–S fuzzy control using neural networks
CN106022482A (zh) 应用改进型模糊神经网络解耦循环流化床床温-床压的方法
CN110197262A (zh) 用于lstm网络的硬件加速器
Dutta et al. Automated deep learning platform for accelerated analog circuit design
WO2021036668A1 (zh) 神经网络的全局池化方法及众核系统
CN109754082A (zh) 卷积神经网络的优化方法、装置、存储介质和系统
KR102505043B1 (ko) 양자화 및 배치 폴딩을 통해 ai 추론을 수행하는 장치
KR102541462B1 (ko) 이형 코어 아키텍처 기반의 캡슐 신경망 추론 장치 및 그 방법
Nai et al. A Design of Reinforcement Learning Accelerator Based on Deep Q-learning Network
Su et al. Improving Optimizers by Runge-Kutta Method: A case study of SGD and Adam
Han et al. Improved constrained learning algorithms by incorporating additional functional constraints into neural networks

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21912484

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 231123)