WO2021007919A1 - 一种基于自动寻址和递归信息整合的内存网络方法 - Google Patents
一种基于自动寻址和递归信息整合的内存网络方法 Download PDFInfo
- Publication number
- WO2021007919A1 WO2021007919A1 PCT/CN2019/101806 CN2019101806W WO2021007919A1 WO 2021007919 A1 WO2021007919 A1 WO 2021007919A1 CN 2019101806 W CN2019101806 W CN 2019101806W WO 2021007919 A1 WO2021007919 A1 WO 2021007919A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- memory
- information
- addressing
- automatic addressing
- network
- Prior art date
Links
- 230000015654 memory Effects 0.000 title claims abstract description 98
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000010354 integration Effects 0.000 title claims abstract description 16
- 238000013528 artificial neural network Methods 0.000 claims abstract description 22
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000000306 recurrent effect Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 4
- 230000006403 short-term memory Effects 0.000 claims description 4
- 230000010365 information processing Effects 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 7
- 230000009467 reduction Effects 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000012795 verification Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 230000001537 neural effect Effects 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000008034 disappearance Effects 0.000 description 2
- 238000004880 explosion Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000013210 evaluation model Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/06—Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1041—Resource optimization
- G06F2212/1044—Space efficiency improvement
Definitions
- the invention belongs to the field of deep learning, and relates to a recurrent neural network and a memory neural network, and more specifically to a memory network method based on automatic addressing and recursive information integration.
- recurrent neural network is a typical neural network used to process time series tasks. Its representative frameworks such as long and short-term memory networks (LSTM) and gated recurrent units (GRU) have good timing Modeling capabilities are applied to sequential tasks in various practical scenarios, such as speech recognition, text reasoning, and video analysis.
- LSTM long and short-term memory networks
- GRU gated recurrent units
- RNN only transmits a hidden state with a limited dimension in the front and back time steps, so the memory capacity of historical information is limited.
- the gradient can be directly propagated through the memory to a certain time step needed in the past during training, avoiding the accumulation of gradients, so it can alleviate the problems of gradient disappearance and explosion.
- Historical information can be directly stored in the memory matrix, which greatly enhances the network's ability to remember historical information.
- the addressing methods that memory reads and writes rely on are content-based addressing and location-based addressing. Such addressing methods consume more memory, and their space complexity is proportional to the size of the entire memory matrix. Due to the complexity of operations, Its speed is also slower.
- the processing unit that comprehensively calculates the read memory information and the hidden state information from the previous time step simply reuses the calculation steps of the LSTM, resulting in the inability to effectively use the memory information.
- the present invention provides a memory network based on automatic addressing and recursive information integration frame.
- the memory network method based on automatic addressing and recursive information integration of the present invention includes the following steps:
- the two gates calculated in formula (1) It is used to control the inflow of information of h t-1 and r t element by element, which is the meaning of formulas (2) and (3). Then, the information processing method of long short-term memory network (LSTM) is reused for comprehensive calculation, such as As shown in formula (4)(5), finally, the generated new hidden state h t and memory information r t are respectively controlled by the two output gates calculated in formula (4) and output after being connected;
- LSTM long short-term memory network
- the automatic addressing method is specifically to connect h t-1 with x t and send it to a fully connected layer to obtain an N-dimensional embedding.
- the embedding is regarded as an unnormalized memory addressing probability.
- gumbel-softmax to sample the probability into a one-hot vetor, and read out the D-dimensional entry r t corresponding to the position of the element 1 in the vector in the memory.
- the writing position written into the memory in step 3) is the position where r t is read out using the one-hot vector in step 1).
- the automatic addressing method uses only h t-1 and x t for memory addressing, and uses the gumbel-softmax function to perform normalization and one-hot sampling on the unnormalized probability vector.
- the calculation unit for recursive information integration has 3 new doors compared to the 4 doors of LSTM with They are used to control the inflow of h t-1 and r t information, and the direct output of r t information.
- the method of the present invention is a memory neural network framework based on automatic addressing and recursive information integration. It is an efficient and lightweight memory network method.
- the memory is read and written through automatic addressing operations with low time and space complexity.
- the whole framework has the characteristics of high efficiency, rapidity, and strong versatility. It is suitable for various timing processing tasks and exhibits performance beyond traditional LSTM and previous memory networks.
- Fig. 1 is a flowchart of the memory network method based on automatic addressing and recursive information integration of the present invention.
- Fig. 2 is the cross-entropy loss reduction curve of the verification set of the present invention on the task of array copy.
- Fig. 3 is the cross-entropy loss reduction curve of the verification set of the present invention on the task of repeated copying.
- Fig. 4 is the cross-entropy loss reduction curve of the verification set of the present invention on the task of related recall.
- Fig. 5 is the cross-entropy loss reduction curve of the verification set of the present invention on the priority ranking task.
- the present invention provides a memory network method based on automatic addressing and recursive information integration.
- the method is a memory neural network framework based on automatic addressing and recursive information integration.
- the method is based on automatic addressing operations with low time and space complexity.
- the memory is read and written, and the memory information is effectively used through a novel computing unit.
- the entire framework is highly efficient, fast, and versatile. It is suitable for various time-series processing tasks, and is superior to traditional LSTM and previous Memory network performance.
- the method of the present invention proposes a new memory network method based on automatic addressing and recursive information integration for time series tasks, that is, a memory loop neural network framework;
- Figure 1 is a flow chart of the memory network method of the present invention, and the specific implementation The way is as follows.
- the memory matrix of the memory recurrent neural network framework is an N ⁇ D-dimensional matrix, where N is the number of memory entries, and D is equal to the dimension of the hidden state of the RNN.
- the automatic addressing method of this memory loop neural network framework is to directly use the hidden state h t-1 transmitted by the RNN between different time steps to encode historical memory addressing information, and to address the memory in combination with the current input x t .
- the embedding vector is considered to be an unnormalized memory addressing probability and used
- a gumbel-softmax function proposed by the researcher samples the probability into a one-hot vector, and reads the D-dimensional memory information r t corresponding to the position of the element 1 in the vector.
- the two gates calculated in formula (1) They are respectively used to control the information flow of h t-1 and r t element by element, that is, the meaning of formulas (2) and (3).
- the information processing method of LSTM is multiplexed for comprehensive calculation, as shown in formula (4)(5), and finally, the new hidden state h t and memory information r t are generated using the two calculated in formula (4).
- the output gate outputs after information control and connection.
- the calculation unit first filters the unnecessary information in the input h t-1 and r t for the time step, and finally uses an output gate to control the information used for output in the r t . In this way, the fault tolerance and flexibility of reading memory information can be greatly increased.
- the calculation unit of recursive information integration compared with the 4 doors of LSTM, adds 3 new doors with They are used to control the inflow of h t-1 and r t information, and the direct output of r t information.
- This step is the calculation process that occurs in the block diagram where the words "ARMIN (Auto-addressing and Recurrent Memory Integration Network) cells" in Figure 1 are located.
- the new hidden state h t generated at this moment is used as the information to be memorized and written into the memory.
- the written position is the position of r t read using the one-hot vector in operation 1). This step is shown in the part of the process framed by the word "write" in Figure 1.
- this set of algorithm tasks are divided into: a) Array copy: In the first 50 time steps, 50 randomly generated 6-bit binary numbers are input into the recursive network, and the network is required to output the previous input in the same order in the next 50 time steps Target array.
- cross-entropy loss to measure the degree of deviation between the actual output array and the target array. The lower the degree of deviation, the lower the cross-entropy loss, indicating that the recursive network uses historical memory information to complete tasks.
- This memory loop neural network framework is the same as the previous memory network, namely TARDIS (Temporal Automatic Relation Discovery in Sequences), AwTA (ARMIN with TARDIS Addressing) shown in Figures 2 to 5, based on TARDIS addressing. ARMIN), SAM (Sparse Access Memory), DNC (Differentiable Neural Computer), NTM (Neural Turing Machine, Neural Turing Machine) and other frameworks have been compared.
- the framework exhibits the characteristics of rapid loss of loss on the four tasks, low final convergence loss, and fewer iterations, indicating that it has a higher utilization of training samples. .
- This shows the advantages of the fast learning speed of the memory addressing mechanism of this framework and the high efficiency of using memory information as mentioned above.
- the actual running speed of the framework is 3 to 4 times that of the NTM with better performance in the comparison framework.
- the present invention is suitable for the field of deep learning, and is particularly suitable for recurrent neural networks and memory neural networks.
- the present invention is based on the memory neural network framework of automatic addressing and recursive information integration, and the memory is controlled by automatic addressing operations with low time and space complexity. Read and write, and make effective use of memory information through a novel computing unit.
- the entire framework is highly efficient, fast, and versatile. It is suitable for various time-series processing tasks, and it surpasses traditional LSTM and previous memory Network performance.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (5)
- 一种基于自动寻址和递归信息整合的内存网络方法,包括如下步骤:1)使用自动寻址对内存矩阵进行读操作,直接利用递归神经网络(RNN)在不同时间步之间传递的隐状态h t-1对历史的内存寻址信息进行编码,结合当前的输入x t对内存进行寻址;2)使用递归信息整合的计算单元对隐状态ht-1、内存信息rt和输入xt进行综合计算:公式(1)中计算的两个门 分别用于逐元素地控制h t-1和r t的信息流入,即公式(2)、(3)的含义,其后复用长短期记忆网络(LSTM)的信息处理方式进行综合计算,如公式(4)(5)所示,最后,将产生的新隐状态h t和内存信息r t分别用公式(4)中计算的两个输出门进行信息控制后并连接后输出;3)对内存的写操作:将该时刻产生的新的隐状态h t作为需要记忆的信息,写入内存;4)进入下一个时间步:将h t传入下一个时间步,接收输入x t+1,回到步骤1)循环执行上述步骤。
- 根据权利要求1所述的内存网络方法,其特征在于,所述自动寻址方法具体为将h t-1与x t连接起来,并送入一个全连接层,得到一个N维的嵌入向量,所述嵌入向量认为是一个未归一化的内存寻址概率,并用gumbel-softmax函数将该概率采样成一个独热向量,并读出内存中对应该矢量中元素为1的位置的D维的内存信息r t。
- 根据权利要求2所述的内存网络方法,其特征在于,步骤3)中写入内存的写入位置为步骤1)中使用独热向量读出r t的位置。
- 根据权利要求1所述的内存网络方法,其特征在于,所述自动寻址方法仅使用h t-1与x t进行内存寻址,并使用gumbel-softmax函数对未归一化的概率矢量进行归一化和独热采样。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/423,223 US20220138525A1 (en) | 2019-07-15 | 2019-08-21 | Memory network method based on automatic addressing and recursive information integration |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910635623.9A CN110348567B (zh) | 2019-07-15 | 2019-07-15 | 一种基于自动寻址和递归信息整合的内存网络方法 |
CN201910635623.9 | 2019-07-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021007919A1 true WO2021007919A1 (zh) | 2021-01-21 |
Family
ID=68175226
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/101806 WO2021007919A1 (zh) | 2019-07-15 | 2019-08-21 | 一种基于自动寻址和递归信息整合的内存网络方法 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20220138525A1 (zh) |
CN (1) | CN110348567B (zh) |
WO (1) | WO2021007919A1 (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210150345A1 (en) * | 2019-11-14 | 2021-05-20 | Qualcomm Incorporated | Conditional Computation For Continual Learning |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9483361B2 (en) * | 2013-05-08 | 2016-11-01 | Commvault Systems, Inc. | Information management cell with failover management capability |
CN106650922A (zh) * | 2016-09-29 | 2017-05-10 | 清华大学 | 硬件神经网络转换方法、计算装置、编译方法和神经网络软硬件协作系统 |
CN107704916A (zh) * | 2016-08-12 | 2018-02-16 | 北京深鉴科技有限公司 | 一种基于fpga实现rnn神经网络的硬件加速器及方法 |
CN108734272A (zh) * | 2017-04-17 | 2018-11-02 | 英特尔公司 | 卷积神经网络优化机构 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9807473B2 (en) * | 2015-11-20 | 2017-10-31 | Microsoft Technology Licensing, Llc | Jointly modeling embedding and translation to bridge video and language |
US10049106B2 (en) * | 2017-01-18 | 2018-08-14 | Xerox Corporation | Natural language generation through character-based recurrent neural networks with finite-state prior knowledge |
EP3566182A1 (en) * | 2017-02-06 | 2019-11-13 | Deepmind Technologies Limited | Memory augmented generative temporal models |
US20180349765A1 (en) * | 2017-05-30 | 2018-12-06 | Xerox Corporation | Log-linear recurrent neural network |
US10258304B1 (en) * | 2017-11-29 | 2019-04-16 | Siemens Healthcare Gmbh | Method and system for accurate boundary delineation of tubular structures in medical images using infinitely recurrent neural networks |
CN109613178A (zh) * | 2018-11-05 | 2019-04-12 | 广东奥博信息产业股份有限公司 | 一种基于递归神经网络预测空气污染的方法及系统 |
CN109753897B (zh) * | 2018-12-21 | 2022-05-27 | 西北工业大学 | 基于记忆单元强化-时序动态学习的行为识别方法 |
-
2019
- 2019-07-15 CN CN201910635623.9A patent/CN110348567B/zh active Active
- 2019-08-21 WO PCT/CN2019/101806 patent/WO2021007919A1/zh active Application Filing
- 2019-08-21 US US17/423,223 patent/US20220138525A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9483361B2 (en) * | 2013-05-08 | 2016-11-01 | Commvault Systems, Inc. | Information management cell with failover management capability |
CN107704916A (zh) * | 2016-08-12 | 2018-02-16 | 北京深鉴科技有限公司 | 一种基于fpga实现rnn神经网络的硬件加速器及方法 |
CN106650922A (zh) * | 2016-09-29 | 2017-05-10 | 清华大学 | 硬件神经网络转换方法、计算装置、编译方法和神经网络软硬件协作系统 |
CN108734272A (zh) * | 2017-04-17 | 2018-11-02 | 英特尔公司 | 卷积神经网络优化机构 |
Also Published As
Publication number | Publication date |
---|---|
US20220138525A1 (en) | 2022-05-05 |
CN110348567B (zh) | 2022-10-25 |
CN110348567A (zh) | 2019-10-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109271522B (zh) | 基于深度混合模型迁移学习的评论情感分类方法及系统 | |
CN111275113B (zh) | 基于代价敏感混合网络的偏斜类时间序列异常检测方法 | |
CN107562784A (zh) | 基于ResLCNN模型的短文本分类方法 | |
CN106934352A (zh) | 一种基于双路分形网络和lstm的视频描述方法 | |
CN113657561B (zh) | 一种基于多任务解耦学习的半监督夜间图像分类方法 | |
EP4131069A1 (en) | Transfer model training method and apparatus and fault detection method and apparatus | |
Hu et al. | Transformation-gated LSTM: efficient capture of short-term mutation dependencies for multivariate time series prediction tasks | |
CN112613571A (zh) | 一种用于图像识别的量子神经网络方法、系统及介质 | |
WO2021208455A1 (zh) | 一种面向家居口语环境的神经网络语音识别方法及系统 | |
Mirza et al. | Efficient online learning with improved LSTM neural networks | |
CN111104555A (zh) | 基于注意力机制的视频哈希检索方法 | |
CN108881254A (zh) | 基于神经网络的入侵检测系统 | |
CN115062727A (zh) | 一种基于多阶超图卷积网络的图节点分类方法及系统 | |
CN115630298A (zh) | 基于自注意力机制的网络流量异常检测方法及系统 | |
WO2021007919A1 (zh) | 一种基于自动寻址和递归信息整合的内存网络方法 | |
CN112653684A (zh) | 一种基于多路特征感知长短期记忆的异常流量检测方法 | |
WO2023202484A1 (zh) | 神经网络模型的修复方法和相关设备 | |
Ghanim et al. | Arabic/English Handwritten Digits Recognition using MLPs, CNN, RF, and CNN-RF | |
Zia | Hierarchical recurrent highway networks | |
Ni et al. | Enhanced knowledge distillation for face recognition | |
Shetty et al. | Comparative analysis of different classification techniques | |
Fang et al. | A method of license plate location and character recognition based on CNN | |
CN114841063A (zh) | 一种基于深度学习的航空发动机剩余寿命预测方法 | |
Sun et al. | An image classification method based on Echo State Network | |
Liu et al. | An attempt to apply the homotopy method to the domain of machine learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19937460 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19937460 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19937460 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 140223) |