KR102158683B1 - 외부 메모리로 신경망들 증강 - Google Patents

외부 메모리로 신경망들 증강 Download PDF

Info

Publication number
KR102158683B1
KR102158683B1 KR1020187017730A KR20187017730A KR102158683B1 KR 102158683 B1 KR102158683 B1 KR 102158683B1 KR 1020187017730 A KR1020187017730 A KR 1020187017730A KR 20187017730 A KR20187017730 A KR 20187017730A KR 102158683 B1 KR102158683 B1 KR 102158683B1
Authority
KR
South Korea
Prior art keywords
neural network
external memory
weights
locations
weight
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
KR1020187017730A
Other languages
English (en)
Korean (ko)
Other versions
KR20180091850A (ko
Inventor
알렉산더 벤자민 그라비스
이보 다니헬카
티모시 제이미스 알렉산더 할리
말콤 케빈 캠벨 레이놀즈
그레고리 던컨 웨인
Original Assignee
딥마인드 테크놀로지스 리미티드
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 딥마인드 테크놀로지스 리미티드 filed Critical 딥마인드 테크놀로지스 리미티드
Publication of KR20180091850A publication Critical patent/KR20180091850A/ko
Application granted granted Critical
Publication of KR102158683B1 publication Critical patent/KR102158683B1/ko
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G06N3/0445—
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/092—Reinforcement learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
KR1020187017730A 2015-12-10 2016-12-09 외부 메모리로 신경망들 증강 Active KR102158683B1 (ko)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201562265912P 2015-12-10 2015-12-10
US62/265,912 2015-12-10
PCT/US2016/066020 WO2017100711A1 (en) 2015-12-10 2016-12-09 Augmenting neural networks with external memory

Publications (2)

Publication Number Publication Date
KR20180091850A KR20180091850A (ko) 2018-08-16
KR102158683B1 true KR102158683B1 (ko) 2020-09-22

Family

ID=57708799

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020187017730A Active KR102158683B1 (ko) 2015-12-10 2016-12-09 외부 메모리로 신경망들 증강

Country Status (6)

Country Link
US (3) US10832134B2 (enExample)
EP (1) EP3371747B1 (enExample)
JP (1) JP6651629B2 (enExample)
KR (1) KR102158683B1 (enExample)
CN (1) CN108431832B (enExample)
WO (1) WO2017100711A1 (enExample)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10044751B2 (en) * 2015-12-28 2018-08-07 Arbor Networks, Inc. Using recurrent neural networks to defeat DNS denial of service attacks
US20180189266A1 (en) * 2017-01-03 2018-07-05 Wipro Limited Method and a system to summarize a conversation
WO2018142378A1 (en) * 2017-02-06 2018-08-09 Deepmind Technologies Limited Memory augmented generative temporal models
US11037330B2 (en) * 2017-04-08 2021-06-15 Intel Corporation Low rank matrix compression
US11243944B2 (en) * 2017-06-29 2022-02-08 Futurewei Technologies, Inc. Dynamic semantic networks for language understanding and question answering
CN107508866B (zh) * 2017-08-08 2020-10-02 重庆大学 减小移动设备端神经网络模型更新的传输消耗的方法
US11188820B2 (en) 2017-09-08 2021-11-30 International Business Machines Corporation Deep neural network performance analysis on shared memory accelerator systems
US10853725B2 (en) * 2018-05-18 2020-12-01 Deepmind Technologies Limited Neural networks with relational memory
JP6906478B2 (ja) * 2018-05-23 2021-07-21 株式会社東芝 情報処理装置、情報処理方法、およびプログラム
KR102788532B1 (ko) 2018-05-30 2025-03-31 삼성전자주식회사 뉴럴 네트워크 시스템, 이를 포함하는 어플리케이션 프로세서 및 뉴럴 네트워크 시스템의 동작방법
US11775815B2 (en) * 2018-08-10 2023-10-03 Samsung Electronics Co., Ltd. System and method for deep memory network
US20200090035A1 (en) * 2018-09-19 2020-03-19 International Business Machines Corporation Encoder-decoder memory-augmented neural network architectures
WO2020064988A1 (en) 2018-09-27 2020-04-02 Deepmind Technologies Limited Scalable and compressive neural network data storage system
CN110377342B (zh) * 2019-06-10 2022-08-30 平安科技（深圳）有限公司 基于卷积神经网络的显存处理方法、装置及存储介质
CN112149049A (zh) 2019-06-26 2020-12-29 北京百度网讯科技有限公司 用于变换矩阵的装置和方法、数据处理系统
US11651209B1 (en) * 2019-10-02 2023-05-16 Google Llc Accelerated embedding layer computations
US11461645B2 (en) * 2019-12-02 2022-10-04 International Business Machines Corporation Initialization of memory networks
US11604976B2 (en) 2020-04-29 2023-03-14 International Business Machines Corporation Crossbar arrays for computations in memory-augmented neural networks
US20220245217A1 (en) * 2021-01-29 2022-08-04 Oracle International Corporation Adaptive Selection of Source Matrix Version for Matrix Multiply Operations
US20260072786A1 (en) * 2024-09-10 2026-03-12 SK Hynix Inc. Read threshold prediction based on weight-sharing deep neural networks in non-volatile memory devices

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5167006A (en) * 1989-12-29 1992-11-24 Ricoh Company, Ltd. Neuron unit, neural network and signal processing method
US6092018A (en) * 1996-02-05 2000-07-18 Ford Global Technologies, Inc. Trained neural network engine idle speed control system
AU5804100A (en) * 1999-05-24 2000-12-12 Ip Century Ag Neuronal network for computer-assisted knowledge management
US20050041453A1 (en) * 2003-08-22 2005-02-24 Brazis Paul W. Method and apparatus for reading and writing to solid-state memory
GB0426982D0 (en) * 2004-12-09 2005-01-12 Secr Defence Early detection of sepsis
US7747070B2 (en) * 2005-08-31 2010-06-29 Microsoft Corporation Training convolutional neural networks on graphics processing units
US8239611B2 (en) * 2007-12-28 2012-08-07 Spansion Llc Relocating data in a memory device
US9514739B2 (en) * 2012-06-06 2016-12-06 Cypress Semiconductor Corporation Phoneme score accelerator
US9141906B2 (en) * 2013-03-13 2015-09-22 Google Inc. Scoring concept terms using a deep network
CN104657776B (zh) * 2013-11-22 2018-06-05 华为技术有限公司 神经网络系统、基于神经网络系统的图像解析方法和装置
CN103617235B (zh) * 2013-11-26 2017-01-25 中国科学院信息工程研究所 一种基于粒子群算法的网络水军账号识别方法及系统
US10366327B2 (en) * 2014-01-31 2019-07-30 Google Llc Generating vector representations of documents
CN103824291B (zh) * 2014-02-24 2017-01-11 哈尔滨工程大学 连续量子雁群算法演化脉冲耦合神经网络系统参数的自动图像分割方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv:1410.5401v1 (2014.10.20.)*
Weston, Jason, Sumit Chopra, and Antoine Bordes. "Memory networks." arXiv:1410.3916v11 (2015.11.29.)*

Also Published As

Publication number Publication date
CN108431832B (zh) 2022-09-13
US20170169332A1 (en) 2017-06-15
WO2017100711A1 (en) 2017-06-15
CN108431832A (zh) 2018-08-21
KR20180091850A (ko) 2018-08-16
US20250315676A1 (en) 2025-10-09
JP2018537788A (ja) 2018-12-20
JP6651629B2 (ja) 2020-02-19
EP3371747B1 (en) 2023-07-19
US10832134B2 (en) 2020-11-10
EP3371747A1 (en) 2018-09-12
US20210117801A1 (en) 2021-04-22
US12299575B2 (en) 2025-05-13

Similar Documents

Publication Publication Date Title
KR102158683B1 (ko) 외부 메모리로 신경망들 증강
US12099928B2 (en) Augmented recurrent neural network with external memory
US11210579B2 (en) Augmenting neural networks with external memory
EP3459021B1 (en) Training neural networks using synthetic gradients
US11080594B2 (en) Augmenting neural networks with external memory using reinforcement learning
CN109155002B (zh) 增强神经网络系统、方法和计算机程序
US10691997B2 (en) Augmenting neural networks to generate additional outputs
US11151443B2 (en) Augmenting neural networks with sparsely-accessed external memory
KR20180091842A (ko) 우선순위화된 경험 메모리를 사용한 신경 네트워크의 트레이닝
US11755879B2 (en) Low-pass recurrent neural network systems with memory
US11797769B1 (en) Artificial intelligence system using hybrid technique for task-oriented dialog management
KR102583943B1 (ko) 태스크 간의 상관관계 분석 알고리즘을 이용하여 연속학습을 수행하는 신경망 장치 및 신경망 학습 방법
CN118350944A (zh) 产品推荐方法、装置、电子设备、介质和计算机程序产品

Legal Events

Date Code Title Description
A201 Request for examination
PA0105 International application

Patent event date: 20180621

Patent event code: PA01051R01D

Comment text: International Patent Application

PA0201 Request for examination
PG1501 Laying open of application
E902 Notification of reason for refusal
PE0902 Notice of grounds for rejection

Comment text: Notification of reason for refusal

Patent event date: 20191212

Patent event code: PE09021S01D

AMND Amendment
E601 Decision to refuse application
PE0601 Decision on rejection of patent

Patent event date: 20200507

Comment text: Decision to Refuse Application

Patent event code: PE06012S01D

Patent event date: 20191212

Comment text: Notification of reason for refusal

Patent event code: PE06011S01I

X091 Application refused [patent]
AMND Amendment
PX0901 Re-examination

Patent event code: PX09011S01I

Patent event date: 20200507

Comment text: Decision to Refuse Application

Patent event code: PX09012R01I

Patent event date: 20200212

Comment text: Amendment to Specification, etc.

PX0701 Decision of registration after re-examination

Patent event date: 20200616

Comment text: Decision to Grant Registration

Patent event code: PX07013S01D

Patent event date: 20200608

Comment text: Amendment to Specification, etc.

Patent event code: PX07012R01I

Patent event date: 20200507

Comment text: Decision to Refuse Application

Patent event code: PX07011S01I

Patent event date: 20200212

Comment text: Amendment to Specification, etc.

Patent event code: PX07012R01I

X701 Decision to grant (after re-examination)
GRNT Written decision to grant
PR0701 Registration of establishment

Comment text: Registration of Establishment

Patent event date: 20200916

Patent event code: PR07011E01D

PR1002 Payment of registration fee

Payment date: 20200917

End annual number: 3

Start annual number: 1

PG1601 Publication of registration
PR1001 Payment of annual fee

Payment date: 20230906

Start annual number: 4

End annual number: 4

PR1001 Payment of annual fee

Payment date: 20240903

Start annual number: 5

End annual number: 5