KR20180084988A - 압축된 순환 신경망 모델 - Google Patents

압축된 순환 신경망 모델 Download PDF

Info

Publication number
KR20180084988A
KR20180084988A KR1020187017732A KR20187017732A KR20180084988A KR 20180084988 A KR20180084988 A KR 20180084988A KR 1020187017732 A KR1020187017732 A KR 1020187017732A KR 20187017732 A KR20187017732 A KR 20187017732A KR 20180084988 A KR20180084988 A KR 20180084988A
Authority
KR
South Korea
Prior art keywords
neural network
network model
circular neural
compressed circular
compressed
Prior art date
Application number
KR1020187017732A
Other languages
English (en)
Other versions
KR102100977B1 (ko
Inventor
오우아시스 알샤리프
로히트 프라카시 프라브하발카르
이안 씨. 맥그로
앙투안 진 브뤼기에르
Original Assignee
구글 엘엘씨
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 구글 엘엘씨 filed Critical 구글 엘엘씨
Publication of KR20180084988A publication Critical patent/KR20180084988A/ko
Application granted granted Critical
Publication of KR102100977B1 publication Critical patent/KR102100977B1/ko

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
    • G10L15/063—Training
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/33—Director till display
    • G05B2219/33025—Recurrent artificial neural network
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40326—Singular value decomposition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Machine Translation (AREA)
KR1020187017732A 2016-02-03 2016-12-28 압축된 순환 신경망 모델 KR102100977B1 (ko)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201662290624P 2016-02-03 2016-02-03
US62/290,624 2016-02-03
PCT/US2016/068913 WO2017136070A1 (en) 2016-02-03 2016-12-28 Compressed recurrent neural network models

Publications (2)

Publication Number Publication Date
KR20180084988A true KR20180084988A (ko) 2018-07-25
KR102100977B1 KR102100977B1 (ko) 2020-04-14

Family

ID=57882138

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020187017732A KR102100977B1 (ko) 2016-02-03 2016-12-28 압축된 순환 신경망 모델

Country Status (7)

Country Link
US (2) US10878319B2 (ko)
EP (1) EP3374932B1 (ko)
JP (1) JP6706326B2 (ko)
KR (1) KR102100977B1 (ko)
CN (1) CN107038476A (ko)
DE (2) DE202016008253U1 (ko)
WO (1) WO2017136070A1 (ko)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021096641A1 (en) * 2019-11-13 2021-05-20 Micron Technology, Inc. Recurrent neural networks and systems for decoding encoded data
WO2021225256A1 (ko) * 2020-05-08 2021-11-11 삼성전자주식회사 전자 장치 및 이의 제어 방법
US11416735B2 (en) 2018-12-27 2022-08-16 Micron Technology, Inc. Neural networks and systems for decoding encoded data
US11563449B2 (en) 2021-04-27 2023-01-24 Micron Technology, Inc. Systems for error reduction of encoded data using neural networks
US11755408B2 (en) 2021-10-07 2023-09-12 Micron Technology, Inc. Systems for estimating bit error rate (BER) of encoded data using neural networks
US11973513B2 (en) 2021-04-27 2024-04-30 Micron Technology, Inc. Decoders and systems for decoding encoded data using neural networks

Families Citing this family (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10176424B2 (en) * 2016-02-05 2019-01-08 Deepmind Technologies Limited Generative neural networks
US10783535B2 (en) 2016-05-16 2020-09-22 Cerebri AI Inc. Business artificial intelligence management engine
US10599935B2 (en) * 2017-02-22 2020-03-24 Arm Limited Processing artificial neural network weights
US10762563B2 (en) 2017-03-10 2020-09-01 Cerebri AI Inc. Monitoring and controlling continuous stochastic processes based on events in time series data
US10402723B1 (en) 2018-09-11 2019-09-03 Cerebri AI Inc. Multi-stage machine-learning models to control path-dependent processes
US11037330B2 (en) * 2017-04-08 2021-06-15 Intel Corporation Low rank matrix compression
US11216437B2 (en) 2017-08-14 2022-01-04 Sisense Ltd. System and method for representing query elements in an artificial neural network
US11106975B2 (en) * 2017-10-20 2021-08-31 Asapp, Inc. Fast neural network implementations by increasing parallelism of cell computations
EP3698287A1 (en) * 2017-10-20 2020-08-26 Deepmind Technologies Limited Parallel execution of gated activation unit operations
US11556775B2 (en) * 2017-10-24 2023-01-17 Baidu Usa Llc Systems and methods for trace norm regularization and faster inference for embedded models
CN109993290B (zh) 2017-12-30 2021-08-06 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
CN109993289B (zh) 2017-12-30 2021-09-21 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
CN109993291B (zh) * 2017-12-30 2020-07-07 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
CN109993292B (zh) 2017-12-30 2020-08-04 中科寒武纪科技股份有限公司 集成电路芯片装置及相关产品
EP3624019A4 (en) 2017-12-30 2021-03-24 Cambricon Technologies Corporation Limited CHIP DEVICE WITH INTEGRATED CIRCUIT AND ASSOCIATED PRODUCT
US11586924B2 (en) * 2018-01-23 2023-02-21 Qualcomm Incorporated Determining layer ranks for compression of deep networks
US10657426B2 (en) * 2018-01-25 2020-05-19 Samsung Electronics Co., Ltd. Accelerating long short-term memory networks via selective pruning
US11593068B2 (en) * 2018-02-27 2023-02-28 New York University System, method, and apparatus for recurrent neural networks
CN110533157A (zh) * 2018-05-23 2019-12-03 华南理工大学 一种基于svd和剪枝用于深度循环神经网络的压缩方法
CN112437930A (zh) * 2018-07-12 2021-03-02 华为技术有限公司 以熟练的推理速度和功耗，生成神经网络的压缩表示
JP2020034625A (ja) * 2018-08-27 2020-03-05 日本電信電話株式会社 音声認識装置、音声認識方法、及びプログラム
US11068942B2 (en) 2018-10-19 2021-07-20 Cerebri AI Inc. Customer journey management engine
CN109523995B (zh) * 2018-12-26 2019-07-09 出门问问信息科技有限公司 语音识别方法、语音识别装置、可读存储介质和电子设备
CN109670158B (zh) * 2018-12-27 2023-09-29 北京及客科技有限公司 一种用于根据资讯数据生成文本内容的方法与设备
CN109740737B (zh) * 2018-12-30 2021-02-19 联想(北京)有限公司 卷积神经网络量化处理方法、装置及计算机设备
US11444845B1 (en) * 2019-03-05 2022-09-13 Amazon Technologies, Inc. Processing requests using compressed and complete machine learning models
CN110580525B (zh) * 2019-06-03 2021-05-11 北京邮电大学 适用于资源受限的设备的神经网络压缩方法及系统
CN112308197B (zh) * 2019-07-26 2024-04-09 杭州海康威视数字技术股份有限公司 一种卷积神经网络的压缩方法、装置及电子设备
US11922315B2 (en) * 2019-08-26 2024-03-05 Microsoft Technology Licensing, Llc. Neural adapter for classical machine learning (ML) models
WO2021117942A1 (ko) * 2019-12-12 2021-06-17 전자부품연구원 저복잡도 딥러닝 가속 하드웨어 데이터 가공장치
US11188616B2 (en) 2020-02-25 2021-11-30 International Business Machines Corporation Multi-linear dynamical model reduction
WO2021234967A1 (ja) * 2020-05-22 2021-11-25 日本電信電話株式会社 音声波形生成モデル学習装置、音声合成装置、それらの方法、およびプログラム
KR20220064054A (ko) * 2020-11-11 2022-05-18 포항공과대학교 산학협력단 행렬곱 연산량 감소 방법 및 장치
CA3168515A1 (en) * 2021-07-23 2023-01-23 Cohere Inc. System and method for low rank training of neural networks

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150170020A1 (en) * 2013-12-13 2015-06-18 Amazon Technologies, Inc. Reducing dynamic range of low-rank decomposition matrices

Family Cites Families (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5408424A (en) * 1993-05-28 1995-04-18 Lo; James T. Optimal filtering by recurrent neural networks
EP1356430B1 (en) * 2000-10-13 2010-07-14 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. A method for supervised teaching of a recurrent artificial neural network
US9235800B2 (en) * 2010-04-14 2016-01-12 Siemens Aktiengesellschaft Method for the computer-aided learning of a recurrent neural network for modeling a dynamic system
WO2012109407A1 (en) * 2011-02-09 2012-08-16 The Trustees Of Columbia University In The City Of New York Encoding and decoding machine with recurrent neural networks
US8489529B2 (en) * 2011-03-31 2013-07-16 Microsoft Corporation Deep convex network with joint use of nonlinear random projection, Restricted Boltzmann Machine and batch-based parallelizable optimization
US9292787B2 (en) * 2012-08-29 2016-03-22 Microsoft Technology Licensing, Llc Computer-implemented deep tensor neural network
US20140156575A1 (en) * 2012-11-30 2014-06-05 Nuance Communications, Inc. Method and Apparatus of Processing Data Using Deep Belief Networks Employing Low-Rank Matrix Factorization
US9519858B2 (en) * 2013-02-10 2016-12-13 Microsoft Technology Licensing, Llc Feature-augmented neural networks and applications of same
US9728184B2 (en) * 2013-06-18 2017-08-08 Microsoft Technology Licensing, Llc Restructuring deep neural network acoustic models
US9620108B2 (en) * 2013-12-10 2017-04-11 Google Inc. Processing acoustic sequences using long short-term memory (LSTM) neural networks that include recurrent projection layers
US9552526B2 (en) * 2013-12-19 2017-01-24 University Of Memphis Research Foundation Image processing using cellular simultaneous recurrent network
US9721202B2 (en) * 2014-02-21 2017-08-01 Adobe Systems Incorporated Non-negative matrix factorization regularized by recurrent neural networks for audio processing
US9324321B2 (en) * 2014-03-07 2016-04-26 Microsoft Technology Licensing, Llc Low-footprint adaptation and personalization for a deep neural network
US11256982B2 (en) * 2014-07-18 2022-02-22 University Of Southern California Noise-enhanced convolutional neural networks
US20160035344A1 (en) * 2014-08-04 2016-02-04 Google Inc. Identifying the language of a spoken utterance
US10783900B2 (en) * 2014-10-03 2020-09-22 Google Llc Convolutional, long short-term memory, fully connected deep neural networks
US10229356B1 (en) * 2014-12-23 2019-03-12 Amazon Technologies, Inc. Error tolerant neural network model compression
US10223635B2 (en) * 2015-01-22 2019-03-05 Qualcomm Incorporated Model compression and fine-tuning
CN104598972A (zh) * 2015-01-22 2015-05-06 清华大学 一种大规模数据回归神经网络快速训练方法
CN104700828B (zh) * 2015-03-19 2018-01-12 清华大学 基于选择性注意原理的深度长短期记忆循环神经网络声学模型的构建方法
US10515301B2 (en) * 2015-04-17 2019-12-24 Microsoft Technology Licensing, Llc Small-footprint deep neural network
US20160328644A1 (en) * 2015-05-08 2016-11-10 Qualcomm Incorporated Adaptive selection of artificial neural networks
US10091140B2 (en) * 2015-05-31 2018-10-02 Microsoft Technology Licensing, Llc Context-sensitive generation of conversational responses
US20160350653A1 (en) * 2015-06-01 2016-12-01 Salesforce.Com, Inc. Dynamic Memory Network
US10515307B2 (en) * 2015-06-05 2019-12-24 Google Llc Compressed recurrent neural network models
GB201511887D0 (en) * 2015-07-07 2015-08-19 Touchtype Ltd Improved artificial neural network for language modelling and prediction
CN105184369A (zh) * 2015-09-08 2015-12-23 杭州朗和科技有限公司 用于深度学习模型的矩阵压缩方法和装置
US10217018B2 (en) * 2015-09-15 2019-02-26 Mitsubishi Electric Research Laboratories, Inc. System and method for processing images using online tensor robust principal component analysis
US20170083623A1 (en) * 2015-09-21 2017-03-23 Qualcomm Incorporated Semantic multisensory embeddings for video search by text
US10366158B2 (en) * 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US10395118B2 (en) * 2015-10-29 2019-08-27 Baidu Usa Llc Systems and methods for video paragraph captioning using hierarchical recurrent neural networks
US9807473B2 (en) * 2015-11-20 2017-10-31 Microsoft Technology Licensing, Llc Jointly modeling embedding and translation to bridge video and language
US10319374B2 (en) * 2015-11-25 2019-06-11 Baidu USA, LLC Deployed end-to-end speech recognition
WO2017095948A1 (en) * 2015-11-30 2017-06-08 Pilot Ai Labs, Inc. Improved general object detection using neural networks
US10832120B2 (en) * 2015-12-11 2020-11-10 Baidu Usa Llc Systems and methods for a multi-core optimized recurrent neural network
US10824941B2 (en) * 2015-12-23 2020-11-03 The Toronto-Dominion Bank End-to-end deep collaborative filtering
US10482380B2 (en) * 2015-12-30 2019-11-19 Amazon Technologies, Inc. Conditional parallel processing in fully-connected neural networks
US10515312B1 (en) * 2015-12-30 2019-12-24 Amazon Technologies, Inc. Neural network model compaction using selective unit removal

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150170020A1 (en) * 2013-12-13 2015-06-18 Amazon Technologies, Inc. Reducing dynamic range of low-rank decomposition matrices

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
A Recurrent Probabilistic Neural Network with Dimensionality Reduction Based on Time-seris Discriminant Component Analysis. Hayashi etl al. 2015. *
Compression of Fully-Connected Layer in Neural Network by Kronecker product. Zhou et al. 2015. *
LONG SHORT-TERM MEMORY BASED RECURRENT NEURAL NETWORK ARCHITECTURES FOR LARGE VOCABULARY SPEECH RECOGNITION. Hasim Sak et al. 2014. *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11416735B2 (en) 2018-12-27 2022-08-16 Micron Technology, Inc. Neural networks and systems for decoding encoded data
US11599773B2 (en) 2018-12-27 2023-03-07 Micron Technology, Inc. Neural networks and systems for decoding encoded data
WO2021096641A1 (en) * 2019-11-13 2021-05-20 Micron Technology, Inc. Recurrent neural networks and systems for decoding encoded data
US11424764B2 (en) 2019-11-13 2022-08-23 Micron Technology, Inc. Recurrent neural networks and systems for decoding encoded data
WO2021225256A1 (ko) * 2020-05-08 2021-11-11 삼성전자주식회사 전자 장치 및 이의 제어 방법
US11563449B2 (en) 2021-04-27 2023-01-24 Micron Technology, Inc. Systems for error reduction of encoded data using neural networks
US11973513B2 (en) 2021-04-27 2024-04-30 Micron Technology, Inc. Decoders and systems for decoding encoded data using neural networks
US11755408B2 (en) 2021-10-07 2023-09-12 Micron Technology, Inc. Systems for estimating bit error rate (BER) of encoded data using neural networks

Also Published As

Publication number Publication date
US10878319B2 (en) 2020-12-29
US20210089916A1 (en) 2021-03-25
DE102016125918A1 (de) 2017-08-03
US20170220925A1 (en) 2017-08-03
CN107038476A (zh) 2017-08-11
JP2019509539A (ja) 2019-04-04
KR102100977B1 (ko) 2020-04-14
EP3374932A1 (en) 2018-09-19
EP3374932B1 (en) 2022-03-16
US11948062B2 (en) 2024-04-02
DE202016008253U1 (de) 2017-05-26
JP6706326B2 (ja) 2020-06-03
WO2017136070A1 (en) 2017-08-10

Similar Documents

Publication Publication Date Title
KR20180084988A (ko) 압축된 순환 신경망 모델
NO2022009I1 (no) Lumasiran, optionally in the form of a salt
DK3407997T3 (da) Legetøj
DK3273703T3 (da) Modulært høreapparat
GB201611857D0 (en) An artificial neural network
GB201607713D0 (en) Convolutional neural network
DK3212255T3 (da) Selvorienterende sprøjte og sprøjtegrænseflade
DK3092043T3 (da) Legetøj
DK3018977T3 (da) Lygtenetværk
UA32259S (uk) Розприскувач
CL2018000957S1 (es) Un respirador
DK3426681T3 (da) Åbne hla-b57-konformere
FI20145222A (fi) Nyrkkeilyharjoitteluapuväline
FR3029422B1 (fr) Microbille d'hydrogel
FR3018718B1 (fr) Modele de prediction de demoulabilite
UA37086S (uk) Іграшка
TH178800B (th) ของเล่น
FI11845U1 (fi) Hengityssuoja
IT201700103032A1 (it) Moschettone
ES1184135Y (es) Muñeco
TH186793B (th) ของเล่น
IT201700049064A1 (it) Filtro pre-compressione
UA35404S (uk) Іграшка «автомобіль-кран»
FR3050084B3 (fr) Talkie-walkie jouet
UA33589S (uk) Оприскувач

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E701 Decision to grant or registration of patent right
GRNT Written decision to grant