CN116547643A - 用于具有工作负载平衡的激活稀疏性的卷积的方法和系统 - Google Patents
用于具有工作负载平衡的激活稀疏性的卷积的方法和系统 Download PDFInfo
- Publication number
- CN116547643A CN116547643A CN202180075198.8A CN202180075198A CN116547643A CN 116547643 A CN116547643 A CN 116547643A CN 202180075198 A CN202180075198 A CN 202180075198A CN 116547643 A CN116547643 A CN 116547643A
- Authority
- CN
- China
- Prior art keywords
- output values
- tensors
- processors
- output
- tensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 91
- 238000013138 pruning Methods 0.000 claims abstract description 33
- 238000003860 storage Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims description 47
- 239000000872 buffer Substances 0.000 claims description 32
- 238000013528 artificial neural network Methods 0.000 claims description 28
- 230000015654 memory Effects 0.000 claims description 21
- 241001442055 Vipera berus Species 0.000 claims description 13
- 239000013598 vector Substances 0.000 claims description 13
- 238000009825 accumulation Methods 0.000 claims description 6
- 239000004973 liquid crystal related substance Substances 0.000 claims 2
- 238000004590 computer program Methods 0.000 abstract description 2
- 238000001994 activation Methods 0.000 description 43
- 230000004913 activation Effects 0.000 description 42
- 230000008569 process Effects 0.000 description 41
- 230000006870 function Effects 0.000 description 17
- 239000011159 matrix material Substances 0.000 description 15
- 238000004422 calculation algorithm Methods 0.000 description 12
- 238000013527 convolutional neural network Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000011176 pooling Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 238000007792 addition Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 101100537629 Caenorhabditis elegans top-2 gene Proteins 0.000 description 3
- 101150107801 Top2a gene Proteins 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 206010028980 Neoplasm Diseases 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
- G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
- G06F7/544—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices for evaluating functions by calculation
- G06F7/5443—Sum of products
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0495—Quantised networks; Sparse networks; Compressed networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2207/00—Indexing scheme relating to methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F2207/38—Indexing scheme relating to groups G06F7/38 - G06F7/575
- G06F2207/48—Indexing scheme relating to groups G06F7/48 - G06F7/575
- G06F2207/4802—Special implementations
- G06F2207/4818—Threshold devices
- G06F2207/4824—Neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Mathematical Optimization (AREA)
- Computational Mathematics (AREA)
- Pure & Applied Mathematics (AREA)
- Neurology (AREA)
- Mathematical Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Bioinformatics & Computational Biology (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Complex Calculations (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17/091,216 US20220147826A1 (en) | 2020-11-06 | 2020-11-06 | Method and system for convolution with workload-balanced activation sparsity |
| US17/091,216 | 2020-11-06 | ||
| PCT/CN2021/129141 WO2022095984A1 (en) | 2020-11-06 | 2021-11-05 | Method and system for convolution with workload-balanced activation sparsity |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116547643A true CN116547643A (zh) | 2023-08-04 |
Family
ID=81454090
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202180075198.8A Pending CN116547643A (zh) | 2020-11-06 | 2021-11-05 | 用于具有工作负载平衡的激活稀疏性的卷积的方法和系统 |
Country Status (7)
| Country | Link |
|---|---|
| US (1) | US20220147826A1 (ja) |
| EP (1) | EP4226286A4 (ja) |
| JP (1) | JP2024502225A (ja) |
| KR (1) | KR20230104235A (ja) |
| CN (1) | CN116547643A (ja) |
| TW (2) | TW202328986A (ja) |
| WO (1) | WO2022095984A1 (ja) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022016257A1 (en) * | 2020-07-21 | 2022-01-27 | The Governing Council Of The University Of Toronto | System and method for using sparsity to accelerate deep learning networks |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11055063B2 (en) * | 2016-05-02 | 2021-07-06 | Marvell Asia Pte, Ltd. | Systems and methods for deep learning processor |
| US11544545B2 (en) * | 2017-04-04 | 2023-01-03 | Hailo Technologies Ltd. | Structured activation based sparsity in an artificial neural network |
| US11275996B2 (en) * | 2017-06-21 | 2022-03-15 | Arm Ltd. | Systems and devices for formatting neural network parameters |
| US20190392287A1 (en) * | 2018-06-22 | 2019-12-26 | Samsung Electronics Co., Ltd. | Neural processor |
| CN111160516B (zh) * | 2018-11-07 | 2023-09-05 | 杭州海康威视数字技术股份有限公司 | 一种深度神经网络的卷积层稀疏化方法及装置 |
| CN109948794A (zh) * | 2019-02-28 | 2019-06-28 | 清华大学 | 神经网络结构化剪枝方法、剪枝装置和电子设备 |
| WO2020190772A1 (en) * | 2019-03-15 | 2020-09-24 | Futurewei Technologies, Inc. | Neural network model compression and optimization |
| US11763156B2 (en) * | 2019-11-15 | 2023-09-19 | Microsoft Technology Licensing, Llc | Neural network compression based on bank-balanced sparsity |
| US20200134417A1 (en) * | 2019-12-24 | 2020-04-30 | Intel Corporation | Configurable processor element arrays for implementing convolutional neural networks |
| US20220101118A1 (en) * | 2020-09-30 | 2022-03-31 | Moffett Technologies Co., Limited | Bank-balanced-sparse activation feature maps for neural network models |
-
2020
- 2020-11-06 US US17/091,216 patent/US20220147826A1/en active Pending
-
2021
- 2021-11-05 EP EP21888676.0A patent/EP4226286A4/en active Pending
- 2021-11-05 CN CN202180075198.8A patent/CN116547643A/zh active Pending
- 2021-11-05 JP JP2023527417A patent/JP2024502225A/ja active Pending
- 2021-11-05 WO PCT/CN2021/129141 patent/WO2022095984A1/en active Application Filing
- 2021-11-05 KR KR1020237018879A patent/KR20230104235A/ko unknown
- 2021-11-05 TW TW112107790A patent/TW202328986A/zh unknown
- 2021-11-05 TW TW110141250A patent/TWI804041B/zh active
Also Published As
| Publication number | Publication date |
|---|---|
| EP4226286A4 (en) | 2024-04-10 |
| US20220147826A1 (en) | 2022-05-12 |
| TW202328986A (zh) | 2023-07-16 |
| JP2024502225A (ja) | 2024-01-18 |
| KR20230104235A (ko) | 2023-07-07 |
| TWI804041B (zh) | 2023-06-01 |
| EP4226286A1 (en) | 2023-08-16 |
| TW202230228A (zh) | 2022-08-01 |
| WO2022095984A1 (en) | 2022-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11144823B1 (en) | Method and system for hierarchical weight-sparse convolution processing | |
| Sze et al. | Efficient processing of deep neural networks: A tutorial and survey | |
| WO2022002157A1 (en) | Method and system for balanced-weight sparse convolution processing | |
| CN113469354A (zh) | 受存储器限制的神经网络训练 | |
| JP7235836B2 (ja) | クラスタ接続ニューラルネットワーク | |
| WO2022095984A1 (en) | Method and system for convolution with workload-balanced activation sparsity | |
| US11636569B1 (en) | Matrix transpose hardware acceleration | |
| WO2021248433A1 (en) | Method and system for dual-sparse convolution processing and parallelization | |
| TWI813414B (zh) | 用於最佳化神經網路訓練之電腦實施之方法、系統及非暫時性電腦可讀取儲存媒體 | |
| CN116935195A (zh) | 基于Spark和FFT卷积的并行DCNN场景匹配识别方法 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |