CN111226233A - 促进神经网络效率 - Google Patents

促进神经网络效率 Download PDF

Info

Publication number
CN111226233A
CN111226233A CN201880067753.0A CN201880067753A CN111226233A CN 111226233 A CN111226233 A CN 111226233A CN 201880067753 A CN201880067753 A CN 201880067753A CN 111226233 A CN111226233 A CN 111226233A
Authority
CN
China
Prior art keywords
activation function
computer
training
component
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201880067753.0A
Other languages
English (en)
Chinese (zh)
Inventor
王洲
崔正旭
K·高帕拉克里斯南
S·温卡塔拉玛尼
C·萨卡尔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of CN111226233A publication Critical patent/CN111226233A/zh
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Feedback Control In General (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Image Analysis (AREA)
CN201880067753.0A 2017-10-24 2018-10-04 促进神经网络效率 Pending CN111226233A (zh)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US15/792,733 2017-10-24
US15/792,733 US11195096B2 (en) 2017-10-24 2017-10-24 Facilitating neural network efficiency
PCT/IB2018/057712 WO2019082005A1 (en) 2017-10-24 2018-10-04 FACILITATING THE EFFECTIVENESS OF ARTIFICIAL NEURONIC NETWORK

Publications (1)

Publication Number Publication Date
CN111226233A true CN111226233A (zh) 2020-06-02

Family

ID=66169344

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201880067753.0A Pending CN111226233A (zh) 2017-10-24 2018-10-04 促进神经网络效率

Country Status (6)

Country Link
US (1) US11195096B2 (enExample)
JP (1) JP7163381B2 (enExample)
CN (1) CN111226233A (enExample)
DE (1) DE112018004693T5 (enExample)
GB (1) GB2581728A (enExample)
WO (1) WO2019082005A1 (enExample)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112749803A (zh) * 2021-03-05 2021-05-04 成都启英泰伦科技有限公司 一种神经网络的激活函数计算量化方法
CN113778655A (zh) * 2020-06-09 2021-12-10 北京灵汐科技有限公司 一种网络精度的量化方法及系统
CN114176603A (zh) * 2020-09-14 2022-03-15 伯恩森斯韦伯斯特(以色列)有限责任公司 局部激活时间分析系统
CN114611682A (zh) * 2020-12-08 2022-06-10 国际商业机器公司 一种用于有用神经网络激活函数的vlsi实现的有效方法
CN114692817A (zh) * 2020-12-31 2022-07-01 合肥君正科技有限公司 一种动态调整量化feature clip值的方法

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11195096B2 (en) * 2017-10-24 2021-12-07 International Business Machines Corporation Facilitating neural network efficiency
US20210166106A1 (en) * 2017-12-12 2021-06-03 The Regents Of The University Of California Residual binary neural network
US12536418B2 (en) * 2018-04-27 2026-01-27 Carnegie Mellon University Perturbative neural network
JP7287388B2 (ja) * 2018-05-14 2023-06-06 ソニーグループ株式会社 情報処理装置および情報処理方法
US11763133B2 (en) * 2018-08-31 2023-09-19 Servicenow Canada Inc. Data point suitability determination from edge device neural networks
KR102621118B1 (ko) * 2018-11-01 2024-01-04 삼성전자주식회사 영상 적응적 양자화 테이블을 이용한 영상의 부호화 장치 및 방법
US20200302276A1 (en) * 2019-03-20 2020-09-24 Gyrfalcon Technology Inc. Artificial intelligence semiconductor chip having weights of variable compression ratio
US20220284300A1 (en) * 2019-09-19 2022-09-08 Intel Corporation Techniques to tune scale parameter for activations in binary neural networks
JP7419035B2 (ja) * 2019-11-22 2024-01-22 キヤノン株式会社 学習モデル管理システム、学習モデル管理方法、およびプログラム
US20210174214A1 (en) * 2019-12-10 2021-06-10 The Mathworks, Inc. Systems and methods for quantizing a neural network
US11935271B2 (en) * 2020-01-10 2024-03-19 Tencent America LLC Neural network model compression with selective structured weight unification
US11823054B2 (en) 2020-02-20 2023-11-21 International Business Machines Corporation Learned step size quantization
WO2021211099A1 (en) * 2020-04-14 2021-10-21 Google Llc Efficient binary representations from neural networks
CN113762452B (zh) * 2020-06-04 2024-01-02 合肥君正科技有限公司 一种量化prelu激活函数的方法
WO2022216109A1 (en) * 2021-04-09 2022-10-13 Samsung Electronics Co., Ltd. Method and electronic device for quantizing dnn model
JP7700577B2 (ja) 2021-08-25 2025-07-01 富士通株式会社 閾値決定プログラム、閾値決定方法および閾値決定装置
KR102650510B1 (ko) * 2022-10-28 2024-03-22 한국전자기술연구원 영상의 노이즈 제거 방법 및 장치

Family Cites Families (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR950012359B1 (ko) * 1992-08-28 1995-10-17 엘지전자주식회사 신경회로망 구조와 학습방법
JPH1063633A (ja) * 1996-08-26 1998-03-06 Denso Corp ニューラルネットワークの演算装置及び車両用空調装置
US7149262B1 (en) * 2000-07-06 2006-12-12 The Trustees Of Columbia University In The City Of New York Method and apparatus for enhancing data resolution
HUP0301368A3 (en) * 2003-05-20 2005-09-28 Amt Advanced Multimedia Techno Method and equipment for compressing motion picture data
US20150120627A1 (en) 2013-10-29 2015-04-30 Qualcomm Incorporated Causal saliency time inference
US20150269480A1 (en) 2014-03-21 2015-09-24 Qualcomm Incorporated Implementing a neural-network processor
US10417525B2 (en) * 2014-09-22 2019-09-17 Samsung Electronics Co., Ltd. Object recognition with reduced neural network weight precision
US10373050B2 (en) 2015-05-08 2019-08-06 Qualcomm Incorporated Fixed point neural network based on floating point neural network quantization
US20170032247A1 (en) * 2015-07-31 2017-02-02 Qualcomm Incorporated Media classification
US11029949B2 (en) * 2015-10-08 2021-06-08 Shanghai Zhaoxin Semiconductor Co., Ltd. Neural network unit
KR102565273B1 (ko) 2016-01-26 2023-08-09 삼성전자주식회사 뉴럴 네트워크에 기초한 인식 장치 및 뉴럴 네트워크의 학습 방법
US10831444B2 (en) * 2016-04-04 2020-11-10 Technion Research & Development Foundation Limited Quantized neural network training and inference
US10621486B2 (en) * 2016-08-12 2020-04-14 Beijing Deephi Intelligent Technology Co., Ltd. Method for optimizing an artificial neural network (ANN)
US11003985B2 (en) * 2016-11-07 2021-05-11 Electronics And Telecommunications Research Institute Convolutional neural network system and operation method thereof
US10373049B2 (en) * 2016-12-20 2019-08-06 Google Llc Generating an output for a neural network output layer
KR102457463B1 (ko) * 2017-01-16 2022-10-21 한국전자통신연구원 희소 파라미터를 사용하는 압축 신경망 시스템 및 그것의 설계 방법
CN107122825A (zh) 2017-03-09 2017-09-01 华南理工大学 一种神经网络模型的激活函数生成方法
US10127495B1 (en) * 2017-04-14 2018-11-13 Rohan Bopardikar Reducing the size of a neural network through reduction of the weight matrices
CN107229942B (zh) 2017-04-16 2021-03-30 北京工业大学 一种基于多个分类器的卷积神经网络分类方法
US20180336469A1 (en) * 2017-05-18 2018-11-22 Qualcomm Incorporated Sigma-delta position derivative networks
EP3657398B1 (en) * 2017-05-23 2025-10-08 Shanghai Cambricon Information Technology Co., Ltd Processing method and accelerating device
KR102526650B1 (ko) * 2017-05-25 2023-04-27 삼성전자주식회사 뉴럴 네트워크에서 데이터를 양자화하는 방법 및 장치
US10878273B2 (en) * 2017-07-06 2020-12-29 Texas Instruments Incorporated Dynamic quantization for deep neural network inference system and method
US10728553B2 (en) * 2017-07-11 2020-07-28 Sony Corporation Visual quality preserving quantization parameter prediction with deep neural network
KR102601604B1 (ko) * 2017-08-04 2023-11-13 삼성전자주식회사 뉴럴 네트워크의 파라미터들을 양자화하는 방법 및 장치
US10839286B2 (en) * 2017-09-14 2020-11-17 Xilinx, Inc. System and method for implementing neural networks in integrated circuits
KR102728799B1 (ko) * 2017-09-25 2024-11-11 삼성전자주식회사 인공 신경망의 양자화 방법 및 장치
US11195096B2 (en) * 2017-10-24 2021-12-07 International Business Machines Corporation Facilitating neural network efficiency
US11132605B2 (en) * 2017-11-20 2021-09-28 International Business Machines Corporation Cardinal sine as an activation function for universal classifier training data
US11295208B2 (en) * 2017-12-04 2022-04-05 International Business Machines Corporation Robust gradient weight compression schemes for deep learning applications
US11551077B2 (en) * 2018-06-13 2023-01-10 International Business Machines Corporation Statistics-aware weight quantization
US20200226459A1 (en) * 2019-01-11 2020-07-16 International Business Machines Corporation Adversarial input identification using reduced precision deep neural networks
US11551054B2 (en) * 2019-08-27 2023-01-10 International Business Machines Corporation System-aware selective quantization for performance optimized distributed deep learning
US12175359B2 (en) * 2019-09-03 2024-12-24 International Business Machines Corporation Machine learning hardware having reduced precision parameter components for efficient parameter update
US20210125063A1 (en) * 2019-10-23 2021-04-29 Electronics And Telecommunications Research Institute Apparatus and method for generating binary neural network

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113778655A (zh) * 2020-06-09 2021-12-10 北京灵汐科技有限公司 一种网络精度的量化方法及系统
CN114176603A (zh) * 2020-09-14 2022-03-15 伯恩森斯韦伯斯特(以色列)有限责任公司 局部激活时间分析系统
CN114611682A (zh) * 2020-12-08 2022-06-10 国际商业机器公司 一种用于有用神经网络激活函数的vlsi实现的有效方法
US12400112B2 (en) 2020-12-08 2025-08-26 International Business Machines Corporation Efficient method for VLSI implementation of useful neural network activation functions
CN114692817A (zh) * 2020-12-31 2022-07-01 合肥君正科技有限公司 一种动态调整量化feature clip值的方法
CN112749803A (zh) * 2021-03-05 2021-05-04 成都启英泰伦科技有限公司 一种神经网络的激活函数计算量化方法
CN112749803B (zh) * 2021-03-05 2023-05-30 成都启英泰伦科技有限公司 一种神经网络的激活函数计算量化方法

Also Published As

Publication number Publication date
US11195096B2 (en) 2021-12-07
DE112018004693T5 (de) 2020-06-18
WO2019082005A1 (en) 2019-05-02
US20190122116A1 (en) 2019-04-25
GB202006969D0 (en) 2020-06-24
JP2021500654A (ja) 2021-01-07
GB2581728A (en) 2020-08-26
JP7163381B2 (ja) 2022-10-31

Similar Documents

Publication Publication Date Title
CN111226233A (zh) 促进神经网络效率
US12530876B2 (en) Systems and methods for progressive learning for machine-learned models to optimize training speed
CN110598838B (zh) 统计感知权重量化
US10643131B1 (en) Training variational autoencoders to generate disentangled latent factors
US20200410384A1 (en) Hybrid quantum-classical generative models for learning data distributions
US11334671B2 (en) Adding adversarial robustness to trained machine learning models
US20240104352A1 (en) Contrastive Learning and Masked Modeling for End-To-End Self-Supervised Pre-Training
EP4397044A1 (en) Vector-quantized image modeling
US11468313B1 (en) Systems and methods for quantizing neural networks via periodic regularization functions
US11562235B2 (en) Activation function computation for neural networks
US20250069382A1 (en) Machine Learning Models Featuring Resolution-Flexible Multi-Axis Attention Blocks
US20200311541A1 (en) Metric value calculation for continuous learning system
US11836220B2 (en) Updating of statistical sets for decentralized distributed training of a machine learning model
US20220067505A1 (en) Certifiably Robust Interpretation
US9536206B2 (en) Method and apparatus for improving resilience in customized program learning network computational environments
CN110728350A (zh) 用于机器学习模型的量化
Dai et al. The state of the art in implementing machine learning for mobile apps: A survey
US20250356210A1 (en) Calibrated Distillation
US20250371413A1 (en) Tied Preference Optimization for Sequence Processing Models
Seo RAQ-VAE: Rate-adaptive vector-quantized variational autoencoder
US11514326B2 (en) Drift regularization to counteract variation in drift coefficients for analog accelerators
US20250217938A1 (en) Weighting Functions and Adaptive Noise Schedule for Training Noise-Based Machine-Learned Models
WO2025166361A1 (en) Mutual alignment vector quantization
US20190258928A1 (en) Artificial neural network
Seo et al. Rate-adaptive quantization: A multi-rate codebook adaptation for vector quantization-based generative models

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200602

RJ01 Rejection of invention patent application after publication