WO2021115262A1 - 脉冲卷积神经网络算法、集成电路、运算装置及存储介质 - Google Patents
脉冲卷积神经网络算法、集成电路、运算装置及存储介质 Download PDFInfo
- Publication number
- WO2021115262A1 WO2021115262A1 PCT/CN2020/134558 CN2020134558W WO2021115262A1 WO 2021115262 A1 WO2021115262 A1 WO 2021115262A1 CN 2020134558 W CN2020134558 W CN 2020134558W WO 2021115262 A1 WO2021115262 A1 WO 2021115262A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- layer
- storage
- calculation
- neural network
- input
- Prior art date
Links
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 135
- 238000003860 storage Methods 0.000 title claims abstract description 130
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 54
- 238000004364 calculation method Methods 0.000 claims description 279
- 210000002569 neuron Anatomy 0.000 claims description 97
- 238000011176 pooling Methods 0.000 claims description 95
- 238000000034 method Methods 0.000 claims description 70
- 230000008569 process Effects 0.000 claims description 46
- 238000005192 partition Methods 0.000 claims description 39
- 238000012549 training Methods 0.000 claims description 23
- 239000002184 metal Substances 0.000 claims description 22
- 229910052751 metal Inorganic materials 0.000 claims description 22
- 239000012491 analyte Substances 0.000 claims description 20
- 230000010354 integration Effects 0.000 claims description 18
- 239000004065 semiconductor Substances 0.000 claims description 16
- 238000009825 accumulation Methods 0.000 claims description 15
- 239000000969 carrier Substances 0.000 claims description 14
- 238000013528 artificial neural network Methods 0.000 claims description 13
- 238000004458 analytical method Methods 0.000 claims description 12
- 230000001186 cumulative effect Effects 0.000 claims description 11
- 238000010606 normalization Methods 0.000 claims description 11
- 238000005468 ion implantation Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 9
- 238000000151 deposition Methods 0.000 claims description 8
- 239000003990 capacitor Substances 0.000 claims description 7
- 238000007667 floating Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 230000003647 oxidation Effects 0.000 claims description 6
- 238000007254 oxidation reaction Methods 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 5
- 238000000844 transformation Methods 0.000 claims description 4
- 230000008021 deposition Effects 0.000 claims description 3
- 238000005137 deposition process Methods 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 230000003139 buffering effect Effects 0.000 claims description 2
- 238000003786 synthesis reaction Methods 0.000 claims description 2
- 238000005265 energy consumption Methods 0.000 abstract description 2
- 238000013473 artificial intelligence Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 47
- 239000000758 substrate Substances 0.000 description 26
- 230000006870 function Effects 0.000 description 16
- 230000008878 coupling Effects 0.000 description 13
- 238000010168 coupling process Methods 0.000 description 13
- 238000005859 coupling reaction Methods 0.000 description 13
- 239000000872 buffer Substances 0.000 description 12
- 238000002955 isolation Methods 0.000 description 11
- 238000012937 correction Methods 0.000 description 10
- 230000005540 biological transmission Effects 0.000 description 9
- 230000005693 optoelectronics Effects 0.000 description 8
- 239000000243 solution Substances 0.000 description 7
- 238000013139 quantization Methods 0.000 description 6
- 210000005056 cell body Anatomy 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- VZSRBBMJRBPUNF-UHFFFAOYSA-N 2-(2,3-dihydro-1H-inden-2-ylamino)-N-[3-oxo-3-(2,4,6,7-tetrahydrotriazolo[4,5-c]pyridin-5-yl)propyl]pyrimidine-5-carboxamide Chemical compound C1C(CC2=CC=CC=C12)NC1=NC=C(C=N1)C(=O)NCCC(N1CC2=C(CC1)NN=N2)=O VZSRBBMJRBPUNF-UHFFFAOYSA-N 0.000 description 3
- WZFUQSJFWNHZHM-UHFFFAOYSA-N 2-[4-[2-(2,3-dihydro-1H-inden-2-ylamino)pyrimidin-5-yl]piperazin-1-yl]-1-(2,4,6,7-tetrahydrotriazolo[4,5-c]pyridin-5-yl)ethanone Chemical compound C1C(CC2=CC=CC=C12)NC1=NC=C(C=N1)N1CCN(CC1)CC(=O)N1CC2=C(CC1)NN=N2 WZFUQSJFWNHZHM-UHFFFAOYSA-N 0.000 description 3
- 210000004556 brain Anatomy 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000009795 derivation Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000000717 retained effect Effects 0.000 description 3
- 230000000630 rising effect Effects 0.000 description 3
- IHCCLXNEEPMSIO-UHFFFAOYSA-N 2-[4-[2-(2,3-dihydro-1H-inden-2-ylamino)pyrimidin-5-yl]piperidin-1-yl]-1-(2,4,6,7-tetrahydrotriazolo[4,5-c]pyridin-5-yl)ethanone Chemical compound C1C(CC2=CC=CC=C12)NC1=NC=C(C=N1)C1CCN(CC1)CC(=O)N1CC2=C(CC1)NN=N2 IHCCLXNEEPMSIO-UHFFFAOYSA-N 0.000 description 2
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 2
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 2
- 210000004027 cell Anatomy 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000002784 hot electron Substances 0.000 description 2
- 238000002347 injection Methods 0.000 description 2
- 239000007924 injection Substances 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000001208 nuclear magnetic resonance pulse sequence Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 229910052710 silicon Inorganic materials 0.000 description 2
- 239000010703 silicon Substances 0.000 description 2
- 210000000225 synapse Anatomy 0.000 description 2
- 230000000946 synaptic effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- OHVLMTFVQDZYHP-UHFFFAOYSA-N 1-(2,4,6,7-tetrahydrotriazolo[4,5-c]pyridin-5-yl)-2-[4-[2-[[3-(trifluoromethoxy)phenyl]methylamino]pyrimidin-5-yl]piperazin-1-yl]ethanone Chemical compound N1N=NC=2CN(CCC=21)C(CN1CCN(CC1)C=1C=NC(=NC=1)NCC1=CC(=CC=C1)OC(F)(F)F)=O OHVLMTFVQDZYHP-UHFFFAOYSA-N 0.000 description 1
- NIPNSKYNPDTRPC-UHFFFAOYSA-N N-[2-oxo-2-(2,4,6,7-tetrahydrotriazolo[4,5-c]pyridin-5-yl)ethyl]-2-[[3-(trifluoromethoxy)phenyl]methylamino]pyrimidine-5-carboxamide Chemical compound O=C(CNC(=O)C=1C=NC(=NC=1)NCC1=CC(=CC=C1)OC(F)(F)F)N1CC2=C(CC1)NN=N2 NIPNSKYNPDTRPC-UHFFFAOYSA-N 0.000 description 1
- AFCARXCZXQIEQB-UHFFFAOYSA-N N-[3-oxo-3-(2,4,6,7-tetrahydrotriazolo[4,5-c]pyridin-5-yl)propyl]-2-[[3-(trifluoromethoxy)phenyl]methylamino]pyrimidine-5-carboxamide Chemical compound O=C(CCNC(=O)C=1C=NC(=NC=1)NCC1=CC(=CC=C1)OC(F)(F)F)N1CC2=C(CC1)NN=N2 AFCARXCZXQIEQB-UHFFFAOYSA-N 0.000 description 1
- 206010034960 Photophobia Diseases 0.000 description 1
- -1 WO x Chemical class 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
- 238000005530 etching Methods 0.000 description 1
- 238000011423 initialization method Methods 0.000 description 1
- 208000013469 light sensitivity Diseases 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 150000004767 nitrides Chemical class 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 229910021420 polycrystalline silicon Inorganic materials 0.000 description 1
- 229920005591 polysilicon Polymers 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000001568 sexual effect Effects 0.000 description 1
- 235000012239 silicon dioxide Nutrition 0.000 description 1
- 239000000377 silicon dioxide Substances 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012421 spiking Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
Definitions
- the invention relates to a pulse convolution neural network algorithm, an integrated circuit, an arithmetic device and a storage medium, which can convert a traditional convolution neural network into a pulse neural network, and specifically relates to the field of image classification.
- the storage-calculation integrated unit solves the above problems because it can directly save the data on the computing unit and does not require on-chip cache.
- a large number of analog-to-digital converters are needed to convert the current into a digital signal, which occupies most of the area and power consumption of the entire system.
- the calculation speed of the entire system is also limited by it and cannot be improved.
- the pulse neural network tries to simulate the calculation method of the human brain as much as possible.
- the obvious feature is that the data flows in the network in the form of pulse signals.
- the power consumption is much less than that of the convolutional neural network.
- the impulse convolutional neural network combines the characteristics of the convolutional neural network and the impulse neural network. After some modifications to the convolutional neural network model, the method of training the convolutional neural network can be used to obtain the weights, and the classification accuracy is relative to all The convolutional neural network used has a very small drop. Because the data in the network is in pulse form, the hardware resource consumption is small. At present, the research in this field only stays at the algorithm level, and there is no relevant hardware implementation scheme.
- one input method is that the input pulse is generated by random number generation. According to the law of large numbers, it takes a long time to generate a large number of pulses before it may converge to the original value and get close to the volume. The classification results of the product neural network, and this requires a lot of calculation time, and the calculation efficiency is very low.
- Another input method is that the input is not a pulse, but an analog value. When the circuit is implemented, on the one hand, the accuracy of the input cannot be guaranteed. On the other hand, considering the actual application, the input source is likely to be an image sensor, and the output is equal. For digital signals, compatibility needs to be considered.
- the Batch Normalization (BN) layer is a commonly used layer in the convolutional neural network to optimize the network, which can improve the accuracy of training and reduce the dependence of the training results on the initialization method.
- BN Batch Normalization
- the convolutional layer/fully connected layer will inevitably generate bias.
- the use of bias is avoided, so that the BN layer cannot be added, which brings difficulties to the training of large-scale convolutional neural networks.
- a pulsed convolutional neural network algorithm is proposed.
- the average pooling layer is merged into the next convolutional layer or fully connected layer, and the convolution with bias is supported.
- the calculation of layers and fully connected layers, support for adding BN layers to the network, setting the judgment conditions for the end of calculation, adding auxiliary judgments for special situations, etc. can greatly save the calculation time of the existing pulse convolutional neural network algorithm.
- improve the accuracy of image classification increase the function support of the pulse convolutional neural network algorithm for the bias and BN layer, and adjust the input mode to increase compatibility.
- a pulsed convolutional neural network operation device is proposed.
- the pulsed convolutional neural network is implemented on the storage-calculation unit, the multi-bit digital signal representing the true value of the convolutional neural network is converted
- the analog-to-digital converter is replaced by a current integration and comparison circuit, thereby greatly reducing the area and power consumption.
- mapping method of the convolutional layer and the fully connected layer is completely expanded, that is, all the output results of each layer are calculated at the same time, and are connected to the next layer as the input of the next layer, and each layer of convolutional layer/full
- the weight coefficients of the connection layer are all stored in the storage and calculation integrated unit, so there is no data to be cached during the calculation process, and the calculation speed of the entire system is significantly accelerated.
- the required storage-calculation unit is proportional to the square of the input image size and the number of channels in the convolutional layer, which requires a lot of area. And the calculation speed of this solution is very fast. In the case of a large image, it far exceeds the transmission speed of the input image data, that is, the calculation speed is limited because the data transmission speed cannot keep up.
- a pulse convolutional neural network computing device with a memory is proposed.
- this method will make the theoretical calculation speed much lower than that of the solution that does not save intermediate data, in fact, because of the bottleneck limitation of the data transmission speed, the final speed is also within an acceptable range.
- a pulsed convolutional neural network algorithm based on a storage-calculation integrated unit, which includes at least one storage input terminal, at least one arithmetic input terminal and one output terminal, characterized in that: 1) Copy the weights of the first layer of the pulse convolutional neural network to several copies, the number of copies is at least the number of binary numbers converted into the quantity used to characterize the properties of the analyte and the storage-calculation integrated unit Save the minimum value of the input terminal, and process the weights of the number of copies after the copy, so that the weights after the copy are sequentially reduced by twice in value, and the obtained values are respectively input to the multiple storage and calculation integrated units
- the number of the storage input terminal of the storage and calculation unit is the same as the number of copies; 2)
- the selected quantity that is used to characterize the properties of the analyte is converted into a binary number, and the input Each digit of a binary number, or a value truncated according to the bit width of the system,
- the input quantity corresponds to the input quantity
- the input quantity of the storage input terminal with the larger absolute value corresponds to the input quantity of the higher calculation input terminal
- the quantity of the storage input terminal is The amount of the calculation input terminal is calculated, and the current value obtained at the output terminal represents the result of the multiplication operation of the value of the storage input terminal of the storage and calculation integrated unit and the value of the calculation input terminal.
- the impulse convolutional neural network algorithm is further characterized by: 1) including the operations of the first layer and the operations of other layers, and in any of the layers, in the In addition to the operation of the input terminal and the calculation input terminal, an additional operation accumulation term is added.
- the operation accumulation term is a modified offset value, and the modified offset value is proportional to its original value and then divided by the The cumulative multiplication of the positive thresholds of all layers before the layer, the proportional ratio is related to the layer where the bias is located and the weight scaling ratio of the previous layer; 2) the pulse convolutional neural network algorithm, the storage calculation The output of the integrated unit is continuously accumulated, and when the accumulated sum exceeds a set positive threshold, the accumulated sum is cleared, and an output pulse is released from the input terminal of the corresponding position in the next layer; and when After the accumulated sum is less than a set negative threshold, the accumulated sum is maintained at the negative threshold.
- the impulse convolutional neural network includes a batch normalization layer, and the weights and biases in a convolution layer or a fully connected layer before the batch normalization layer are linearly transformed, wherein The parameters in the linear transformation are obtained in the previous training process.
- a plurality of counters are used to count the number of pulses of each neuron in the last fully connected layer of the pulse convolutional neural network and the time of the earliest pulse appearing.
- the number is the number of said neurons or twice.
- the category value corresponding to the counter that receives the pulse earliest is selected as the final result.
- the output is terminated, and the final classification result is output as the category value corresponding to the maximum value of the counting results of the plurality of counters.
- At least one of average pooling, maximum pooling, convolutional layer and fully connected layer operations is also performed.
- the impulse convolutional neural network algorithm is further characterized by: 1) setting the duration of several clock signals as one analysis period; 2) dividing the object to be analyzed into several Partition; 3) Taking the analysis period as the time unit, analyze the time series signal of a partition one by one, and send the calculation result representing the partition to a memory; 4) Analyze the signal of the next partition, and transfer the signal representing the partition The calculation result is sent to the memory until the completed signals of the multiple partitions jointly meet the analysis conditions of the next layer; 5) the signals of each partition stored in the memory are sent to the next layer for calculation.
- the memory is at least one of a register, an on-chip cache, an off-chip storage, or a cloud storage, or a combination thereof.
- an integrated circuit based on a pulsed convolutional neural network characterized in that the integrated circuit executes the above-mentioned pulsed convolutional neural network algorithm.
- a computer-readable recording medium on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a computer, the computer is caused to execute a pulsed convolutional neural network algorithm, so
- the characteristics of the impulse convolutional neural network algorithm are: 1) copy the weight of the first layer of the impulse convolutional neural network to several copies, the number of copies is at least the binary number converted into the quantity used to characterize the properties of the analyte
- the number of digits and the minimum value of the storage input terminal of the storage-calculation integrated unit, and the weights of the copies after the copy are processed, so that the weights after the copy are reduced by two times in numerical value, and the resulting value is They are respectively input to the storage input terminals of a plurality of the storage-calculation integrated units, and the number of the storage-calculation integrated units is the same as the number of copies; 2) Convert the selected quantity that is used to characterize the properties of the analyte Into a binary number, and input each
- the calculation unit until the processing of the attribute of the analyte is completed; 3) for each basic binary number in the one group that is used to characterize the attribute of the analyte, the storage of each The input quantity of the input terminal corresponds to the input quantity of a calculation input terminal, and the input quantity of the storage input terminal with a larger absolute value corresponds to the input quantity of the higher calculation input terminal in a one-to-one correspondence; 4) The storage and calculation are integrated in each In the unit, the amount of the storage input terminal and the amount of the calculation input terminal are operated, and the current value obtained at the output terminal represents the result of the multiplication operation of the value of the storage input terminal of the integrated unit of storage and calculation and the value of the calculation input terminal.
- the computer-readable recording medium is further characterized in that: 1) the impulse convolutional neural network algorithm includes the operations of the first layer and the operations of other layers, and In any layer, in addition to the operation of the storage input terminal and the calculation input terminal, an operation accumulation item is added.
- the operation accumulation item is a corrected offset value, and the corrected offset value is proportional to it
- the original value is divided by the cumulative multiplication of the positive thresholds of all layers before the layer, and the ratio of the proportional ratio is related to the layer where the bias is located and the weight scaling ratio of the previous layer; 2) the impulse convolutional neural network algorithm , To continuously accumulate the output of the storage and calculation integrated unit, when the accumulation sum exceeds a set positive threshold, the accumulation sum is cleared, and the calculation input terminal at the corresponding position in the next layer is released An output pulse; and when the accumulated sum is less than a set negative threshold, the accumulated sum is maintained at the negative threshold.
- the impulse convolutional neural network includes a batch normalization layer, and the weights and biases in a convolution layer or a fully connected layer before the batch normalization layer are linearly transformed, wherein The parameters in the linear transformation are obtained in the previous training process.
- a plurality of counters are used to count the number of pulses of each neuron in the last fully connected layer of the pulse convolutional neural network and the time of the earliest pulse appearing. It is the number of said neurons or twice.
- the category value corresponding to the counter that receives the pulse earliest is selected as the final result.
- one counter collects significantly more pulses than the other counters, then the output is terminated, and the final classification result is used as the plurality of counters.
- the category value corresponding to the maximum value of the counter count is output.
- At least one of average pooling, maximum pooling, convolutional layer and fully connected layer operations is also performed.
- the impulse convolutional neural network algorithm includes the following: 1) setting the duration of several clock signals as an analysis period; 2) dividing the object to be analyzed into several partitions; 3 ) Using the analysis period as the time unit, analyze the time series signals of a partition one by one, and send the calculation result representing the partition to a memory, and the analyzed signal can be overwritten by subsequent signals; 4) Analyze the next partition Signal, the operation result representing the partition is sent to the memory until the completed signals of multiple partitions jointly meet the analysis conditions of the next layer; 5) the signal of each partition stored in the memory Send to the next layer for calculation.
- the memory is at least one of a register, an on-chip cache, an off-chip storage, or a cloud storage, or a combination thereof.
- an integrated circuit based on a pulsed convolutional neural network includes multiple layers of neurons, each layer of neurons includes multiple neuron components, and each layer of neurons The multiple neurons of are not connected to each other, but are connected to the neurons in the back layer; at least one of the neuron components has at most one digital logic circuit, and the digital logic circuit is used for operation, and the operation includes data distribution, It can also include maximum pooling, clock synchronization, and data buffering; and, each neuron component of the last layer has a counter group to count the number of pulses with high level in the output pulse of the neuron component; where Each neuron includes at least one storage-calculation integrated unit and at least one integration comparison circuit, and the current output ends of the multiple storage-calculation integrated units are connected to each other and collectively connected to the integration comparison circuit; each of the The integral comparison circuit includes at least one integrator and at least one comparator, the integrator is used to accumulate the output of the current output terminal, and the comparator
- each of the storage-calculation integrated units includes at least one storage input terminal and at least one The calculation input terminal and at least one current output terminal, the storage input terminal is set to receive the carrier characterizing the weight issued by the host computer, and the calculation input terminal is set to receive The upper layer of the input pulse carrier; the current output terminal is set to output the carrier as the weighted value and the carrier as the input pulse in the form of current.
- the storage-computer integrated unit is one of a photoelectric computing unit, a memristor, and a flash memory based on semiconductor principles.
- the digital logic circuit is configured to find a number of output signals output from the neuron component of the upper layer of the current pooling layer whose number is the square of the size of the pooling layer. Output the first high-level pulse signal; and the digital logic circuit is also set as a functional device including a multiplexer, so that the high-level pulse signal is maintained after passing through the multiplexer.
- the path corresponding to the high-level pulse signal is opened, and the path is connected to the next convolutional layer or the fully connected layer; at the same time, signals of other paths parallel to the path corresponding to the high-level pulse signal are ignored, or Close the other passages.
- the average pooling operation is merged into the next convolutional layer or fully connected layer, including: 1) a convolutional layer or a fully connected layer, the convolutional layer or a fully connected layer
- the number of storage and calculation integrated units in each neuron component is several times the original size of the corresponding algorithm of the layer, and the multiple is the square of the size of the pooling layer, and each weight in the corresponding algorithm is in the neuron There are several times in the component, the number of times is the square of the size of the pooling layer, 2) Among them, the output pulses output from the neuron component of the previous layer and to be transmitted to the next pooling layer, the number of which is the square of the size of the pooling layer
- the signal is directly used as the calculation input of the storage and calculation integrated unit in the convolutional layer or the fully connected layer, and the storage and calculation integrated unit respectively corresponds to the same weight.
- each of the neuron components includes a neuron and has a register, and the register is used to synchronize the involved data operations in time.
- a pulse convolution neural network operation device for performing pulse convolution neural network operations, including a host computer and the above-mentioned integrated circuit; wherein, the host computer is configured to process and Generate the weights of the first layer, and the process of generating the weights of the first layer includes: generating a set of weights according to an initial weight obtained by training through several linear transformations, and the set of weights includes a plurality of weights. , Where the latter weight value is 1/2 of the previous weight value; and, the host computer sends the set of weight values to the memory in each neuron component of the first layer of the impulse convolutional neural network.
- the storage input terminal in the integrated unit; and the host computer sends the initial weight value to the storage input terminal of the integrated storage unit of the other layers after the first layer after a number of linear transformations.
- the weights of the convolutional layer or the fully connected layer after the pooling layer are also copied in several copies according to the pooling size, and the number of copies is the square of the size of the pooling layer.
- the device is used to analyze the subject matter by partition, and then synthesize the subject matter signals of each partition to form complete subject matter information
- the pulse convolutional neural network computing device It also includes a memory, which is used to store the signals of at least one partition of the subject matter that have been processed step by step, and after all the partition signals are processed, all the partition signals are synthesized, or all the partition signals are processed.
- the partition signal of is sent to another processor for synthesis; the memory is at least one of registers, on-chip cache, off-chip storage, or cloud storage.
- a method for manufacturing the above-mentioned integrated circuit comprising the following steps: 1) Forming a digital logic circuit, an integral comparison circuit, and a medium of a transistor in a storage-calculation unit by thermal oxidation and deposition Layers and gates; the transistors include at least ordinary logic transistors, high-voltage transistors and floating-gate transistors; 2) forming the capacitor in the integral comparison circuit by depositing MIM dielectric layers and depositing metal layers, or thermal oxidation and deposition processes; 3) The source and drain of the transistor in the digital logic circuit, the integral comparison circuit and the storage-calculation integrated unit are formed by ion implantation, as well as the P and N levels of the PN junction; 4) The metal layer process, the metal layer dielectric process And the through hole process forms the metal connection and active area-metal layer and metal layer-metal layer through holes of the overall circuit; 5) Through the process applied to the memristor or flash memory, a CMOS process is generated to
- the purpose of the present invention is at least to convert the data in the convolutional neural network into a time pulse sequence, and replace the analog-to-digital converter with a large power consumption and area by a current integration and comparison circuit, thereby greatly reducing the area and size of the entire system. Power consumption.
- Another purpose of the present invention is to directly connect the output results of each layer of convolutional layer/fully connected layer with the next layer of convolutional layer/fully connected layer, and the weight data can be directly stored in the storage-calculation integrated unit, There is no need for on-chip cache in the entire system, which saves a lot of data transfer process and speeds up calculations.
- the present invention proposes a pulse convolutional neural network computing device with memory. The output result of each layer of convolutional layer/fully connected layer is directly connected to the next layer of convolutional layer/fully connected layer. There are too many storage and calculation integrated units required for connection, and the area is too large. Therefore, storing part of the data through on-chip or off-chip memory and changing time for space will greatly reduce the required hardware resources.
- Fig. 1 is a block diagram of a multi-function area of a computing unit according to an embodiment.
- Fig. 2 is a schematic diagram of the structure of an optoelectronic computing array according to an embodiment.
- Example 3 is a cross-sectional view (a) and a perspective view (b) of the structure of the computing unit of Example 1-1.
- FIG. 4 is a cross-sectional view (a) and a perspective view (b) of the structure of the calculation unit of the embodiment 1-2.
- Fig. 5 is a schematic diagram (a) and a schematic diagram (b) of the multi-function area of the calculation unit of the embodiment 1-3.
- Fig. 6 is a schematic diagram of the structure of an RRAM device according to an embodiment and its three-terminal overview.
- Fig. 7 is a basic cell structure diagram of a flash memory according to an embodiment.
- Fig. 8 is a schematic diagram of the structure of Spiking-Lenet-5 of Example 4-1 (average pooling).
- Fig. 9 is a schematic diagram of the structure of Spiking-Lenet-5 of Example 4-1 (maximum pooling).
- Fig. 10 is a schematic diagram of a neuron composed of an integrated storage-calculation unit of Embodiment 4-1.
- Fig. 11 is a block diagram of the entire system of the embodiment 4-1 (average pooling).
- Fig. 12 is a block diagram of the entire system of the embodiment 4-1 (maximum pooling).
- Fig. 13 is a calculation flowchart (average pooling) of the entire system of the embodiment 4-1.
- Fig. 14 is a calculation flowchart (maximum pooling) of the entire system of the embodiment 4-1.
- Fig. 15 is a schematic diagram of a neuron composed of a storage-calculation integrated unit of the embodiment 4-2 (without registers).
- Fig. 16 is a block diagram of the entire system of the embodiment 4-2 (average pooling, register removal).
- Fig. 17 is a block diagram of the entire system of the embodiment 4-2 (max pooling, register removal).
- FIG. 18 is a schematic diagram of the structure of Spiking-Alexnet in Example 4-3.
- Fig. 19 is a schematic diagram of a neuron composed of an integrated storage-calculation unit in embodiment 4-3.
- Fig. 20 is a block diagram of the entire system of the embodiment 4-3.
- Fig. 21 is a calculation flowchart of the entire system of the embodiment 4-3.
- Fig. 22 is a block diagram of the entire system of the embodiment 4-4.
- Fig. 23 is a block diagram of the entire system of the embodiment 4-5.
- Fig. 24 is a block diagram of the entire system of the embodiment 4-6.
- FIG. 25 is a diagram of the Alexnet network structure of the fifth embodiment.
- Fig. 26 is a diagram of the Spiking-Alexnet network structure of the fifth embodiment (average pooling).
- Fig. 27 is a Spiking-Alexnet network structure diagram of Embodiment 5 (maximum pooling).
- FIG. 28 is a structural diagram of a neuron in Example 5.
- FIG. 28 is a structural diagram of a neuron in Example 5.
- the storage-calculation integrated unit described in the present invention is not specific to a specific device, as long as the storage-calculation integrated unit can store data, and the vector dot multiplication operation can be completed by combining multiple storage-calculation integrated units.
- For each storage and calculation unit there are storage input, calculation input and output.
- the data of the storage input can be stored for a long time.
- the value of the output is proportional to the product of the calculation input and the storage input, and there are multiple storage calculations.
- the output of the integrated unit can be summed.
- the calculation unit in the optoelectronic calculation unit is a multifunctional area structure including three functional areas, as shown in Figure 1, the three functional areas are: carrier control area, coupling area, photo-generated carrier collection area and readout area ,
- the specific functions are as follows:
- Carrier control area Responsible for controlling and modulating the carriers in the optoelectronic computing unit, and as the electrical input port of the computing unit, inputting one of the calculations as the electrical input; or only controlling and modulating the carriers in the computing unit , Input the electrical input through other areas.
- Coupling area Responsible for connecting the photo-generated carrier collection area and the read-out area, so that the photo-generated carriers generated by the incidence of photons act on the carriers in the optoelectronic computing unit to form a calculation relationship.
- Photo-generated carrier collection area and read-out area the collection area is responsible for absorbing incident photons and collect the generated photo-generated carriers, and is used as the optical input port of the calculation unit, and one of the calculations is input as the light input; the read-out area It can be used as the electrical input port of the calculation unit, inputting one of the calculation quantities as the electrical input, and as the output port of the calculation unit, outputting the carriers after the light input and the electrical input are the unit output; or through other The area inputs the electrical input, and the readout area is only used as the output port of the calculation unit, and the carriers that are acted on by the light input and the electrical input are output as the output of the unit.
- the light input is actually the photogenerated carriers stored in the semiconductor device, this carrier can be used in a relatively long period of time (usually on the order of seconds, longer than a few years).
- the light input is the storage input in the storage-calculation integrated unit, and the collection area in the photo-generated carrier collection and readout area is the storage input of the storage-calculation integrated unit; electrical input It does not have the function of long-term storage in the unit, so the point input is the calculation input in the storage-calculation integrated unit, and the photo-generated carrier collection and readout area in the readout area or the carrier control area are integrated storage and calculation
- the calculation input terminal of the unit depends on the specific working mode; the final calculation result of the photoelectric calculation unit is output in the form of current in the readout area of the photogenerated carrier collection and readout area, so the photogenerated carrier collection and readout area
- the read-out area in is the output terminal of the storage-calculation integrated unit.
- FIG. 2 is a schematic diagram of the structure of an optoelectronic computing array, in which: 1 is a light-emitting array, and 2 is a computing array.
- the optoelectronic computing array includes a light emitting array 1 and a computing array 2.
- the light-emitting array 1 is composed of a plurality of light-emitting units arranged periodically
- the computing array 2 is composed of a plurality of computing units that are periodically arranged.
- the calculation unit of this embodiment includes: a control gate as a carrier control region, a charge-coupled layer as a coupling region, and a P-type substrate as a photo-generated carrier collection region and a readout region
- the P-type substrate is divided into a left collection area and a right readout area.
- the right readout area includes shallow trench isolation, N-type source terminals and N-type drain terminals formed by ion implantation.
- the shallow trench isolation is located in the middle of the semiconductor substrate, between the collection area and the readout area.
- the shallow trench isolation is formed by etching and filling silicon dioxide to isolate the electrical signals in the collection area and the readout area.
- the N-type source terminal is located on the side close to the bottom dielectric layer in the readout area, and is formed by doping by ion implantation.
- the N-type drain terminal is located on the other side of the semiconductor substrate, which is close to the bottom dielectric layer and the N-type source terminal, and is also formed by doping method by ion implantation. It should be understood that the left side, right side, upper side, and lower side mentioned in this text only represent that the relative position under observation through the viewing angle shown in the figure changes as the viewing angle changes, and is not understood as a limitation on the specific structure.
- a pulse with a voltage range of negative voltage is applied to the substrate in the collection area, or a pulse with a voltage range of positive voltage is applied to the control gate, so that a depletion layer for photoelectron collection is generated in the substrate of the collection area and passes through
- the read-out area on the right reads the collected photoelectron quantity as the input quantity of the light input terminal.
- Apply a positive voltage to the control gate to form a conductive channel between the N-type source terminal and the N-type drain terminal of the collection area, and then apply a bias pulse between the N-type source terminal and the N-type drain terminal The voltage accelerates the electrons in the conductive channel to form a current between the source and the drain.
- the carriers that form current in the channel between the source and drain are affected by the control gate voltage, the voltage between the source and the drain, and the number of photoelectrons collected in the collection area.
- the output is in the form of output, where the control gate voltage and the voltage between the source and drain can be used as the electrical input of the device, and the number of photoelectrons is the light input of the device.
- the charge-coupled layer in the coupling area is used to connect the collection area and the readout area, so that after the depletion area in the substrate of the collection area begins to collect photoelectrons, the surface potential of the substrate in the collection area will be affected by the number of photoelectrons collected; Connection, so that the surface potential of the semiconductor substrate in the readout area is affected by the surface potential of the semiconductor substrate in the collection area, which in turn affects the current between the source and drain in the readout area, so that the photoelectrons collected in the collection area can be read by judging the current between the source and drain in the readout area Quantity
- the control gate of the carrier control region is used to apply a pulse voltage to it to generate a depletion region for exciting photoelectrons in the readout region of the P-type semiconductor substrate, and it can also be used as an electrical input terminal to input into it One bit of calculation.
- the calculation unit of this embodiment includes: a control gate as a carrier control region, a charge-coupled layer as a coupling region, and a P-type semiconductor substrate as a photo-generated carrier collection region and a readout region.
- the P-type substrate contains N-type source and drain terminals formed by ion implantation.
- the P-type semiconductor substrate can simultaneously undertake the work of light-sensing and reading.
- the N-type source terminal is located on the side close to the bottom dielectric layer in the readout area, and is formed by doping by ion implantation.
- the N-type drain terminal is located on the other side of the semiconductor substrate close to the bottom dielectric layer opposite to the N-type source terminal, and is also formed by doping method by ion implantation.
- a pulse with a voltage range of negative voltage is applied to the P-type semiconductor substrate, and a pulse with a voltage range of positive voltage is applied to the control gate as the carrier control region, so that the P-type substrate generates A depletion layer for photoelectron collection.
- the electrons generated in the depletion area are accelerated by the electric field between the control gate and the P-type substrate, and reach a sufficiently high energy to pass through the P-type substrate.
- the underlying dielectric barrier between the substrate and the charge-coupled layer enters the charge-coupled layer and is stored there. The amount of charge in the charge-coupled layer will affect the threshold when the device is turned on, and then affect the current between the source and drain during readout.
- the control gate voltage and the voltage between the source and drain can be used as the electrical input of the device, and the amount of photoelectrons stored in the charge-coupled layer is the light input of the device.
- the charge-coupled layer in the coupling area is used to store the photoelectrons that enter it and change the threshold value of the device during readout, which in turn affects the current between the source and drain of the readout area, so that by judging the current between the source and drain of the readout area, it can be used to read out and The number of photoelectrons entering the charge-coupled layer.
- the control gate of the carrier control region is used to apply a pulse voltage to it to generate a depletion region for exciting photoelectrons in the readout region of the P-type semiconductor substrate, and it can also be used as an electrical input terminal to input into it One bit of calculation.
- Fig. 5 is a schematic diagram (a) and a schematic diagram (b) of the multi-function area of the calculation unit of the embodiment 1-3.
- the calculation unit of this embodiment includes: a photodiode and a readout tube as a photo-generated carrier collection and readout area, where the photodiode is formed by ion doping and is responsible for light sensitivity.
- the N area of the photodiode is connected to the control gate of the readout tube and the source end of the reset tube through the photoelectron coupling lead as the coupling area, and a positive voltage pulse is applied to the drain end of the readout tube as the driving voltage of the readout current;
- the drain terminal voltage of the reset tube is applied to the photodiode, so that the photodiode as the collection area is in a reverse biased state, resulting in a depletion layer; during exposure, the reset tube is turned off, and the photodiode is electrically isolated. After entering the depletion area of the photodiode, photoelectrons are generated and accumulated in the diode.
- the read-out tube is responsible for reading. A positive pulse voltage is applied to the drain terminal, and the source terminal is connected to the drain terminal of the address selection tube. When reading, the address selection tube is turned on, and a current is generated in the read-out tube. , The drain terminal voltage of the read-out tube and the number of incident photons are affected together.
- the electrons in the read-out tube channel are output in the form of current as the electrons that are combined by the light input and the electrical input.
- the reset tube drain voltage, The voltage at the drain terminal of the readout tube can be used as the electrical input of the device, and the number of electrically incident photons is the light input of the device.
- the photoelectron coupling lead of the coupling area is used to connect the photodiode as the collection area in the photo-generated carrier collection and readout area and the readout tube as the readout area, and apply the N region potential of the photodiode to the readout tube control grid.
- a positive voltage is input through its drain terminal to act on the photodiode.
- the positive voltage will act on the photodiode, causing the photodiode to produce a depletion area and light-sensing. It can also be used as an electrical input terminal to input one of the calculations.
- the address selection tube is used to control the output of the entire arithmetic device as the output current of the output, and can be used for row and column address selection when the photoelectric calculation unit forms an array.
- Memristor is called memristor. This device can be summarized as a special non-volatile type that can switch between "high resistance state” and “low resistance state” and can store the resistance value for a long time. Sexual (NVM) storage devices.
- FIG. 6 is a schematic diagram of the RRAM device structure and its three-terminal overview.
- the device usually consists of two layers of metal electrodes sandwiching a special through hole layer group that can be formed into conductive through holes.
- the through hole layer is mostly composed of metal oxide, such as WO x , TaO x and so on.
- the RRAM device When the RRAM device is in the initial mode, the device is in a high-resistance state.
- a large bias voltage is applied to both ends of the device, the device enters the programming state, and a conductive channel is formed in the special via layer, and continues to maintain this after the voltage is reduced.
- the conductive channel exists and stores the current resistance value until the device enters the erasing state after a large negative bias is applied, and the conductive channel tube section causes the device to return to the initial high resistance state.
- RRAM device as a storage and calculation integrated device, because it has the function of storing resistance value for a long time, so its storage input terminal is the two ends of the device in the programming state; after the resistance input is completed, the device is in a low resistance state and can be in a certain state.
- the voltage range is used as a linear resistance, and the calculation required by the storage and calculation unit can be completed by using the linear resistance range. Therefore, the calculation input terminal is the two ends of the device within the linear resistance range; when there is a linear resistance range
- the bias voltage is applied to both ends of the device, the current flows from one end of the RRAM to the other end, so the end of the device current flowing out at this time is the output end of the storage-calculation integrated device.
- RRAM is usually a two-terminal device, it stores the input terminal, and the input terminal and output terminal are usually the same area in different working modes.
- Flash memory FLASH is currently the most common non-volatile (NVM) storage device, and its basic storage unit is a floating gate device, such as a structure similar to the optoelectronic computing unit described in Embodiment 1-2, or as shown in Figure 7. The structure shown.
- Figure 7 is a basic cell structure diagram of a flash memory. As shown in Figure 7, EG and WL for erasing and selection are added.
- the basic principle is to add a charge storage layer surrounded by an oxide isolation layer between the channel of a normal MOSFET transistor and the control gate, use this isolation storage layer to store charge to store data, and determine the threshold of the transistor To read out the stored charge.
- the isolation layer can be a floating gate made of polysilicon, as shown in FG (floating gate) in Figure 7, or a nitride layer, etc.
- the charge stored in the isolation layer is mostly through channel hot electron injection (CHE) Mechanism to achieve.
- CHE channel hot electron injection
- the flash device is used as a storage-calculation integrated device, because the electric nucleus stored in the isolated charge storage layer can be stored in the device for a long time, so the amount of stored charge is the storage input in the storage-calculation integrated device.
- the storage input terminal is the hot electron injection terminal. This mechanism usually occurs directly below the charge storage layer of the surface channel in the P-type substrate of the flash device, as shown in Figure 7 directly below the FG (floating gate); the flash device reads When output, the channel current of the MOSFET transistor is affected by the source-drain voltage Vds, the control gate voltage Vgs, and the amount of charge stored in the charge storage layer. Therefore, the input terminal can be the control gate of the flash device, as shown in Figure 7.
- CG coupled gate
- WL word line
- source and drain because the data after the final power input and the stored input work together in the form of current flows between the source and drain of the flash, so the flash
- the output terminals of the device as a storage-computing integrated device are the source terminal and the drain terminal.
- any one of the above embodiments is used as an integrated storage-calculation unit to perform pulse convolutional neural network calculations.
- the following specific implementations are available:
- the data set is MNIST as an example, the data set size is 10000*28*28, a total of 10000 sets of test data, the image size is 28*28, the number of channels is 1, the data is a floating point number between 0-1, and the number of classifications is 10. .
- the convolutional neural network takes Lenet-5 as an example.
- the pooling layer can be maximum pooling or average pooling.
- the specific network structure is shown in Figure 8 and Figure 9.
- Figure 8 is a schematic diagram of the structure of Spiking-Lenet-5 (average pooling)
- Figure 9 is a schematic diagram of the structure of Spiking-Lenet-5 (maximum pooling).
- the input image size in Figure 8 and Figure 9 are both 28*28, and each pixel value needs to be converted into a binary number with a bit width of width.
- the first layer is the convolutional layer
- the size of the convolution kernel is 5*5, and the number is 6, and each weight value needs to be copied into a proportional sequence of 1/2, which is copied into width in total, and The 0/1 of the different bits of the same pixel value are multiplied correspondingly;
- the size of each convolution window is 5*5, because the first layer has only 1 channel, if it is a multi-channel input, each convolution kernel is also There should be multiple channels, and the pixel value in each channel is multiplied by the corresponding weight of the convolution kernel; for the 5*5 convolution window, select 5*5 pixel values on the input image, and the pixel values in the same position and the volume
- the weight of the product core corresponds to multiplication; the result of all the multiplication and accumulation in the same convolution window corresponds to the increment of the integral value
- the 28*28*6 above the convolutional layer 1 is the total number of neurons in the convolutional layer 1
- 28*28 is the size of the output image (when performing convolution, the image
- 6 corresponds to the number of convolution kernels, and represents the number of channels of the output image.
- the 28*28*6 output image is directly used as the input of the convolutional layer 2, and new calculations are started. Because the average pooling method is used here, the original 28*28 image should be 2*2 averaged to generate a 14*14 image. Here, any pixel in the 14*14 image is directly assigned to the corresponding 4 in the 28*28 image. The pixels are integrated in the same convolution window. Correspondingly, the original 5*5 convolution window becomes 10*10, and the corresponding weights of adjacent 2*2 pixels are the same. The convolution calculation process is similar to convolution layer 1.
- the maximum pooling layer 1 needs to be added before the convolutional layer 2.
- the function is to choose 1 from 4, so that the input image size of the convolutional layer 2 is 14*14.
- the last 10 counters respectively count the number of high-level pulse signals in the output pulse signals of the 10 neurons in the fully connected layer 3. According to the specific implementation of the system, 10 counters can be added to record the earliest time when each neuron generates a high level.
- the convolutional neural network In the host computer, the convolutional neural network must be trained first, and the trained convolutional neural network is calculated according to the following formula:
- I is the input of a certain layer of the convolutional neural network
- W is the weight
- B is the bias
- O is the output
- channel is the number of input channels
- kernelsize is the size of the convolution kernel.
- ii is the row of the output image
- jj is the column of the output image
- nn is the channel of the output image.
- weight W and bias B of each layer and the input data from the data set, that is, the I of the first layer are processed as follows:
- the gray value in the data set is quantized according to the false data bit width width, and the binary number of width bits is obtained, and the insufficient bits are filled with zeros in the high bits.
- the original input data is expanded into a binary number times width, that is, a pulse signal.
- the gray value of the input image should be multiplied by a certain weight W'in a certain convolution kernel and accumulated according to the calculation formula of the convolutional neural network, then The weights of the first layer are copied into width parts, which are kept unchanged in turn, divided by 2, divided by 4, etc. to the power of 2, and the corrected weight is recorded as W".
- the offset of this layer should be corrected in the above Multiply B'by 2 and record it as B".
- the binary numbers obtained by the quantization of the gray value are sorted corresponding to W', W'/2, W'/4, W'/8, ... from high order to low order.
- each weight W'in the convolutional layer or the fully connected layer of the average pooling layer will be copied into several copies. Is the square of the size of the pooling layer. For example, the pooling layer is 2*2, then each weight is copied into 4 copies, and the corrected weight is recorded as W". If the layer has a bias, then The offset value is magnified 4 times on the basis of B', and denoted as B”.
- the first is that the user scales the weights of each layer according to actual needs (for example, according to the actual quantization bit width and the maximum absolute value of the weight of the layer to achieve the highest possible accuracy), the weight of each layer will be scaled to make the new weight
- the value is W"'.
- I changed I into a number of binary expanded inputs I
- weights and biases are arranged in order according to the calculation formula of the convolutional neural network and written into the storage-calculation integrated unit.
- the host computer After the input of the storage input terminal is all completed, the host computer sends an input pulse to the calculation input terminal of the first-level storage-calculation integrated unit, and the device starts the calculation task.
- Fig. 10 is a schematic diagram of a neuron composed of an integrated storage-calculation unit of Embodiment 4-1.
- all circulating data are impulse signals, that is, 0 or 1
- the basic calculation unit is a storage and calculation unit. The unit is responsible for multiplication.
- a neuron includes multiple storage-calculation integrated units.
- the input of the storage input terminal in these storage-calculation integrated units corresponds to the synapse of the neuron in the human brain, that is, W””, which is the input
- the input of the terminal corresponds to the strength of the synaptic connection, that is, I"".
- the neuron also needs a cell body, which is responsible for accumulating the output results ⁇ I"" ⁇ W""+1*B"" of the storage-calculation unit in each clock cycle, and accumulating it with the neuron
- the potential v(t-1) of the cell body at this time is accumulated.
- the formula is:
- the neuron potential undergoes the following changes, which are all completed before the arrival of the next clock cycle:
- vth - and vth + are hyperparameters that can be set by each layer.
- vth + is a positive threshold
- vth - is a negative threshold.
- the negative threshold can also be set to 0.
- This function is realized by the current integration and comparison circuit, and the output result is saved in the register, aligned with the rising edge of the clock, and transmitted to the neuron in the next layer.
- the corrected bias value has been stored in a storage-calculation unit in each neuron of the layer, and only the storage-calculation unit needs to be calculated
- the calculation input of is always set to 1.
- Scheme 1 The values of 10 counters are transmitted to the upper computer in real time.
- the number of pulses in the type 1 counter is a more than other types
- a is the set constant, that is, the calculation can be ended, and the output The category number with the largest number of pulses.
- the recommended setting is 4.
- the number of pulses is the same, then compare who receives the pulse first, and output the type.
- the upper computer sends the corresponding control signal to the control system, clears and resets some places in the system that need to be cleared and reset, and then sends the input pulse signal of the next picture to start the next round of calculation.
- the values of these 10 counters are transmitted to a special end condition judgment module, which needs to realize such a function: at a certain moment, there are a number of pulses in the counter of type 1, which is a more than that of other types.
- a is the set constant (it is recommended to set it to 4), which means that the calculation can be ended. Or after the set maximum time length is reached, the end judgment condition is not met, and the end is forced to end.
- the end here refers to the end signal of raising an output, which is transmitted to the control system and the upper computer, and the corresponding position of the hardware part is reset.
- the upper computer transmits new image data to the control system, and the control system saves the image data sent by the upper computer, and distributes the image data that needs to be calculated next to the storage-calculation integrated unit. (Here, there can be several different data transmission schemes according to the usage of the memory capacity in the actual system, which are not limited).
- Figure 11 is a block diagram of the entire system (average pooling).
- Figure 12 is a block diagram of the entire system (max pooling).
- CONV convolutional layer
- FC fully connected layer.
- Spiking-Lenet-5 impulse convolutional neural network structure diagram described in conjunction with Figures 8 and 9
- Figure 11 and Figure 12 implement each layer in hardware separately, and the data is circulated in different modules; in addition, the hardware part is also
- Figure 13 is the calculation flow chart of the entire system (average pooling).
- Figure 14 is a calculation flow chart of the entire system (max pooling).
- CONV stands for convolutional layer
- FC stands for fully connected layer.
- the host computer Before calculating the image, it is necessary to first write the trained weights and offsets, after correction, into the storage input terminal of the storage-calculation integrated unit. Afterwards, all modules of the entire hardware accelerator are reset except for the data written in the input terminal. Then the host computer starts to transmit input data to the hardware accelerator, and the control system receives these data. After all the input data of the first picture is transmitted, it starts to distribute data to the storage-calculation integrated unit at the same time.
- the input data of the same image remains unchanged until the image is calculated, according to the specific design of the system, you can wait for one image to be calculated before transmitting the input data of the next image, or in the first Before the images are calculated, the input data of the next or several images is saved in the hardware accelerator to realize the ping-pong operation.
- the storage and calculation integrated unit receives the input signal from the input terminal, and the calculation results of all storage and calculation integrated units add the currents in series and input them into the current integral comparison circuit.
- the circuit is integrated, compared with the threshold, and then generates an output pulse, and completes the operation of aligning with the rising edge of the clock in the following register to obtain the output of this layer.
- the maximum pooling module is also added.
- the outputs of conv1 and conv2 are first connected to the maximum pooling 1 and 2, in the maximum pooling module, the one with the highest level is selected and transmitted to the next layer.
- the counter group counts the number of high-level pulses in the output pulses of each neuron in the last fully connected layer, and the result has been transmitted back to the upper computer by the control system.
- the upper computer judges whether the calculation of this picture is completed according to the conditions set by the user. If it is not completed, it will continue to maintain the status quo. If it is completed, it will change the control signal and combine the hardware accelerator, current integral comparison circuit and other registers in the system with The counter is reset and new pictures are transferred.
- Solution 2 The counter group counts the number of high-level pulses in the output pulses of each neuron in the last fully connected layer, and the earliest time each neuron generates high-level pulses. The result is only after the end of this round of calculation It is sent back to the upper computer by the control system. The judgment of the end of this round of calculation is completed by the digital logic according to the results of the counter group statistics. If it is not over, it will continue to maintain the status quo. If it is over, it will send a high end signal to the control system, and the hardware accelerator, the current integral comparison circuit And other registers and counters in the system are reset, and the storage and calculation unit waits for the control system to distribute the data of the next image. After the upper computer receives the end signal, it sends a new picture to the control system, and processes the returned counter group data to obtain the final classification result.
- the current integration and comparison circuit replaces the analog-to-digital converter with large power consumption and area, which greatly reduces the area and power consumption of the entire system.
- the output results of each layer of convolutional layer/fully connected layer are directly connected with the next layer of convolutional layer/fully connected layer, and the weight data can be directly stored in the storage and calculation unit, and the entire system does not need on-chip Caching saves a lot of data handling process, thereby speeding up calculations.
- the function of synchronizing the output result of the integral comparison circuit with the clock signal is added, and each register connected to the integral comparison circuit in the embodiment 4-1 is removed ,
- the output of the integral comparison circuit is directly connected to the next layer of neurons, maximum pooling module or counter. See Figure 15 for a schematic diagram of neurons without registers, see Figure 16 for a block diagram of the entire system (register removal, average pooling), and Figure 17 for a block diagram of the entire system (register removal, maximum pooling).
- the data set is Cifar-10 as an example, the data set size is 10000*32*32*3, a total of 10000 sets of test data, the image size is 32*32, the number of channels is 3, the data is an integer from 0 to 255, and the number of categories is 10.
- the convolutional neural network takes Alexnet as an example.
- the model used here has changed.
- the first and second convolutional layers are followed by a BN layer, the pooling layer is changed to average pooling, and all convolutional layers
- the size of the convolution kernel is 3*3, and the specific network structure is shown in Figure 18.
- the input image size in FIG. 18 is 32*32, the number of channels is 3, and each pixel value needs to be converted into a binary number with a bit width of width.
- the first layer is the convolutional layer, the size of the convolution kernel is 3*3, the number of channels is 3, the number is 96, and each weight value needs to be copied into a geometric sequence of 1/2, which is copied in total Into width, it is multiplied by 0/1 corresponding to different bits of the same pixel value; the convolution kernel of 3 channels corresponds to the input image; the size of the convolution window is 3*3, and 3*3 is selected on the input image Pixel value, the pixel value of the same position is multiplied with the weight of the convolution kernel; the result of all the multiplication and accumulation in the same convolution window corresponds to the increment of the integral value in a neuron current integration and comparison circuit;
- the convolution window slides the window in a fixed order on the input image, which corresponds to different neurons; afterwards, a different convolution kernel is replaced, which
- the 32*32*96 above the convolutional layer 1 is the total number of neurons in the convolutional layer 1
- 32*32 is the size of the output image (when convolution is performed, the edge of the image is in the convolution
- the insufficient part in the product window is filled with 0)
- 96 corresponds to the number of convolution kernels, and represents the number of channels of the output image.
- the 3 output image is directly used as the input of the convolutional layer 2, and a new calculation is started. Because the average pooling method is used here, the original 32*32 image should be 2*2 averaged to generate a 16*16 image.
- any pixel in the 16*16 image is directly corresponding to 4 in the 32*32 image
- the pixels are integrated in the same convolution window.
- the original 3*3 convolution window becomes 6*6, and the corresponding weights of adjacent 2*2 pixels are the same.
- the convolution calculation process is similar to convolution layer 1. The same is true for other convolutional layers.
- the last 10 counters respectively count the number of high-level pulse signals in the output pulse signals of the 10 neurons in the fully connected layer 3. According to the specific implementation of the system, 10 counters can be added to record the earliest time when each neuron generates a high level.
- the convolutional neural network In the host computer, the convolutional neural network must be trained first, and the trained convolutional neural network is calculated according to the following formula:
- I is the input of a certain layer of the convolutional neural network
- W is the weight
- B is the bias
- O is the output
- channel is the number of input channels
- kernelsize is the size of the convolution kernel.
- ii is the row of the output image
- jj is the column of the output image
- nn is the channel of the output image.
- weight W and bias B of each layer and the input data from the data set, that is, the I of the first layer are processed as follows:
- the RGB values in the data set are quantized according to the false data bit width width to obtain the binary number of width bits, and the insufficient bits are filled with zeros in the high bits.
- the original input data is expanded into a binary number times width, that is, a pulse signal.
- the first layer assuming that the RGB value of the input image should be multiplied by a certain weight W'in a certain convolution kernel and accumulated according to the calculation formula of the convolutional neural network, then the first The weights of a layer are copied into width parts, keep unchanged in turn, divided by 2, divided by 4, etc. to the power of 2, and the corrected weight is recorded as W".
- the offset of this layer should be in the above-mentioned correction B Multiply it by 2 on the basis of', and record it as B".
- the binary numbers obtained by the quantization of the RGB values are sorted corresponding to W', W'/2, W'/4, W'/8, ... from high order to low order.
- each weight W'in the convolutional layer or the fully connected layer of the average pooling layer will be copied into several copies. Is the square of the size of the pooling layer. For example, the pooling layer is 2*2, then each weight is copied into 4 copies, and the corrected weight is recorded as W". If the layer has a bias, then The offset value is magnified 4 times on the basis of B', and denoted as B”.
- the first is that the user scales the weights of each layer according to actual needs (for example, according to the actual quantization bit width and the maximum absolute value of the weight of the layer to achieve the highest possible accuracy), the weight of each layer will be scaled to make the new weight
- the value is W"'.
- I changed I into a number of binary expanded inputs I
- weights and biases are arranged in order according to the calculation formula of the convolutional neural network and written into the storage-calculation integrated unit.
- the host computer After the input of the storage input terminal is all completed, the host computer sends an input pulse to the calculation input terminal of the first-level storage-calculation integrated unit, and the device starts the calculation task.
- Fig. 19 is a schematic diagram of a neuron composed of an integrated storage-calculation unit in embodiment 4-2.
- all circulating data are impulse signals, that is, 0 or 1
- the basic calculation unit is a storage and calculation unit. The unit is responsible for multiplication.
- a neuron includes multiple storage-calculation integrated units.
- the input of the storage input terminal in these storage-calculation integrated units corresponds to the synapse of the neuron in the human brain, that is, W””, which is the input
- the input of the terminal corresponds to the strength of the synaptic connection, that is, I"".
- the neuron also needs a cell body, which is responsible for accumulating the output results ⁇ I"" ⁇ W""+1*B"" of the storage-calculation unit in each clock cycle, and accumulating it with the neuron
- the potential v(t-1) of the cell body at this time is accumulated.
- the formula is:
- the neuron potential undergoes the following changes, which are all completed before the arrival of the next clock cycle:
- vth - and vth + are hyperparameters that can be set by each layer.
- vth + is a positive threshold
- vth - is a negative threshold.
- the negative threshold can also be set to 0.
- This function is implemented by a current integration and comparison circuit, and the output result is saved in the on-chip buffer.
- a fixed-duration output pulse signal needs to be collected as a data packet, and the fixed-duration is consistent with the transmission time of the input pulse.
- the output result is buffered in all the data packets needed in the calculation of the next layer, it will be transmitted to the neurons of the next layer in the form of data packets, and the accumulated value in the current integral comparison circuit will be cleared. zero.
- the capacity of the on-chip cache and the number of neurons required need to be considered based on the actual situation, comprehensive area, power consumption, speed, and the balance of the calculation speed of each layer.
- the corrected bias value has been stored in a storage-calculation unit in each neuron of the layer, and only the storage-calculation unit needs to be calculated
- the calculation input of is always set to 1.
- zero padding is required in many convolutional layers, because the position of zero padding is fixed, so long as the corresponding input is always set to 0 during its sending time.
- the upper computer sends the corresponding control signal to the control system, clears and resets some places in the system that need to be cleared and reset, and then sends the input pulse signal of the next picture to start the next round of calculation.
- FIG. 20 The block diagram of the entire system is shown in Figure 20.
- each layer is implemented in hardware separately in Figure 20, and the data is circulated in different modules; in addition, the hardware part also has a control system for Receive the input data and control signal from the upper computer, write the input data into the on-chip buffer, and receive the statistical result from the counter module, and then send it to the upper computer.
- the hardware part also has a control system for Receive the input data and control signal from the upper computer, write the input data into the on-chip buffer, and receive the statistical result from the counter module, and then send it to the upper computer.
- the logic circuits receive control signals from the control system, and generate on-chip buffer control signals and memory addresses according to the sequence of data distribution calculations.
- the received output pulse is saved to the on-chip buffer; according to the actual buffer capacity, the data in the buffer that will no longer be used is covered by new data; according to the sequence of data distribution calculations, the on-chip buffer control signal and reading address are generated, and Conv1 ⁇ Conv5 and FC1 calculation of the required input data read out.
- the calculation flow chart of the whole system is shown in Figure 21.
- Fig. 21 before calculating the image, it is necessary to first write the trained weights and biases into the storage input terminal of the storage-calculation integrated unit after correction. Afterwards, all modules of the hardware accelerator except for the data written in the input terminal are reset. Then the host computer starts to transmit input data to the hardware accelerator. The control system receives these data and writes them into the on-chip buffer. After all the input data required for a calculation of the Conv1 module is transmitted, the on-chip buffer starts to store and calculate the Conv1 module. The integrated unit distributes the data. After all the distribution is completed, the Conv1 module starts to calculate.
- the transmission speed of the host computer should be considered in conjunction with the calculation speed of each module of the entire system and the on-chip buffer capacity. But it should be ensured that after a fixed duration T, the data required for the next calculation of the Conv1 module has been saved on the on-chip cache within T time.
- the storage and calculation integrated unit receives the input signal from the input terminal, and the calculation results of all storage and calculation integrated units add the currents in series and input them into the current integral comparison circuit.
- the circuit is integrated, compared with the threshold, and then generates an output pulse, and completes the operation of aligning with the rising edge of the clock in the following register to obtain the output of this layer.
- the input signal of the input terminal comes from the on-chip buffer, that is, for each calculation of these modules with a duration of T clock cycles, the prerequisite for the beginning is all required for the calculation
- the input has been read from the on-chip cache.
- the input signal comes from the output signal of the previous fully connected layer.
- the output signal of each neuron will be packed according to the unit size of T and stored in the on-chip cache.
- the counter group counts the number of high-level pulses in the output pulses of each neuron in the last fully connected layer.
- the result is sent back to the host computer by the control system.
- the host computer changes the control signal, resets the current integral comparison circuit and its registers and counters in the hardware accelerator, and starts to transmit new pictures.
- 10 counters can also be added to record the earliest time when each neuron generates a high level for auxiliary judgment of the classification result.
- the on-chip cache is changed to a register, and the logic control circuit should be modified accordingly, because the register positioning method is different from the on-chip cache.
- the system block diagram is shown in Figure 22.
- the on-chip cache is changed to off-chip memory.
- the hardware accelerator part it only includes the storage-calculation integrated unit and counter part of each layer.
- the on-chip cache and its logic control circuit are moved off-chip, and its functions are replaced by FPGA development boards (field programmable gate array) and DDR (double rate synchronous dynamic random access memory).
- FPGA development boards field programmable gate array
- DDR double rate synchronous dynamic random access memory
- the on-chip cache is changed to off-chip cloud storage.
- the hardware accelerator part it only includes the storage-calculation integrated unit and counter part of each layer.
- the on-chip cache and its logic control circuit have been moved off-chip, and their functions are replaced by host computers and cloud storage.
- the system block diagram is shown in Figure 24.
- any one of the above embodiments is used as an integrated storage-calculation unit to perform pulse convolutional neural network calculations.
- the following specific implementations are available:
- FIG. 25 is a diagram of the Alexnet network structure of the fifth embodiment.
- the data set is cifar-10 as an example.
- the size of the data set is 10000*32*32*3.
- the input image size is 32*32, the number of channels is 3, and the data is 0.
- An integer between -255, the number of classifications is 10.
- the convolutional neural network takes Alexnet as an example.
- the model used here has changed.
- the first and second convolutional layers are followed by a BN layer.
- the pooling layer can be maximum pooling or average pooling.
- the size of the convolution kernel of all convolutional layers is 3*3.
- I is the input of the layer
- W is the weight
- B is the bias
- O is the output
- channel is the number of input channels
- kernelsize is the size of the convolution kernel, both of which are 3 here.
- I is the input of the layer
- W is the weight
- B is the bias
- O is the output
- channel is the number of input channels.
- FIG. 28 is a structural diagram of a neuron in Example 5.
- a neuron includes multiple storage-calculation integrated units, and the storage-calculation integrated unit is similar to the description in the foregoing embodiment 4, and will not be repeated here.
- bn.weight( ⁇ ), bn.bias( ⁇ ), bn.running_mean(mean), bn.running_var(var), and bn.eps need to be derived during training. (eps, a small amount added to the denominator, the default is 1e-5), and modify the weight and offset of Layer M according to the following formula:
- the pixel value is 64, which is converted to a binary value of 01000000, 0-255 can be represented by an 8-bit binary number, and the insufficient number of bits is filled with zeros in the high position.
- the pixel value should be multiplied by a certain weight W in a certain convolution kernel.
- the original 64*W needs to be converted to 128*(0*W+1*W/2+0*W/4+0*W/8+0*W/16+0*W/32+0* W/64+0*W/128), that is, the input is expanded to 8 times the original value, and the weight is also copied 7 copies, and then divided by different exponential powers of 2, and then accumulated.
- the whole round of calculation corresponds to a set of test data in the data set.
- Cifar-10 it is a 32*32*3 image.
- the bias value should be multiplied by 2 on the basis of the above correction formula; if the weight of the layer is scaled as a whole, the bias value It should also be scaled in the same proportion.
- the input pulse is generated by a random number, that is, a decimal number of 0-1 is randomly generated, and compared with the pixel value/255. If the random number is smaller than this number, a pulse is generated, otherwise No pulse is generated.
- this method has great randomness, and only after calculating a large number of pulses can it be as close to the original pixel value as possible.
- the input pulse is completely equivalent to the original pixel value, and a large number of pulses are not required.
- the weights are quantified. That is, assuming that the weight can only be represented by a WW-bit binary number at most, for WW not less than 8 (the 8-bit binary number required for 0-255, for a larger range of input data, it is not 8), the input data and the weight Copy as described above. But if WW is less than 8, then the weights divided by the larger exponential power of 2 may be directly equal to 0, and the input has no effect on it and can be omitted directly. That is, the input takes the WW bits starting from the high bit, and the copied weights also take the WW bits with the larger absolute value.
- the weight scaling of the previous layers will be accumulated in this layer.
- the weight of the first layer is adjusted, and finally in the integral comparison circuit
- the cumulative result in the convolutional neural network model is twice the theoretical value, so in addition to the first layer's bias needs to be doubled, to the second layer, this 2 times will still be from the frequency of the input pulse.
- pooling layers in the convolutional neural network. Two types of maximum pooling and average pooling are commonly used. If the size of the pooling window is 2*2, then pooling The function of is to change the original 4 inputs into 1 output, reducing the size of the image and reducing the amount of calculation. Maximum pooling is to select the maximum output from these 4 inputs, and average pooling is to calculate the average output of these 4 inputs.
- each counter corresponds to a neuron, and also corresponds to a type of result of image classification,
- the number of pulses in one type of counter is a more than other types
- a is the set hyperparameter, that is, it is considered that the calculation can be ended, and the type number of the type with the largest number of pulses is output.
- the recommended setting is 4.
- the number of pulses is the same, then compare who receives the pulse first, and output the type.
- the average pooling layer is merged into the next convolutional layer or the fully connected layer, and the calculation and support of the biased convolutional layer and the fully connected layer are supported.
- Adding the BN layer to the network, setting the conditions for determining the end of the calculation, and adding auxiliary judgments for special situations can greatly save the calculation time of the existing pulse convolutional neural network algorithm and improve the accuracy of image classification.
- improvements have been made to the calculation time.
- the storage-computing integrated unit according to the above-mentioned embodiments of the present invention can be implemented in an integrated circuit. Next, the manufacturing method of such an integrated circuit will be described, which includes the following steps:
- the dielectric layers and gates of the transistors in the digital logic circuit, the integral comparison circuit and the storage-calculation integrated unit are formed by thermal oxidation and deposition;
- the transistors include ordinary logic transistors, high-voltage transistors and floating gate transistors, etc.;
- the capacitor in the integration comparison circuit is formed by depositing a MIM dielectric layer and a metal layer, or thermal oxidation and deposition processes, and the capacitor can be a MIM capacitor or a MOS capacitor;
- CMOS process memory-calculation integrated unit is generated.
- the production process of integrated circuits based on pulsed convolutional neural networks can all be produced using standard CMOS technology.
- the storage and calculation integrated unit in the neuron if the photoelectric computing unit or flash memory is used If it is, it can also be produced using standard CMOS process.
- CMOS process for semiconductor devices such as transistors, diodes or capacitors based on this process, it will not be described in detail here. Get better device performance.
- a memristor is used as a storage-calculation unit in a neuron, a special process compatible with this memristor is required, including a storage-calculation integrated device with a special process and a digital logic circuit and an integral comparison circuit using standard CMOS technology.
- the integration method can be achieved by directly using special processes to produce special devices on a silicon-based substrate, or by wafer-level integration or off-chip integration. For example, the method of generating high-durability memristors on silicon-based substrates mentioned in Chinese Patent CN110098324A and other memristor manufacturing process methods.
- the devices and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the present disclosure.
- the disclosed device and method may be implemented in other ways.
- the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another device, or some features can be ignored or not implemented.
- the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
- the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
- the technical solution of the present disclosure essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present disclosure.
- the aforementioned storage media include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Neurology (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Complex Calculations (AREA)
- Error Detection And Correction (AREA)
Abstract
Description
Claims (29)
- 一种脉冲卷积神经网络算法,基于存算一体单元,所述存算一体单元包括至少一个存输入端,至少一个算输入端以及一个输出端,其特征在于:1)将脉冲卷积神经网络的第一层的权值复制至若干份,份数至少为用于表征待分析物属性的量所转换成的二进制数的位数以及所述存算一体单元的存输入端的最小值,并且将复制后的所述份数的权值进行处理,使复制后的各个权值在数值上依次缩小两倍,所得数值被分别输入到多个所述存算一体单元的存输入端,所述存算一体单元的个数与所述份数相同;2)将所选的、集中用于表征待分析物属性的量转换成二进制数,并将待输入的所述二进制数的每一位数值,或者根据系统位宽截位后的数值作为输入脉冲,输入到所述脉冲卷积神经网络的存算一体计算单元中;并且,对于每个表征待分析物属性的输入集合,在对应于所述输入集合的时间周期内,使所述输入脉冲保持不变并不间断地输入到所述脉冲卷积神经网络中相应的计算单元,直到完成对该被分析物的所述属性的处理;3)对于用于表征待分析物属性的、对应于所述一个组中的每个基本的二进制数,使所述每个存输入端的输入量,分别与一个算输入端的输入量相对应,并且绝对值较大的存输入端的输入量与较高位的算输入端的输入量一一对应;4)在每个所述存算一体单元中,使所述存输入端的量与所述算输入端的量进行运算,输出端得到的电流值代表所述存算一体单元的存输入端的值与算输入端的值进行乘法运算的结果。
- 如权利要求1所述的脉冲卷积神经网络算法,其特征在于:1)包括所述第一层的运算以及其它层的运算,并且在其中的任意层,在所述存输入端与所述算输入端的运算以外,再加一个运算累加项,所述运算累加项为一个经过修正的偏置值,所述经过修正的偏置值正比于其原始值再除以该层之前所有层的正阈值的累乘,所述正比的比例与该偏置所在的层以及之前的层的权值缩放比例有关;2)所述脉冲卷积神经网络算法,对所述存算一体单元的输出持续地进行 累加,当所述累加和超过一个设定的正阈值后,对所述累加和进行清零,并且向下一层相应位置的算输入端释放一个输出脉冲;并且当所述累加和小于一个设定的负阈值之后,使该累加和保持在该负阈值上。
- 如权利要求2所述的脉冲卷积神经网络算法,其特征在于,所述脉冲卷积神经网络中包括批标准化层,对该批标准化层之前的一个卷积层或全连接层中的权值和偏置进行线性变换,其中所述线性变换中的参数由前面的训练过程中得到。
- 如权利要求1至3中任一项所述的脉冲卷积神经网络算法,其特征在于,用多个计数器对所述脉冲卷积神经网络最后一个全连接层中每个神经元的脉冲个数以及最早出现脉冲的时间进行统计,所述计数器个数为所述神经元的数目或其两倍。
- 如权利要求4所述的脉冲卷积神经网络算法,其特征在于,如果所述多个计数器中至少两个计数器计数结果均为相同的最大值,则选取最早接收到脉冲的计数器所对应的类别值为最终结果。
- 如权利要求4所述的脉冲卷积神经网络算法,其特征在于,计数器显著地多,则输出终止运算,将最终的分类结果作为所述多个计数器计数结果的最大值所对应的类别值进行输出。
- 如权利要求1所述的脉冲卷积神经网络算法,其特征在于,在所述第一层的运算之后,还进行平均池化、最大池化、卷积层和全连接层运算中的至少一种。
- 如权利要求1至3,及7中任一项所述的脉冲卷积神经网络算法,其特征在于:1)设定若干个时钟信号的时长为一个分析周期;2)将待分析的标的物分为若干分区;3)以所述分析周期为时间单位,逐次分析一个分区的时间序列信号,将代表该分区的运算结果送至一个存储器;4)分析下一个分区的信号,将所述代表该分区的运算结果送至所述存储器,直到所完成的多个分区的信号联合地满足下一层的分析条件;5)将所述存储器存储的各个所述分区的信号送入下一层进行运算。
- 如权利要求8所述的脉冲卷积神经网络算法,其特征在于,所述存储器为寄存器、片上缓存、片外存储或者云存储中的至少一种,或者它们的组合。
- 一种基于脉冲卷积神经网络的集成电路,其特征在于,所述集成电路执行如权利要求1-3,及7中任一项所述的脉冲卷积神经网络算法。
- 一种基于脉冲卷积神经网络的集成电路,其特征在于,所述集成电路执行如权利要求8所述的脉冲卷积神经网络算法。
- 一种计算机可读记录介质,其上存储计算机可读指令,当所述计算机可读指令由计算机执行时,使得所述计算机执行脉冲卷积神经网络算法,所述脉冲卷积神经网络算法的特征在于:1)将脉冲卷积神经网络的第一层的权值复制至若干份,份数至少为用于表征待分析物属性的量所转换成的二进制数的位数以及所述存算一体单元的存输入端的最小值,并且将复制后的所述份数的权值进行处理,使复制后的各个权值在数值上依次缩小两倍,所得数值被分别输入到多个所述存算一体单元的存输入端,所述存算一体单元的个数与所述份数相同;2)将所选的、集中用于表征待分析物属性的量转换成二进制数,并将待输入的所述二进制数的每一位数值,或者根据系统位宽截位后的数值作为输入脉冲,输入到所述脉冲卷积神经网络的存算一体计算单元中;并且,对于每个表征待分析物属性的输入集合,在对应于所述输入集合的时间周期内,使所述输入脉冲保持不变并不间断地输入到所述脉冲卷积神经网络中相应的计算单元,直到完成对该被分析物的所述属性的处理;3)对于用于表征待分析物属性的、对应于所述一个组中的每个基本的二 进制数,使所述每个存输入端的输入量,分别与一个算输入端的输入量相对应,并且绝对值较大的存输入端的输入量与较高位的算输入端的输入量一一对应;4)在每个所述存算一体单元中,使所述存输入端的量与所述算输入端的量进行运算,输出端得到的电流值代表所述存算一体单元的存输入端的值与算输入端的值进行乘法运算的结果。
- 如权利要求12所述的计算机可读记录介质,其特征在于:1)所述脉冲卷积神经网络算法包括所述第一层的运算以及其它层的运算,并且在其中的任意层,在所述存输入端与所述算输入端的运算以外,再加一个运算累加项,所述运算累加项为一个经过修正的偏置值,所述经过修正的偏置值正比于其原始值再除以该层之前所有层的正阈值的累乘,所述正比的比例与该偏置所在的层以及之前的层的权值缩放比例有关;2)所述脉冲卷积神经网络算法,对所述存算一体单元的输出持续地进行累加,当所述累加和超过一个设定的正阈值后,对所述累加和进行清零,并且向下一层相应位置的算输入端释放一个输出脉冲;并且当所述累加和小于一个设定的负阈值之后,使该累加和保持在该负阈值上。
- 如权利要求12所述的计算机可读记录介质,其特征在于,所述脉冲卷积神经网络中包括批标准化层,对该批标准化层之前的一个卷积层或全连接层中的权值和偏置进行线性变换,其中所述线性变换中的参数由前面的训练过程中得到。
- 如权利要求12至14中任一项所述的计算机可读记录介质,其特征在于,用多个计数器对所述脉冲卷积神经网络最后一个全连接层中每个神经元的脉冲个数以及最早出现脉冲的时间进行统计,所述计数器个数为所述神经元的数目或其两倍。
- 如权利要求15所述的计算机可读记录介质,其特征在于,如果所述多个计数器中至少两个计数器计数结果均为相同的最大值,则选取最早接收到 脉冲的计数器所对应的类别值为最终结果。
- 如权利要求15所述的计算机可读记录介质,其特征在于,在所述多个计数器进行计数的过程中,一个计数器收集的脉冲数比其他计数器显著地多,则输出终止运算,将最终的分类结果作为所述多个计数器计数结果的最大值所对应的类别值进行输出。
- 如权利要求12所述的计算机可读记录介质,其特征在于,在所述第一层的运算之后,还进行平均池化、最大池化、卷积层和全连接层运算中的至少一种。
- 如权利要求12至14,及18中任一项所述的计算机可读记录介质,其特征在于,所述脉冲卷积神经网络算法包括以下:1)设定若干个时钟信号的时长为一个分析周期;2)将待分析的标的物分为若干分区;3)以所述分析周期为时间单位,逐次分析一个分区的、时间序列信号,将代表该分区的运算结果送至一个存储器,已分析的信号可以被后续的信号覆盖;4)分析下一个分区的信号,将所述代表该分区的运算结果送至所述存储器,直到所完成的多个分区的信号联合地满足下一层的分析条件;5)将所述存储器存储的各个所述分区的信号送入下一层进行运算。
- 如权利要求19所述的计算机可读记录介质,其特征在于,所述存储器为寄存器、片上缓存、片外存储或者云存储中的至少一种,或者它们的组合。
- 一种基于脉冲卷积神经网络的集成电路,所述脉冲卷积神经网络包括多层神经元,每层神经元包括多个神经元组件,每层神经元中的多个神经元彼此不连接,而连接到后层的神经元;至少一个所述神经元组件带有至多一个数字逻辑电路,所述数字逻辑电路被用于操作,所述操作包括数据分发;并且,最后一层的每个神经元组件带有 一个计数器组,统计该神经元组件的输出脉冲中具有高电平的脉冲个数;其中,每个神经元包括至少一个存算一体单元和至少一个积分比较电路,所述多个存算一体单元的电流输出端彼此连接,并且集体地连接到所述积分比较电路上;每个所述积分比较电路包括至少一个积分器和至少一个比较器,所述积分器用于累加电流输出端的输出量,所述比较器用于将积分器中被累加的输出量与在先设定的阈值进行比较,并且进行比较器的清零和脉冲输出,所述清零的操作使所述的积分器可以进行下一次的累加操作;并且,每个所述存算一体单元包括至少一个存输入端和至少一个算输入端以及至少一个电流输出端,所述存输入端被设置为接收表征所述上位机所下发的权值的载流子,所述算输入端被设置为接收表征外界或所设定的上层输入脉冲的载流子;所述电流输出端被设置为以电流的形式输出被作为权值的载流子和作为输入脉冲的载流子共同作用后的载流子。
- 如权利要求21所述的集成电路,其特征在于,所述存算一体单元为半导体原理的光电计算单元、忆阻器、快闪存储器中的一种。
- 如权利要求21或22所述的集成电路,其特征在于,所述数字逻辑电路的所述操作还包括最大池化、时钟同步和数据缓存。
- 如权利要求21所述的集成电路,其特征在于,所述数字逻辑电路被设置为从当前池化层的上一层神经元组件中输出的、数量为池化层尺寸的平方的多个输出信号中,找出最先出现的高电平脉冲信号;并且,所述数字逻辑电路还被设置为包括一个多路选择器的功能器件,使所述高电平脉冲信号经过所述多路选择器后,保持该高电平脉冲信号所对应的通路开启,将所述通路与下一个卷积层或全连接层连通;同时忽略与该高电平脉冲信号所对应的通路相并行的其它通路的信号,或者关闭所述其它通路。
- 如权利要求21所述的集成电路,其特征在于,将平均池化运算合并到下一个卷积层或全连接层中进行,包括:1)卷积层或全连接层,所述卷积层或全连接层的每个神经元组件中的存算一体单元数量为该层对应算法的原始尺寸的若干倍,倍数为池化层尺寸的平方,并且所述对应算法中的每一个权值在所述神经元组件中出现若干次,次数为池化层尺寸的平方,2)其中从上一层神经元组件中输出的、待传输到下一个池化层的、数量为池化层尺寸的平方的输出脉冲信号,直接作为所述卷积层或全连接层中的存算一体单元的算输入量,所述存算一体单元分别与同样的权值对应。
- 如权利要求21所述的集成电路,其特征在于,每个所述神经元组件包括一个神经元,并且带有寄存器,所述寄存器用于实现所涉及的数据操作在时间上的同步。
- 一种脉冲卷积神经网络运算装置,用于进行脉冲卷积神经网络运算,包括一个上位机和如权利要求21所述的集成电路;其中,所述上位机被设置为处理并生成第一层的权值,所述生成第一层的权值的过程包括:根据一个训练得出的初始权值经过若干线性变换生成一组权值,该组权值包括多个权值数,其中后一个权值数值为前一个权值数值的1/2;并且,所述上位机将该组权值发送给所述脉冲卷积神经网络的第一层的各个神经元组件中的存算一体单元中的存输入端;并且,所述上位机将初始权值经过若干线性变换后发送给所述第一层之后的其它层的存算一体单元的存输入端中,对于紧接着平均池化层之后的卷积层或全连接层的权值,还根据池化尺寸将权值复制若干份,份数为池化层尺寸的平方。
- 如权利要求27所述的脉冲卷积神经网络运算装置,其特征在于,所述装置被用于按分区来分析标的物,再将各分区的标的物信号合成,构成完整的标的物信息,并且所述脉冲卷积神经网络运算装置还包括存储器,所述存储器用于存储已分步处理过的、代表所述标的物的至少一个分区的信号,并在所有的分区信号处 理完以后,将所有的分区信号进行合成,或将所有的分区信号发送至另一个处理器进行合成;所述存储器为寄存器、片上缓存、片外存储或者云存储中的至少一种。
- 一种集成电路的制造方法,其特征在于,所述集成电路为权利要求21所述的集成电路,所述方法包括以下步骤:1)通过热氧化和淀积形成数字逻辑电路、积分比较电路和存算一体单元中晶体管的介质层和栅极;所述晶体管至少包括普通逻辑晶体管,高压晶体管以及浮栅晶体管;2)通过淀积MIM介质层以及淀积金属层,或热氧化和淀积工艺形成积分比较电路中的电容;3)通过离子注入的方式形成数字逻辑电路、积分比较电路和存算一体单元中晶体管的源极和漏极,以及PN结的P级和N级;4)通过金属层工艺、金属层介质工艺以及通孔工艺形成整体电路的金属连线和有源区-金属层以及金属层-金属层通孔;5)通过应用于忆阻器或快闪存储器的工艺,生成一个CMOS工艺的存算一体单元。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911249006.1 | 2019-12-09 | ||
CN201911249006.1A CN113033759A (zh) | 2019-12-09 | 2019-12-09 | 脉冲卷积神经网络算法、集成电路、运算装置及存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021115262A1 true WO2021115262A1 (zh) | 2021-06-17 |
Family
ID=76329546
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/134558 WO2021115262A1 (zh) | 2019-12-09 | 2020-12-08 | 脉冲卷积神经网络算法、集成电路、运算装置及存储介质 |
Country Status (3)
Country | Link |
---|---|
CN (1) | CN113033759A (zh) |
TW (1) | TWI774147B (zh) |
WO (1) | WO2021115262A1 (zh) |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113642724A (zh) * | 2021-08-11 | 2021-11-12 | 西安微电子技术研究所 | 一种高带宽存储的cnn加速器 |
CN113657580A (zh) * | 2021-08-17 | 2021-11-16 | 重庆邮电大学 | 基于微环谐振器和非易失性相变材料的光子卷积神经网络加速器 |
CN113837373A (zh) * | 2021-09-26 | 2021-12-24 | 清华大学 | 数据处理装置以及数据处理方法 |
CN114723023A (zh) * | 2022-03-03 | 2022-07-08 | 北京大学 | 数据通信方法及系统、脉冲神经网络运算系统 |
CN115545190A (zh) * | 2022-12-01 | 2022-12-30 | 四川轻化工大学 | 一种基于概率计算的脉冲神经网络及其实现方法 |
WO2023045114A1 (zh) * | 2021-09-22 | 2023-03-30 | 清华大学 | 存算一体芯片及数据处理方法 |
CN116167424A (zh) * | 2023-04-23 | 2023-05-26 | 深圳市九天睿芯科技有限公司 | 基于cim的神经网络加速器、方法、存算处理系统与设备 |
CN116205274A (zh) * | 2023-04-27 | 2023-06-02 | 苏州浪潮智能科技有限公司 | 一种脉冲神经网络的控制方法、装置、设备及存储介质 |
WO2023103149A1 (zh) * | 2021-12-06 | 2023-06-15 | 成都时识科技有限公司 | 脉冲事件决策装置、方法、芯片及电子设备 |
CN116663632A (zh) * | 2023-08-02 | 2023-08-29 | 华中科技大学 | 感存算一体智能感知系统 |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113625963A (zh) * | 2021-07-16 | 2021-11-09 | 南京大学 | 基于存算一体器件的卷积神经网络层间的存储装置及方法 |
CN114692681B (zh) * | 2022-03-18 | 2023-08-15 | 电子科技大学 | 基于scnn的分布式光纤振动及声波传感信号识别方法 |
CN114997388B (zh) * | 2022-06-30 | 2024-05-07 | 杭州知存算力科技有限公司 | 存算一体芯片用基于线性规划的神经网络偏置处理方法 |
WO2023160735A2 (zh) * | 2023-06-09 | 2023-08-31 | 南京大学 | 一种运算方法和运算单元 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190005376A1 (en) * | 2017-06-30 | 2019-01-03 | Intel Corporation | In-memory spiking neural networks for memory array architectures |
CN110119785A (zh) * | 2019-05-17 | 2019-08-13 | 电子科技大学 | 一种基于多层spiking卷积神经网络的图像分类方法 |
CN110276047A (zh) * | 2019-05-18 | 2019-09-24 | 南京惟心光电系统有限公司 | 一种利用光电计算阵列进行矩阵向量乘运算的方法 |
CN110543933A (zh) * | 2019-08-12 | 2019-12-06 | 北京大学 | 基于flash存算阵列的脉冲型卷积神经网络 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FR3025344B1 (fr) * | 2014-08-28 | 2017-11-24 | Commissariat Energie Atomique | Reseau de neurones convolutionnels |
US9646243B1 (en) * | 2016-09-12 | 2017-05-09 | International Business Machines Corporation | Convolutional neural networks using resistive processing unit array |
US10565492B2 (en) * | 2016-12-31 | 2020-02-18 | Via Alliance Semiconductor Co., Ltd. | Neural network unit with segmentable array width rotator |
CN109460817B (zh) * | 2018-09-11 | 2021-08-03 | 华中科技大学 | 一种基于非易失存储器的卷积神经网络片上学习系统 |
CN109800870B (zh) * | 2019-01-10 | 2020-09-18 | 华中科技大学 | 一种基于忆阻器的神经网络在线学习系统 |
CN109871940B (zh) * | 2019-01-31 | 2021-07-27 | 清华大学 | 一种脉冲神经网络的多层训练算法 |
CN110263926B (zh) * | 2019-05-18 | 2023-03-24 | 南京惟心光电系统有限公司 | 基于光电计算单元的脉冲神经网络及其系统和运算方法 |
CN110334799B (zh) * | 2019-07-12 | 2022-05-24 | 电子科技大学 | 基于存算一体的神经网络推理与训练加速器及其运行方法 |
-
2019
- 2019-12-09 CN CN201911249006.1A patent/CN113033759A/zh active Pending
-
2020
- 2020-12-04 TW TW109142950A patent/TWI774147B/zh not_active IP Right Cessation
- 2020-12-08 WO PCT/CN2020/134558 patent/WO2021115262A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190005376A1 (en) * | 2017-06-30 | 2019-01-03 | Intel Corporation | In-memory spiking neural networks for memory array architectures |
CN110119785A (zh) * | 2019-05-17 | 2019-08-13 | 电子科技大学 | 一种基于多层spiking卷积神经网络的图像分类方法 |
CN110276047A (zh) * | 2019-05-18 | 2019-09-24 | 南京惟心光电系统有限公司 | 一种利用光电计算阵列进行矩阵向量乘运算的方法 |
CN110543933A (zh) * | 2019-08-12 | 2019-12-06 | 北京大学 | 基于flash存算阵列的脉冲型卷积神经网络 |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113642724B (zh) * | 2021-08-11 | 2023-08-01 | 西安微电子技术研究所 | 一种高带宽存储的cnn加速器 |
CN113642724A (zh) * | 2021-08-11 | 2021-11-12 | 西安微电子技术研究所 | 一种高带宽存储的cnn加速器 |
CN113657580A (zh) * | 2021-08-17 | 2021-11-16 | 重庆邮电大学 | 基于微环谐振器和非易失性相变材料的光子卷积神经网络加速器 |
CN113657580B (zh) * | 2021-08-17 | 2023-11-21 | 重庆邮电大学 | 基于微环谐振器和非易失性相变材料的光子卷积神经网络加速器 |
WO2023045114A1 (zh) * | 2021-09-22 | 2023-03-30 | 清华大学 | 存算一体芯片及数据处理方法 |
CN113837373A (zh) * | 2021-09-26 | 2021-12-24 | 清华大学 | 数据处理装置以及数据处理方法 |
WO2023103149A1 (zh) * | 2021-12-06 | 2023-06-15 | 成都时识科技有限公司 | 脉冲事件决策装置、方法、芯片及电子设备 |
CN114723023A (zh) * | 2022-03-03 | 2022-07-08 | 北京大学 | 数据通信方法及系统、脉冲神经网络运算系统 |
CN114723023B (zh) * | 2022-03-03 | 2024-04-23 | 北京大学 | 数据通信方法及系统、脉冲神经网络运算系统 |
CN115545190B (zh) * | 2022-12-01 | 2023-02-03 | 四川轻化工大学 | 一种基于概率计算的脉冲神经网络及其实现方法 |
CN115545190A (zh) * | 2022-12-01 | 2022-12-30 | 四川轻化工大学 | 一种基于概率计算的脉冲神经网络及其实现方法 |
CN116167424A (zh) * | 2023-04-23 | 2023-05-26 | 深圳市九天睿芯科技有限公司 | 基于cim的神经网络加速器、方法、存算处理系统与设备 |
CN116205274A (zh) * | 2023-04-27 | 2023-06-02 | 苏州浪潮智能科技有限公司 | 一种脉冲神经网络的控制方法、装置、设备及存储介质 |
CN116663632A (zh) * | 2023-08-02 | 2023-08-29 | 华中科技大学 | 感存算一体智能感知系统 |
CN116663632B (zh) * | 2023-08-02 | 2023-10-10 | 华中科技大学 | 感存算一体智能感知系统 |
Also Published As
Publication number | Publication date |
---|---|
TW202123032A (zh) | 2021-06-16 |
CN113033759A (zh) | 2021-06-25 |
TWI774147B (zh) | 2022-08-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021115262A1 (zh) | 脉冲卷积神经网络算法、集成电路、运算装置及存储介质 | |
Ambrogio et al. | Equivalent-accuracy accelerated neural-network training using analogue memory | |
Xue et al. | 24.1 A 1Mb multibit ReRAM computing-in-memory macro with 14.6 ns parallel MAC computing time for CNN based AI edge processors | |
Peng et al. | DNN+ NeuroSim: An end-to-end benchmarking framework for compute-in-memory accelerators with versatile device technologies | |
JP6995131B2 (ja) | 抵抗型処理ユニットアレイ、抵抗型処理ユニットアレイを形成する方法およびヒステリシス動作のための方法 | |
Bong et al. | 14.6 A 0.62 mW ultra-low-power convolutional-neural-network face-recognition processor and a CIS integrated with always-on haar-like face detector | |
Amir et al. | 3-D stacked image sensor with deep neural network computation | |
WO2021088248A1 (zh) | 基于忆阻器的神经网络的并行加速方法及处理器、装置 | |
CN105144699A (zh) | 阈值监测的有条件重置的图像传感器 | |
Abedin et al. | Mr-pipa: An integrated multilevel rram (hfo x)-based processing-in-pixel accelerator | |
Geng et al. | An on-chip layer-wise training method for RRAM based computing-in-memory chips | |
Zhao et al. | Crossbar-level retention characterization in analog RRAM array-based computation-in-memory system | |
Bose et al. | A 75kb SRAM in 65nm CMOS for in-memory computing based neuromorphic image denoising | |
Peng et al. | Inference engine benchmarking across technological platforms from CMOS to RRAM | |
CN113936717A (zh) | 一种复用权重的存算一体电路 | |
CN110263926B (zh) | 基于光电计算单元的脉冲神经网络及其系统和运算方法 | |
Chen et al. | WRAP: Weight RemApping and processing in RRAM-based neural network accelerators considering thermal effect | |
Choi et al. | Implementation of an On-Chip Learning Neural Network IC Using Highly Linear Charge Trap Device | |
Khokhar et al. | Nanoscale memristive crossbar circuits for approximate edge detection in smart cameras | |
CN111667064B (zh) | 基于光电计算单元的混合型神经网络及其运算方法 | |
US20230289577A1 (en) | Neural network system, high density embedded-artificial synaptic element and operating method thereof | |
CN116049094B (zh) | 一种基于光电存算一体单元的多阈值配置装置及其方法 | |
Narayanan et al. | Neuromorphic technologies for next-generation cognitive computing | |
US20230292533A1 (en) | Neural network system, high efficiency embedded-artificial synaptic element and operating method thereof | |
Nägele et al. | Charge based mixed-signal multiply-accumulate circuit for energy efficient in-memory computing |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20898013 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20898013 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20898013 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 27.01.2023) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20898013 Country of ref document: EP Kind code of ref document: A1 |