TWI672643B - Full index operation method for deep neural networks, computer devices, and computer readable recording media - Google Patents

Abstract

A full exponential calculation method applied to a deep neural network, which pre-normalizes and quantizes the weight value in the weight matrix of each neuron of the deep neural network into a quantized weight that can be expressed by an exponent 2 ^-i Value, and in advance normalize and quantize the complex pixel values of an image data to be input into the deep neural network into quantized pixel values that can be expressed by exponents 2 ^-j , and then input the quantized pixel values The deep neural network enables each neuron in the first layer of the deep neural network to use the quantized weight matrix, exponential multiplier, exponential adder, and exponential subtractor to scroll the quantized pixel values Product operation, thereby reducing the computational complexity and circuit complexity of the deep neural network, increasing the computing speed of the deep neural network and reducing the occupation of memory space.

Description

Full exponential calculation method applied to deep neural network, computer device and computer readable recording medium

The invention relates to a full exponential calculation method, in particular to a full exponential calculation method applied to a deep neural network.

The deep neural network is a method of deep learning in machine learning. By imitating the mathematical model of the biological nervous system, it continuously provides a large amount of data for multiple calculations and training of different levels and structures, that is, it can train the most An optimized and most effective data recognition model. As shown in FIG. 1, a deep neural network generally includes an input layer 11, an output layer 12, and a hidden layer 13 connecting the input layer 11 and the output layer 12 and located between the input layer 11 and the output layer 12 , The hidden layer 13 is composed of a plurality of layers 14 connected to each other, wherein each layer 14 has a plurality of neurons 141, and each neuron 141 has a weight value W as shown in FIG. 2 (for example, 3x3)的 weight matrix 10. The data D input by the input layer 11, for example, an image 20 with 5x5 pixels D shown in FIG. 2 is input to each neuron 141 of the first layer 14 in the hidden layer 11, and each of the neurons Element 141 will move the weight matrix 10 in sequence on the image, and at each passing position of the weight matrix 10, let the weights in the weight matrix 10 The value W is multiplied by the overlapping (corresponding) partial pixels D on the image 20 and summed (ie, convolution operation) to obtain a eigenvalue R. Each neuron 141 then passes the weight matrix 10 through all of the image 20 The eigenvalues R calculated after the position are output to each neuron 141 of the second layer 14, so that each neuron 141 of the second layer 14 performs the convolution operation on the input eigenvalues R as described above And output the operation result to each neuron 141 of the next layer 14 and so on, until each neuron 141 of the last layer 141 outputs the operation result to the output layer 12.

However, the weight value W and the input data D in the weight matrix 10 are usually expressed as floating point numbers, so the neuron 141 needs to use a floating point multiplier to multiply the weight value W and the input data D by floating point Since the multiplication operation of floating-point numbers is large and the multiplication operation is relatively more complicated than the addition operation, the multiplication operation of floating-point numbers takes more time than the addition operation. Moreover, the volume of the floating-point multiplier realized by the logic circuit is larger than that of the adder, so that the deep neural network using the floating-point multiplier is implemented as a hardware circuit with a relatively large volume. In addition, the weight value W of the deep neural network and the final output operation result are stored as floating point numbers, so it will occupy a large amount of memory space.

Therefore, the purpose of the present invention is to provide a full exponential calculation method applied to a deep neural network, and a computer device and a computer-readable recording medium implementing the method, which can reduce the calculation amount of the deep neural network, Operation complexity and circuit Complexity, as well as reducing memory space usage and increasing the speed of deep neural network operations.

Therefore, the present invention is a full exponential calculation method applied to a deep neural network, wherein the deep neural network is carried in a computer device and has a hidden layer composed of a plurality of layers connected to each other, and each of the hidden layers One layer has a plurality of neurons, and each neuron has a weight matrix composed of m (m is an integer and m ≧ 1) weight values; the method includes: a preprocessing module of the computer device pre-processes the deep nerve The m weight values of each neuron in the network are normalized, so that the m normalized weight values fall within the range of -1 to +1, and each of the normalized weight values is quantized to an index of 2 ^-i represents the quantized weight value, and the preprocessing module expresses the multiple groups of the quantized weight value in X bits; the preprocessing module will input an image data of the deep neural network in advance The normalization of the complex pixel values of, so that the normalized pixel values fall within the range of -1 to +1, and each of the normalized pixel values is quantized into a quantized graph that can be expressed by an index of 2 ^-j Prime values, and the pre-processing module expresses the multiple groups of the quantized pixel values in Y bits; the pre-processing module inputs the quantized pixel values into the hidden layer of the deep neural network In the first layer of the first layer, each neuron of the first layer performs a convolution operation on the quantized pixel values with the quantized weight matrix, and in each convolution operation, each neuron uses An exponential multiplier multiplies the m quantized weight values 2- ⁱ by the overlapping quantized pixel values 2- ^{j in the} quantized pixel values to obtain m products, where the exponential multiplier The formula for calculating each product is: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ (2 ^N -1) and i + j ≦ (2 ^N -1); , If i + j > (2 ^N -1) or i == (2 ^N -1) or j == (2 ^N -1); where X = Y then N = X, if X ≠ Y then N takes The larger of X and Y; and in each convolution operation, each neuron inputs the partial product of the equal product m which is a positive value into an exponential adder to accumulate to obtain a positive accumulated value 2 ^-p , and additionally input the partial product of the negative product m into the first exponential adder to obtain a negative accumulated value 2 ^-q , and then add the positive accumulated value 2 ^-p and the negative accumulated value 2 ^-q index input a subtractor subtracting a characteristic value obtained R < wherein, the index adder calculates two index 2 ^-a 2 ^-b added and the formula is: 2 ^-a +2 ^-b = 2 ^{- c} , if a ≠ b, then c takes the smaller of a and b; 2 ^-a +2 ^-b = 2 ^{-a + 1} , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} <2 ⁰ ; 2 ^-a +2 ^-b = 2 ⁰ , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} 2 ⁰ ; 2 ^-a +2 ^-b = 2 ^-a , if b == (2 ^N -1); where, the formula of the exponential subtractor to calculate the eigenvalue r is: r = 2 ^-p -2 ^-q = 2 ^-p if p ≦ q-1 or q == (2 ^N -1); , If p == q; r = 2 ^-p -2 ^-q = -2 ^-q , if p == q + 1; r = 2 ^-p -2 ^-q = 2 ^-p , if q == p + 1; r = 2 ^-p -2 ^-q = -2 ^-q , if q ≦ p-1 or p == (2 ^N -1); where == is a comparison operator to compare the variable on the left side of == Is it equal to the variable on the right side of ==; and i, j, p, q, r, a, b are positive integers.

In some embodiments of the present invention, each of the neurons will also perform the rectification operation on the eigenvalues r generated by all convolution operations through a linear rectification function, and generate corresponding rectified eigenvalues r ′ , And input the rectified eigenvalue r ′ to the neurons in the next layer connected to it, so that each neuron in the next layer uses the quantized weight matrix, the exponential multiplier, the exponential adder and the The exponential subtractor performs a convolution operation on the rectified characteristic value r ′.

In some embodiments of the present invention, the linear rectification (Rectified Linear Unit, ReLU) function may be a ramp function, a leaky linear rectification (Leaky ReLU) function, a randomized linear rectification (Randomized Leaky ReLU) function, and a noise linear rectification (Noisy ReLU) One of the functions.

Furthermore, a computer device for realizing the above method according to the present invention includes a deep neural network and a preprocessing module, wherein the deep neural network has a hidden layer composed of a plurality of interconnected layers, the hidden layer Each layer in has a plurality of neurons, and each neuron has a weight matrix composed of m (m is an integer and m ≧ 1) weight values, an exponential multiplier, an exponential adder, and an exponential subtractor; The pre-processing module pre-normalizes the m weight values of each neuron of the deep neural network so that the m normalized weight values fall within the range of -1 to +1, and Each of the normalized weight values is quantized into a quantized weight value that can be represented by an exponent 2 ^-i , and the preprocessing module represents a plurality of groups of the quantized weight values in X bits; and the preprocessing module Normalize the complex pixel values of an image data to be input into the deep neural network in advance, so that the normalized pixel values fall within the range of -1 ~ + 1, and each normalized pixel value Quantized into quantized pixel values that can be represented by exponents 2 ^-j , and the pre-processing module expresses Y-bits of the plurality of groups of quantized pixel values; and the pre-processing module quantizes these After the pixel values are input to the first layer of the hidden layer of the deep neural network, each neuron of the first layer performs a convolution operation on the quantized pixel values with the quantized weight matrix, and In each convolution operation, each neuron uses the exponential multiplier to divide the m quantized weight values 2 ^-i with the quantized pixel values that overlap the quantized pixel values 2 ^-j Multiply to obtain m products, where the exponential multiplier calculates each product as: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ (2 ^N -1) and i + j ≦ (2 ^N -1); , If i + j > (2 ^N -1) or i == (2 ^N -1) or j == (2 ^N -1); where X = Y then N = X, if X ≠ Y then N takes The larger of X and Y; and in each convolution operation, each neuron inputs the positive product of the equal product m into the exponential adder and accumulates it to obtain a positive accumulated value 2 ^-p , and input the partial product of the negative value in the product m into the exponential adder to accumulate to obtain a negative accumulated value 2 ^-q , and then the positive accumulated value 2 ^-p and the negative accumulated value 2 ^-q Enter the exponential subtractor to get a eigenvalue r; where, the exponential adder calculates the formula for adding two exponents 2 ^-a and 2 ^-b : 2 ^-a +2 ^-b = 2 ^-c , if a ≠ b, then c takes the smaller of a and b; 2 ^-a +2 ^-b = 2 ^{-a + 1} , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} <2 ⁰ ; 2 ^-a +2 ^-b = 2 ⁰ , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} 2 ⁰ ; 2 ^-a +2 ^-b = 2 ^-a , if b == (2 ^N -1); where, the formula of the exponential subtractor to calculate the eigenvalue r is: r = 2 ^-p -2 ^-q = 2 ^-p if p ≦ q-1 or q == (2 ^N -1); , If p == q; r = 2 ^-p -2 ^-q = -2 ^-q , if p == q + 1; r = 2 ^-p -2 ^-q = 2 ^-p , if q == p + 1; r = 2 ^-p -2 ^-q = -2 ^-q , if q ≦ p-1 or p == (2 ^N -1); where == is a comparison operator to compare the variable on the left side of == Is it equal to the variable on the right side of ==; and i, j, p, q, r, a, b are positive integers.

In some embodiments of the present invention, the deep neural network and the preprocessing module are software programs stored in a storage unit of the computer device and can be read and executed by a processing unit of the computer device .

In some embodiments of the present invention, the deep neural network and / or the pre-processing module are integrated into a special application integrated circuit chip or a programmable logic circuit device of the computer device, or are burned Firmware recorded in a microprocessor of the computer device.

In addition, the present invention is a computer-readable recording medium that implements the above method, in which a software program including a deep neural network and a preprocessing module is stored The deep neural network has a hidden layer composed of a plurality of interconnected layers, each layer in the hidden layer has a plurality of neurons, and each neuron has a number of m (m is an integer and m ≧ 1) A weight matrix, an exponential multiplier, an exponential adder and an exponential subtractor composed of weight values, and after the software program is loaded and executed by a computer device, the computer device can complete the application as described above in depth Full exponential calculation method of neural network.

The effect of the present invention is: by pre-normalizing and quantifying the weight values in the weight matrix of the neurons in the deep neural network and the image data prepared for input to the deep neural network respectively Into the exponents 2 ^-i and 2 ^-j with a base of 2, and then input the quantized pixel values of the image data into the hidden layer of the deep neural network so that each neuron in the first layer of the hidden layer The quantized weight matrix is convoluted, and the exponent multiplier, exponential adder, and exponential subtractor in each neuron are used to perform simple addition, subtraction, and judgment on the input exponent instead of the conventional floating-point multiplication Computing, and reducing the complexity of neurons and completing convolution operations quickly, not only improves the speed of deep neural networks, but also effectively simplifies and reduces the circuit volume of deep neural networks implemented as hardware. And when the deep neural network is implemented by software program, it can also effectively improve its calculation speed.

10‧‧‧ weight matrix

11‧‧‧ input layer

12‧‧‧Output layer

13‧‧‧ hidden layer

14‧‧‧ storey

141‧‧‧ Neuron

20‧‧‧Image

4‧‧‧Computer device

41‧‧‧Storage unit

42‧‧‧Processing unit

43‧‧‧ Deep Neural Network

44‧‧‧Pretreatment module

51‧‧‧Exponential multiplier

52‧‧‧ Index Adder

53‧‧‧Index subtractor

W‧‧‧ weight value

D‧‧‧ pixels

S1 ~ S3‧‧‧Step

Other features and functions of the present invention will be clearly shown in the embodiments with reference to the drawings, in which: Figure 1 is a schematic diagram of the basic composition architecture of a conventional deep neural network; Figure 2 illustrates the process of a neuron performing a convolution operation on an input data with a weight matrix; Figure 3 is a full exponent of the present invention applied to a deep neural network The main flow chart of an embodiment of the calculation method; FIG. 4 is a block diagram of main components of an embodiment of the computer device of the present invention that implements the method of FIG. 3; FIG. 5 illustrates each nerve in the deep neural network of this embodiment The element has an exponential multiplier, an exponential adder, and an exponential subtractor; and FIG. 6 illustrates the weight value in the neuron of the deep neural network and the input to the deep neural network by the preprocessing module of this embodiment. The process of data normalization and quantification.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are denoted by the same numbers.

Referring to FIG. 3, it is the main flow of an embodiment of the full exponential arithmetic method of the present invention applied to a deep neural network, which is executed by a computer device 4 shown in FIG. 4, the computer device 4 mainly includes a storage unit 41 ( A computer-readable recording medium) and a processing unit 42, the storage unit 41 stores a program including a deep neural network 43 and a pre-processing module 44, and the program can be used by the computer device 4 Processing unit 42 Load and execute, and complete the method flow shown in Figure 3, but not limited to this. That is, the deep neural network 43 and / or the pre-processing module 44 can also be integrated into an application-specific integrated circuit (abbreviated as ASIC) chip or a programmable chip of the computer device 4 In a logic device (Programmable Logic Device, abbreviated as PLD), the special application integrated circuit chip or the programmable logic circuit device can complete the method flow shown in FIG. 3. And the special application integrated circuit chip or the programmable logic circuit device is the processing unit 42 of this embodiment; or the deep neural network 43 and / or the preprocessing module 44 may also be burned in the The firmware in a microprocessor of the computer device 4 enables the microprocessor to execute the firmware to complete the method flow shown in FIG. 3, and the microprocessor is the processing unit 42 of this embodiment.

As shown in FIG. 1, in addition to an input layer 11 and an output layer 12, the deep neural network 43 of this embodiment also has a plurality of back-to-back connections between the input layer 11 and the output layer 12. Hidden layer 14 composed of layers 141, each layer 141 in the hidden layer 14 has a plurality of neurons 141, and each neuron 141 has a weight matrix composed of m (m is an integer and m ≧ 1) weight values For example, a weight matrix 10 having 3 × 3 weight values W shown in FIG. 2 and an exponential multiplier 51, an exponential adder 52, and an exponential subtractor 53 shown in FIG.

In order to reduce the computational complexity and computational complexity of the deep neural network, as shown in step S1 of FIG. 3, the preprocessing module 44 pre-regulates the m weight values W of each neuron 141 of the deep neural network 43 to be regular To make the m normalized weight values fall within the range of -1 to +1, and quantize each normalized weight value with log ₂ into a quantized weight value that can be expressed by the index 2 ^-i , and the The preprocessing module 44 divides the quantized weight values into a plurality of groups, and expresses these groups in X bits.

For example, as shown in FIG. 6 (a), assuming that the weight values W have values distributed between -2, -1.5, 0, 0.5, 1, 1.5, etc., between -2 and 1.5, the preprocessing mode Group 44 divides these equal weight values W by 2 to normalize, so that the normalized weight values (-1, -0.75, 0, 0.25, 0.5, 0.75) fall within the range of -1 to +1, As shown in FIG. 6 (b); then, each of the normalized weight values (-1, -0.75, 0, 0.25, 0.5, 0.75) is quantized by log ₂ into a quantized weight value expressed by an exponent 2- ⁱ . Therefore, the absolute values of the four normalized weight values of 1, 0.5, 0.25, and 0 will be represented by 2 ⁰ , 2 ^-1 , 2 ^-2, and 2 ^-3 , and since the base 2 index cannot represent 0.75, the thus quantized to the nearest 0.75 to 1 and ²⁰ to represent. Therefore, the preprocessing module 44 divides the absolute values of these quantized weight values into four groups (that is, 2 ⁰ , 2 ^-1 , 2 ^-2, and 2 ^-3 ), and expresses these groups in 2 bits , That is, 00, 01, 10, and 11 respectively represent 2 ⁰ , 2 ^-1 , 2 ^-2, and 2 ^-3 , and additionally record (mark) the quantized weight value as a positive or negative value.

Therefore, assuming that the normalized weight values falling between -1 and +1 also contain positive or negative 0.125, 0.0625, 0.03125, and 0.015625 or values close to these values, the quantized weight value expressed by the index 2 ^-i respectively as ^{2-4, 2-5, 2-6} and ^2-7, the need to use pre-processing module 44 to 3 yuan ²⁰ ~ ^2-7 represents a total of eight groups, i.e. 000,001 , 010, 011, 100, 101, 110, 111 represent 2 ⁰ , 2 ^-1 , 2 ^-2 ... 2 ^{-7 respectively} . Similarly, if the quantized weight value also includes a value less than 2 ^-7 , the preprocessing module 44 needs to use 4 or more bits to represent more than 8 quantized weight values ( Absolute value) group, and so on.

Then, as shown in step S2 of FIG. 3, the pre-processing module 44 pre-normalizes an image data to be input to the deep neural network 43, for example, an image 20 with 5x5 pixels D shown in FIG. 2 is normalized to make the The pixel value of the normalized pixel falls within the range of -1 to +1, and each normalized pixel value is quantized by log ₂ into a quantized pixel value that can be expressed by the index 2 ^-j , and The pre-processing module 44 divides the quantized pixel values into a plurality of groups, and represents the groups in Y bits. Similarly, the example shown in Figure 6, if the pixel values are quantized in FIG. ^{20, 2-1, 2-2} and ^2-3 of the four absolute values of the group, the preprocessing module 44 will be 2 Bits represent these groups, that is, 00, 01, 10, and 11 represent 2 ⁰ , 2 ^-1 , 2 ^-2, and 2 ^-3 , respectively, and additionally record the quantized pixel values as positive or negative values. If the pixel values after quantization also contain smaller absolute values (for example, 0.125, 0.0625, 0.03125, and 0.015625 or less), the preprocessing module 44 will be represented by 3 bits or more ²⁰ ~ ^2-7, or eight groups of more than 8 or more groups, i.e., to represent the ²⁰ 000,001,010,011,100,101,110,111, ^2-1, 2 ^{- 2} ... 2 ^-7 , and so on. Therefore, the number of bits used to represent the quantized weight value 2 ^-i and the quantized pixel value 2 ^-j may be the same or different, depending on the quantized weight value 2 ^-i and the quantized pixel value 2 ^-j The absolute value depends on the number of groups. In addition, step S1 and step S2 have no order, and can also be performed synchronously.

Then, as shown in step S3 of FIG. 3, the pre-processing module 44 inputs the quantized pixel values 2- ^j to the first layer 14 of the hidden layer 13 through the input layer 11 of the deep neural network 43, so that Each neuron 141 of the first layer 14 performs a convolution operation on the quantized pixel values 2- ^j with the quantized weight matrix 10. That is, as shown in FIG. 2, the neuron 141 will sequentially move the quantized weight matrix 10 one unit (pixel) at a time on the image 20, and each of the quantized weight matrix 10 After passing the position, multiply the quantized weight value 2- ⁱ in the quantized weight matrix 10 by the quantized pixel value 2- ^j in the overlapping (corresponding) part of the image 20 and add the products together ( That is, convolution operation) to obtain a feature value r.

In each convolution operation, each neuron 141 uses the exponential multiplier 51 to quantize the overlapping portions of the m quantized weight values 2- ⁱ and the quantized pixel values 2- ^j The pixel values 2- ^{j are} multiplied to obtain m products, where the formula of the exponential multiplier 51 to calculate each product is: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ (2 ^N -1) and i + j ≦ (2 ^N -1); , If i + j > (2 ^N -1) or i == (2 ^N -1) or j == (2 ^N -1); where, if X = Y then N = X, if X ≠ Y then N Take the larger of X and Y; for example, if the absolute value group number of the quantized weight value 2 ^-i and the quantized pixel value 2 ^-j are 4 or less, then X = Y = 2, N = 2, the formula of the exponential multiplier 51 to calculate each product is: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ 3 and i + j ≦ 3; 2 ^-i × 2 ^{- j} = 2 ^-3 if i + j > 3 or i == 3 or j == 3; and if the quantized weight value 2 ^{-i has} an absolute value group number of 4 or less, but the absolute value of the quantized pixel values 2 ^-j group is between 5 and 7, then X = 2, Y = 3, N = 3, the index multiplier 51 calculates a product of each formula was: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ 7 and i + j ≦ 7; 2 ^{- i} × 2 ^{- j} = 2 ^-7 , if i + j > 7 or i == 7 Or j == 7; it can be seen that, in the case of i , j ≠ (2 ^N -1) and i + j ≦ (2 ^N -1), the exponential multiplier 51 is actually just the quantized weight value 2 ^-i and the quantized pixel value 2 ^-j exponents i, j are added, that is, the product of the two is not required, and no multiplication is required, and i + j > (2 ^N -1) or i == ( 2 ^N -1) or j == (2 ^N -1), the exponential multiplier 51 can output the quantized weight value 2 ^-i and the quantized pixel value 2 ^-j without even performing actual calculations The product of.

And in each convolution operation, each of the neurons 141 inputs the partial product of the equal product m that is a positive value into the exponential adder 52 to accumulate to obtain a positive accumulated value 2 ^-p , and then additionally The partial product of the equal product m which is a negative value is input to the first exponential adder 52 for accumulation to obtain a negative accumulated value 2 ^-q , and then the positive accumulated value 2 ^-p and the negative accumulated value 2 ^{-q are} input to the The index subtractor 53 subtracts to obtain the characteristic value r.

Wherein, the index adder 53 calculates two index (e.g. 2 ^-a product or a product and a 2 ^-b 2 ^-a accumulated value and a 2 ^-b) added to the formula: 2 ^-a +2 ^-b = 2 ^-c , if a ≠ b, then c takes the smaller of a and b; 2 ^-a +2 ^-b = 2 ^{-a + 1} , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} <2 ⁰ ; 2 ^-a +2 ^-b = 2 ⁰ , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1 2 0; 2 -a +2 -b =} 2 -a, if b == (2 ^N -1); for example, if a = 2, b = 3, then c = a, 2 ^-a +2 ^{- b} = 2 ^-a ; if a = b = 2, and N = 2, then 2 ^-a +2 ^-b = 2 ^{-a + 1} , if a == b and a ≠ 3 and 2 ^{-a + 1} < 2 ⁰ ; 2 ^-a +2 ^-b = 2 ⁰ , if a == b and a ≠ 3 and 2 ^{-a + 1} 2 ⁰ ; 2 ^-a +2 ^-b = 2 ^-a , if b == 3; where, the formula of the exponential subtractor 53 to calculate the characteristic value r is: r = 2 ^-p -2 ^-q = 2 ^-p , If p ≦ q-1 or q == (2 ^N -1); , If p == q; r = 2 ^-p -2 ^-q = -2 ^-q , if p == q + 1; r = 2 ^-p -2 ^-q = 2 ^-p , if q == p + 1; r = 2 ^-p -2 ^-q = -2 ^-q , if q ≦ p-1 or p == (2 ^N -1); for example, if p = 1, q = 3, then r = 2 ^-p -2 ^-q = 2 ^-p ; if p = q = 3 and N = 2, then r = 2 ^-p -2 ^-q = 2 ^-3 ; if p = 2, q = 1, then r = 2 ^-p -2 ^-q = -2 ^-q ; if p = 1, q = 2, then r = 2 ^-p -2 ^-q = 2 ^-q ; if q = 1, p = 3, then r = 2 ^-p -2 ^-q = -2 ^-q

Therefore, each neuron 141 passes the exponential multiplier 51 (actually only Perform addition operations), exponential adder 52 and exponential subtractor 53 to replace the conventional floating-point multiplier, and replace multiplication with simple addition and subtraction, which not only reduces the amount of operation and the operation speed is fast, but also requires a relatively simple logic circuit That is, the exponential multiplier 51, exponential adder 52, and exponential subtractor 53 can be implemented, so when the deep neural network 43 is implemented as a physical circuit, the circuit of the neuron 141 will be simplified, thereby effectively reducing the deep nerve The overall circuit volume of the network 43.

Furthermore, after each neuron 141 of the first layer 14 of the deep neural network 43 completes all convolution operations to obtain a plurality of eigenvalues r, each neuron 141 will also complete all convolution operations. The eigenvalues r are rectified by a linear rectification (Rectified Linear Unit, ReLU) function to generate a corresponding plurality of rectified eigenvalues r ′, and then input the rectified eigenvalues r ′ to the next The neurons 141 of the first layer 14 make each neuron 141 of the next layer 14 also use the quantized weight matrix 10, the exponential multiplier 51, the exponential adder 52 and the exponential subtractor 53 for the After the rectified eigenvalue r 'is subjected to a convolution operation, the eigenvalue r resulting from the completion of all convolution operations is then rectified through a linear rectification function to generate a corresponding plurality of rectified eigenvalues r', and then The rectified eigenvalue r ′ is input to the neurons 141 of the next layer 14 connected thereto, and so on, until the last layer 14 of the deep neural network 43 outputs the operation result to the output layer 12. The linear rectification function can be a ramp function, a leaky linear rectification (Leaky ReLU) function, and a random linear rectification with leakage (Randomized Leaky) Relu) function and noise linear rectification (Noisy ReLU) function is one of, but not limited to this.

And since the weight values in the image data 20 and the neuron 141 input to the deep neural network 43 have been converted from the original floating-point representation to at least 2 bits, the deep neural network 43 outputs The calculation result is also expressed in at least 2 bits, which greatly reduces the memory space occupied by the computer device 4.

In summary, in the above embodiment, the weight values W in the weight matrix 10 of each neuron 141 in the deep neural network 43 and the image data 20 prepared for input to the deep neural network 43 are normalized in advance. And quantize them into 2-based exponents 2 ^-i and 2 ^{-j respectively} , and then input the quantized pixel values of the image data 20 into the deep neural network 43 and the quantized weight matrix of each neuron 141 10 Perform convolution operation, and use the exponential multiplier 51, exponential adder 52, and exponential subtractor 53 in each neuron 141 to perform simple addition, subtraction, and judgment on the input exponent, replacing the conventional floating point The number multiplication operation reduces the computational complexity and can complete the convolution operation quickly, which not only improves the operation speed of the deep neural network 43, but also replaces the multiplier with a simple adder to effectively simplify and reduce the depth of the implemented hardware The circuit volume of the neural network 43. When the deep neural network 43 is implemented in software, since the convolution operation only needs simple addition, subtraction and judgment of the input exponent, and no multiplication operation, it can effectively improve its operation speed, and indeed achieve The effect and purpose of the invention.

However, the above are only the embodiments of the present invention. This limits the scope of implementation of the present invention, and any simple equivalent changes and modifications made in accordance with the scope of the patent application of the present invention and the content of the patent specification are still covered by the patent of the present invention.

Claims (10)

A full-exponential arithmetic method applied to a deep neural network, which is carried in a computer device and has a hidden layer composed of a plurality of layers connected to each other, and each layer in the hidden layer has a plurality of nerves Each neuron has a weight matrix composed of m (m is an integer and m ≧ 1) weight values; the method includes: a pre-processing module of the computer device pre-registers each of the deep neural networks The m weight values of the neuron are normalized so that the m normalized weight values fall within the range of -1 to +1, and each of the normalized weight values is quantized into a quantization that can be expressed by an exponent 2- ⁱ The post-weight value, and the pre-processing module expresses the plurality of groups of the quantized weight values in X bits; the pre-processing module will input the complex pixel value of an image data of the deep neural network in advance Normalization, so that the normalized pixel values fall within the range of -1 to +1, and each normalized pixel value is quantized into a quantized pixel value that can be expressed by an index of 2- ^j , and the The preprocessing module expresses the plural groups of the quantized pixel values in Y bits; the preprocessing module inputs the quantized pixel values into the first layer of the hidden layer of the deep neural network , So that each neuron of the first layer performs a convolution operation on the quantized pixel values with the quantized weight matrix, and in each convolution operation, each neuron uses an exponential multiplier to The m quantized weight values 2- ⁱ are respectively multiplied by the quantized pixel values 2- ^j overlapping in the quantized pixel values to obtain m products, wherein the exponential multiplier calculates the product of each product The formula is: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ (2 ^N -1) and i + j ≦ (2 ^N -1);

, If i + j > (2 ^N -1) or i == (2 ^N -1) or j == (2 ^N -1); where X = Y then N = X, if X ≠ Y then N takes The larger of X and Y; and in each convolution operation, each neuron inputs the partial product of the equal product m which is a positive value into an exponential adder to accumulate to obtain a positive accumulated value 2 ^-p , and additionally input the partial product of the negative product m into the first exponential adder to obtain a negative accumulated value 2 ^-q , and then add the positive accumulated value 2 ^-p and the negative accumulated value 2 ^-q index input a subtractor subtracting a characteristic value obtained R < wherein, the index adder calculates two index 2 ^-a 2 ^-b added and the formula is: 2 ^-a +2 ^-b = 2 ^{- c} , if a ≠ b, then c takes the smaller of a and b; 2 ^-a +2 ^-b = 2 ^{-a + 1} , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} <2 ⁰ ; 2 ^-a +2 ^-b = 2 ⁰ , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1}

2 ⁰ ; 2 ^-a +2 ^-b = 2 ^-a , if b == (2 ^N -1); where, the formula of the exponential subtractor to calculate the eigenvalue r is: r = 2 ^-p -2 ^-q = 2 ^-p if p ≦ q-1 or q == (2 ^N -1);

, If p == q; r = 2 ^-p -2 ^-q = -2 ^-q , if p == q + 1; r = 2 ^-p -2 ^-q = 2 ^-p , if q == p + 1; r = 2 ^-p -2 ^-q = -2 ^-q , if q ≦ p-1 or p == (2 ^N -1); where == is a comparison operator to compare the variable on the left side of == Is it equal to the variable on the right side of ==; and i, j, p, q, r, a, b are positive integers. The full exponential calculation method applied to the deep neural network as described in claim 1, wherein each of the neurons will also complete the eigenvalue r generated by all convolution operations through a linear rectification function to perform rectification operations, and generate corresponding A plurality of rectified eigenvalues r ′, and input the rectified eigenvalues r ′ to the neurons in the next layer connected to them, so that each neuron in the next layer uses the quantized weight matrix and the index The multiplier, the exponential adder and the exponential subtractor perform convolution operations on the rectified eigenvalues r ′. The full exponential calculation method applied to the deep neural network as described in claim 2, wherein the linear rectification (Rectified Linear Unit, ReLU ) function is a ramp function, a leaky linear rectification (Leaky ReLU) function, and a random linear rectification with leakage (Randomized Leaky) One of the ReLU function and the noise linear rectification (Noisy ReLU) function. A computer device, including: a deep neural network, which has a hidden layer composed of a plurality of interconnected layers, each layer in the hidden layer has a plurality of neurons, and each neuron has a number of ( m is an integer and m ≧ 1) a weight matrix composed of weight values, an exponential multiplier, an exponential adder, and an exponential subtractor; and a pre-processing module that pre-processes each nerve of the deep neural network The m weight values of the element are normalized, so that the m normalized weight values fall within the range of -1 to +1, and each normalized weight value is quantized into a quantization that can be expressed by an exponent 2 ^-i Weight value, and the preprocessing module expresses the plural groups of the quantized weight values in X bits; and the preprocessing module will input the complex pixel value of an image data of the deep neural network in advance Normalization, so that the normalized pixel values fall within the range of -1 to +1, and each normalized pixel value is quantized into a quantized pixel value that can be expressed by an index of 2- ^j , and the The preprocessing module represents the plural groups of the quantized pixel values in Y bits; and the preprocessing module inputs the quantized pixel values into the first of the hidden layers of the deep neural network Layer, so that each neuron of the first layer performs a convolution operation on the quantized pixel values with the quantized weight matrix, and in each convolution operation, each neuron uses the exponential multiplier Multiplying the m quantized weight values 2- ⁱ by the overlapping quantized pixel values 2- ^{j in the} quantized pixel values respectively to obtain m products, where the exponential multiplier calculates each product The formula is: 2 ^{- i} × 2 ^{- j} = 2- ^{( i + j )} , if i , j ≠ (2 ^N -1) and i + j ≦ (2 ^N -1);

, If i + j > (2 ^N -1) or i == (2 ^N -1) or j == (2 ^{N -} 1); where X = Y then N = X, if X ≠ Y then N takes The larger of X and Y; and in each convolution operation, each neuron inputs the positive product of the equal product m into the exponential adder and accumulates it to obtain a positive accumulated value 2 ^-p , and input the partial product of the negative value in the product m into the exponential adder to accumulate to obtain a negative accumulated value 2 ^-q , and then the positive accumulated value 2 ^-p and the negative accumulated value 2 ^-q Enter the exponential subtractor to get a eigenvalue r; where, the exponential adder calculates the formula for adding two exponents 2 ^-a and 2 ^-b : 2 ^-a +2 ^-b = 2 ^-c , if a ≠ b, then c takes the smaller of a and b; 2 ^-a +2 ^-b = 2 ^{-a + 1} , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1} <2 ⁰ ; 2 ^-a +2 ^-b = 2 ⁰ , if a == b and a ≠ (2 ^N -1) and 2 ^{-a + 1}

2 ⁰ ; 2 ^-a +2 ^-b = 2 ^-a , if b == (2 ^N -1); where, the formula of the exponential subtractor to calculate the eigenvalue r is: r = 2 ^-p -2 ^-q = 2 ^-p if p ≦ q-1 or q == (2 ^N -1);

, If p == q; r = 2 ^-p -2 ^-q = -2 ^-q , if p == q + 1; r = 2 ^-p -2 ^-q = 2 ^-p , if q == p + 1; r = 2 ^-p -2 ^-q = -2 ^-q , if q ≦ p-1 or p == (2 ^N -1); where == is a comparison operator to compare the variable on the left side of == Is it equal to the variable on the right side of ==; and i, j, p, q, r, a, b are positive integers. The computer device according to claim 4, wherein each of the neurons further performs the rectification operation on the eigenvalues r generated by performing all convolution operations, and generates corresponding rectified eigenvalues r ', And input the rectified characteristic value r' to the neurons in the next layer connected to it, so that each neuron in the next layer uses the quantized weight matrix, the exponential multiplier, the exponential adder and The exponential subtractor performs a convolution operation on the rectified characteristic value r ′. The computer device according to claim 4, wherein the linear rectification (Rectified Linear Unit, ReLU) function is a ramp function, a leakage linear rectification (Leaky ReLU) function, a random linear rectification with leakage (Randomized Leaky ReLU) function, and a noise linear rectification ( Noisy ReLU) function. The computer device according to any one of claims 4 to 6, wherein the deep neural network and the preprocessing module are stored in a storage unit of the computer device and can be used by a processing unit of the computer device Read and execute the software program. The computer device according to any one of claims 4 to 6, wherein the deep neural network and / or the pre-processing module is a special application integrated circuit chip or a programmable logic circuit integrated in the computer device In the device. The computer device according to any one of claims 4 to 6, wherein the deep neural network and / or the preprocessing module are firmware burned in a microprocessor of the computer device. A computer-readable recording medium in which a software program including a deep neural network and a preprocessing module is stored. The deep neural network has a hidden layer composed of a plurality of interconnected layers. The hidden Each layer in the layer has a plurality of neurons, and each neuron has a weight matrix composed of m (m is an integer and m ≧ 1) weight values, an exponential multiplier, an exponential adder, and an exponential subtractor After the software program is loaded and executed by a computer device, the computer device can complete the full exponential calculation method applied to the deep neural network as described in any one of the request items 1 to 3.