TWI630544B - Operation device and method for convolutional neural network - Google Patents

Abstract

A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data to output an accumulated data; performing a one-bit shift operation on the accumulated data to output a shift data; and performing shift data on the shift data a weighting operation to output a weighted data, wherein a factor of the weighting operation is based on the number of input data, the number of bits shifted to the right in the bit shift operation, and a scaling weight of a subsequent layer of the convolutional neural network And set.

Description

Operation device and method for convolutional neural network

The present invention relates to a method of computing a convolutional neural network, and more particularly to an apparatus and method for performing an average pooling operation.

The Convolutional Neural Network (CNN) is a feedforward neural network that typically contains multiple sets of convolution layers and pooling layers. The pooling layer can perform maximum pooling or average pooling operations on specific features on a certain area of the input data to reduce the amount of parameters and operations in the neural network. In terms of the average pooling operation, the conventional method is to perform the addition operation first, and then divide the total result. However, the division operation requires a lot of processor performance, so it is easy to cause excessive burden on the hardware resources. Further, when an accumulation operation of a plurality of data is performed, an overflow situation is likely to occur.

Therefore, how to provide a pooling operation method, which can perform average pooling operation with less processor performance, is one of the current important topics.

In view of the above, an object of the present invention is to provide a convolution operation device and a pooling operation method, which can avoid overloading hardware resources and improve the performance of the pooling operation.

A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data to output an accumulated data; performing a one-bit shift operation on the accumulated data to output a shift data; and performing shift data on the shift data a weighting operation to output a weighted data, wherein a factor of the weighting operation is based on the number of input data, the number of bits shifted to the right in the bit shift operation, and a scaling weight of a subsequent layer of the convolutional neural network And set.

In an embodiment, the weighting operation factor is shifted with the scaling weight and the bit The number of bits shifted to the right in the calculation is proportional, and the weighting operation factor is inversely proportional to the amount of input data, and the weighted data is equal to the shift data multiplied by the factor.

In one embodiment, the number of bits shifted to the right in the bit shift operation depends on the size of a pooled window, and the amount of input data depends on the size of the pooled window.

In one embodiment, the subsequent layer is a one-level convolution layer of the convolutional neural network, the scaling weight is a filter coefficient of the next-level convolution layer, and the addition and bit shift operations are one of the convolutional neural networks. The operation in the pooling layer.

In one embodiment, the division operation in the pooling layer is performed in a multiplication operation of the next layer convolutional layer.

A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data in a pooling layer to output an accumulated data; and performing a weighting operation on the accumulated data in a subsequent layer to output a weighting Data, wherein a factor of the weighting operation depends on the number of input data and a scaling weight of subsequent layers, the weighting data being equal to the accumulated data multiplied by a factor.

In an embodiment, the subsequent layer is a one-layer convolution layer, the scaling weight is a filter coefficient, and the weighting operation is a convolution operation, and the weighting operation factor is equal to the filter coefficient divided by the number of input data.

In an embodiment, the amount of input data depends on the size of the pooling window.

A method for computing a convolutional neural network includes: multiplying a scaled weight by an original filter coefficient to produce a weighted filter coefficient; and volumeizing an input data and weighted filter coefficients in a convolution layer Product operation.

In an embodiment, the operation method further comprises: performing a one-bit shift operation on the input data; and inputting the input data after the bit shift operation to the convolution layer, wherein the scaling weight is based on an original scaling weight And the number of bits shifted to the right in the bit shift operation.

An arithmetic device for a convolutional neural network capable of performing the aforementioned method.

As described above, in the arithmetic device and the arithmetic method of the present invention, the average pooling operation is performed in two stages, and the pooling unit performs only the addition operation, and is combined with the bit shift operation to avoid data overflow caused by the accumulation process. Then, the output result of the pooling unit is weighted to obtain the final average result. Since the pooling unit does not perform the division operation, the processor can be prevented from using more performance, thereby achieving the effect of improving the performance of the pooling operation.

1‧‧‧ memory

2‧‧‧buffering device

21‧‧‧Memory Control Unit

3‧‧‧Convolutional computing module

4‧‧‧Interlaced summing unit

5‧‧‧Additional buffer unit

51‧‧‧Partial total block

52‧‧‧ pooling unit

6‧‧‧ coefficient acquisition controller

7‧‧‧Control unit

71‧‧‧ instruction decoder

72‧‧‧Data Read Controller

ADD_IN, RD, WR‧‧‧ lines

C1~Cn‧‧‧ data

DMA‧‧‧direct memory access

F1~Fn, WF1~WFn‧‧‧ filter coefficients

P1~Pn‧‧‧ data

W‧‧‧Scale weight

Figure 1 is a schematic illustration of a portion of a convolutional neural network.

Figure 2 is a schematic diagram of the integration operation of a convolutional neural network.

Figure 3 is a schematic diagram of a convolutional neural network.

4 is a functional block diagram of a convolution operation device in accordance with an embodiment of the present invention.

The convolution operation apparatus and method according to the specific embodiments of the present invention will be described with reference to the accompanying drawings, wherein the same elements will be described with the same element symbols, and the drawings are for illustrative purposes only and are not intended to limit the present invention. .

Figure 1 is a schematic illustration of a portion of a convolutional neural network. Referring to FIG. 1 , the convolutional neural network has multiple operation layers, such as a convolution layer, a pooling layer, etc., and the number of layers of the convolution layer and the pooling layer may be multiple layers, and the output of each layer may be regarded as another layer or subsequent. The input of the layer, for example, the output of the Nth layer convolutional layer is the input of the Nth layer pooling layer or the input of other subsequent layers, and the output of the Nth layer pooling layer is the input of the N+1th layer convolution layer or other For the input of the subsequent layer, the output of the Nth operation layer may be the input of the N+1th operation layer.

In order to improve the performance of the operation, different layers but close to each other can be properly integrated. For example, the pooling operation of the pooling layer is an average pooling operation, and the division operation can be integrated into the next layer of the operation layer. A layer of operation is, for example, a convolutional layer, that is, the average pooled division of the pooled layer is computed with the convolutional multiplication of the next layer of convolutional layers. In addition, the pooling layer can also perform shift operations instead of the division required for the average calculation, and integrate the unfinished parts into the next-level operation layer, that is, the average pooling division of the pooling layer fails. The part that is completely replaced by the shift operation is operated together with the convolution multiplication of the next layer of the convolutional layer.

Figure 2 is a schematic diagram of the integration operation of a convolutional neural network. Referring to FIG. 2, in the convolutional layer, a plurality of data P1~Pn and a plurality of filter coefficients F1~Fn are convoluted to generate a plurality of data C1~Cn, and data C1~Cn are used as a pooling layer. Input data. In the pooling layer, a plurality of input data are subjected to an addition operation to output an accumulated data. In a subsequent layer, the accumulated data is subjected to a weighting operation to output a weighted data, wherein one of the weighting operations The scaling weight W is dependent on the number of input data and a scaling weight of the subsequent layer, the weighting data being equal to the accumulated data multiplied by the scaling weight W.

For example, the subsequent layer may be a next-level convolution layer, the scaling weight is a filter coefficient, and the weighting operation is a convolution operation, and the weighting operation factor is equal to the filter coefficient divided by the amount of input data. In addition, the amount of input data depends on the size of the pooling window.

On the other hand, the accumulated data can be partially shifted by the shift operation before another layer is calculated. For example, the accumulated data may perform a one-bit shift operation to output a shift data, and then perform a weighting operation on the shift data to output a weighted data, wherein a weighted weight W of the weighting operation is based on the amount of input data. The number of bits shifted to the right in the bit shift operation and a scaling weight of a subsequent layer of the convolutional neural network. The weighted weight W of the weighting operation is proportional to the scale weight and the number of bits shifted to the right in the bit shift operation. The weighted weight W of the weighting operation is inversely proportional to the number of input data, and the weighted data is equal to the shift data. Multiply by the scale weight W.

The number of bits shifted to the right in the bit shift operation depends on the size of a pooled window. Shifting one bit to the right is equivalent to dividing by 2, and if the number of bits shifted to the right is n, Then the nth power of 2 is the closest but not exceeding the scale power of the pooling window. Taking a 2×2 pooling window as an example, the size of the pooling window is 4, n is 2, and 2 bits are shifted to the right; taking a 3×3 pooling window as an example, the size of the pooling window is 9 , n is 3, shifting 3 bits to the right.

The amount of input data depends on the size of the pooling window. The subsequent layer is a one-layer convolutional layer of the convolutional neural network, and the scaling weight is a filter coefficient of the next-layer convolutional layer. The addition operation and the bit shift operation are operations in a pooled layer of the convolutional neural network. .

For example, if there are 9 data in a feature area to be averaged, the first 9 data may be accumulated to obtain an accumulated value. To avoid overflow of the accumulated value, The accumulated value performs a bit shift operation, for example, shifting the accumulated value to the right by two bits to obtain a shift value, which is equivalent to dividing the accumulated value by 4, and multiplying the shift value by one. The weighting coefficients are obtained to obtain a weighted value. The selection of the weighting coefficient is determined according to the offset of the bit shift. In the present embodiment, the weighting coefficient is 1/2.25, so the weighting value finally obtained is equivalent to the effect of dividing the accumulated value by 9. Since the bit shift operation and the weighting operation do not occupy too many processing procedures, the processor can be used by the bit shifting and weighting two-stage operation. With less performance, the average pooling operation can be performed, thereby improving the performance of the pooling operation.

Figure 3 is a schematic diagram of the integration operation of a convolutional neural network. Referring to FIG. 3, the convolution operation of the convolution layer is to multiply the input data by the filter coefficients. When the input data needs to be weighted or scaled, the weighted or scaled operation can be integrated in the convolution operation. . That is, the input weighting (or scaling) and convolution operations of the convolutional layer can be done in the same multiplication operation.

The data P1~Pn input to the convolutional layer may be the pixel of the image or the output of the upper layer of the convolutional neural network, for example, the previous layer of the pooling layer, the hidden layer, and the like. In FIG. 3, the operation method of the convolutional neural network includes: multiplying a scaling weight W by the original filter coefficients F1 to Fn to generate weighted filter coefficients WF1 to WFn; and a convolution layer pair The input data P1 to Pn and the weighted filter coefficients WF1 to WFn are convoluted. The original convolution operation is that the input data P1~Pn are multiplied with the original filter coefficients F1~Fn. In order to integrate the weighting or scaling operations, the coefficients used in the actual operation of the convolutional layer are the weighted filter coefficients WF1~WFn. Rather than the original filter coefficients F1~Fn. The input of the convolutional layer is not weighted or scaled without additional multiplication operations.

In addition, when the weighting or scaling needs to be divided, or the weighted or scaled value is less than 1, the operation method may first perform a one-bit shift operation on the input data, and then input the input data after the bit shift operation to the volume. Laminated. The scaling weight W is based on an original scaling weight and the number of bits shifted to the right in the bit shift operation. For example, the original scaling weight is 0.4, and the bit shifting operation is set to shift 1 bit to the right (equivalent to multiplying by 0.5), and then the scaling weight W is set to 0.8, so that the entire operation result will still be equivalent. The input data is multiplied by the original scaling weight (0.5*0.8=0.4). In addition, changing the division operation to shift operation can reduce the hardware burden, and the input of the convolution layer is not additionally weighted or scaled by multiplication.

4 is a functional block diagram of a convolution operation device in accordance with an embodiment of the present invention. Referring to FIG. 3, the convolution operation device includes a memory 1, a buffer device 2, a convolution operation module 3, an interleaving and summing unit 4, a total buffer unit 5, and a coefficient extraction controller. 6 and a control unit 7. The convolutional computing device can be used in the application of the Convolutional Neural Network (CNN).

The memory 1 stores data to be convoluted, and can be, for example, an image, a video, or a sound. Frequency, statistics, convolutional neural network, one layer of data, and so on. In the case of image data, for example, pixel data; in the case of video data, for example, pixel data of a video frame or a motion vector, or audio in a video; In terms of data, it is usually a two-dimensional array of data, and in the case of image data, it is usually a two-dimensional array of pixel data. In addition, in the embodiment, the memory 1 is a static random-access memory (SRAM), which can store the volume in addition to the data to be convoluted. The data is completed by the product, and may have a multi-layered storage structure and separately store the data to be calculated and operated. In other words, the memory 1 can be used as a cache memory inside the convolution operation device.

In practical applications, all or most of the data may be stored elsewhere, for example, in another memory, and another memory may be selected as dynamic random access memory (DRAM) or other types. Memory. When the convolution operation device is to perform the convolution operation, the data is completely or partially loaded into the memory 1 by another memory, and then the data is input to the convolution operation module 3 through the buffer device 2 to perform the volume. Product operation. If the input data is streaming data, the memory 1 will write the latest stream data for convolution at any time.

The buffer device 2 is coupled to the memory 1, the convolution operation module 3, and the total buffer unit 5. Further, the buffer device 2 is also coupled to other elements of the convolution operation device, such as the interleaving summing unit 4 and the control unit 7. In addition, for frame data calculation of image data or video, the order of processing is to read multiple columns at the same time, so in one clock, the buffer device 2 is from the memory 1 The data on the different columns of the same row are input. For this, the buffer device 2 of the present embodiment functions as a buffer device for a column buffer. When the calculation is to be performed, the buffer device 2 can first extract the data required by the convolution operation module 3 from the memory 1, and adjust the data to be successfully written into the convolution operation module 3 after the capture. Data type. On the other hand, since the buffer device 2 is also coupled to the summing buffer unit 5, the data after the calculation of the total buffer unit 5 is also reordered by the buffer device 2 and then transferred back to the memory 1 for storage. . In other words, the buffer device 2 has a function similar to relaying temporary data in addition to the function of line buffering, or the buffer device 2 can be used as a data register having a sorting function.

It is worth mentioning that the buffer device 2 further includes a memory control unit 21, When the buffer device 2 performs data capture or writing with the memory 1, it can be controlled via the memory control unit 21. In addition, since it has a limited memory access width with the memory 1, or is also called bandwidth or bandwidth, the convolution operation module 3 can actually perform convolution operations with the memory. 1 is related to the access width. In other words, the computational efficiency of the convolutional computing module 3 is limited by the aforementioned access width. Therefore, if there is a bottleneck in the input of the memory 1, the performance of the convolution operation will be affected by the impact and fall.

The convolution operation module 3 has a plurality of convolution units, each convolution unit performs a convolution operation based on a filter and a plurality of current data, and retains part of the current data after the convolution operation. The buffer device 2 takes a plurality of new data from the memory 1, and inputs the new data to the convolution unit, and the new data does not overlap with the current data. The convolution unit of the convolution operation module 3 performs a second round convolution operation based on the filter, the retained current data, and the new data. The interleaving summation unit 4 is coupled to the convolution operation module 3 to generate a feature output result according to the result of the convolution operation. The summing buffer unit 5 is coupled to the interleaving summing unit 4 and the buffer device 2 to temporarily store the feature output result; wherein, when the convolution operation of the specified range is completed, the buffer device 2 will temporarily store all the data from the summing buffer unit 5. Write to memory 1.

The coefficient acquisition controller 6 is coupled to the convolution operation module 3, and the control unit 7 is coupled to the buffer device 2. In actual application, for the convolution operation module 3, the input source required for the convolution operation module 3 needs to input a coefficient of a filter in addition to the data itself, and the operation is started. The coefficient input of the 3 × 3 convolution unit array is referred to in the present embodiment. The coefficient acquisition controller 6 can directly input the filter coefficients from the external memory by direct memory access DMA (direct memory access). In addition to being coupled to the convolution operation module 3, the coefficient capture controller 6 can also be coupled to the buffer device 2 to accept various commands from the control unit 7, so that the convolution operation module 3 can be controlled by the control unit 7. The control coefficient capture controller 6 performs input of filter coefficients.

Control unit 7 can include an instruction decoder 71 and a data read controller 72. The instruction decoder 71 obtains a control instruction from the data read controller 72 and decodes the instruction, thereby obtaining the size of the current input data, the number of rows of the input data, the number of columns of the input data, the feature number of the input data, and the input data in the memory. The starting address in body 1. In addition, the instruction decoder 71 can also obtain information about the type of the filter and the output characteristics from the data reading controller 72. The number is output and the appropriate vacant signal is output to the buffer device 2. The buffer device 2 operates according to the information provided by the instruction decoding, and further controls the operation of the convolution operation module 3 and the summation buffer unit 5, for example, data is input from the memory 1 to the buffer device 2 and the convolution operation module. The timing of 3, the scale of the convolution operation of the convolution operation module 3, the read address of data from the memory 1 to the buffer device 2, the write address of the data from the sum buffer unit 5 to the memory 1, and the volume The convolution mode operated by the product computing module 3 and the buffer device 2.

On the other hand, the control unit 7 can also extract the required control commands and convolution information from the external memory by means of direct memory access DMA. After the instruction decoder 71 decodes the instructions, the control commands and The convolution information is retrieved by the buffer device 2, and the instruction may include the stride size of the moving window, the address of the moving window, and the number of image data rows and columns of the feature to be extracted.

The summing buffer unit 5 is coupled to the interleaving summing unit 4, and the summing buffer unit 5 includes a part of the summing block 51 and a pooling unit 52. The partial summing block 51 temporarily stores the data output by the interleaving summing unit 4. The pooling unit 52 performs a pooling operation on the data temporarily stored in the partial summing block 51. Pooling operations are either pooled or averaged.

For example, the summed buffer unit 5 may temporarily store the result of the convolution calculation via the convolution operation module 3 and the output feature result of the interleaving summation unit 4 in the partial summing block 51. Then, the pooling unit 52 performs a pooling operation on the data temporarily stored in the partial summing block 51. The pooling operation may take an average value or take a specific feature on a certain area of the input data. The maximum value is used as a summary feature extraction or statistical feature output. This statistical feature not only has a lower dimension than the previous feature, but also improves the processing result of the operation.

It should be noted that the temporary storage here still temporarily stores the partial operation result in the input data and then temporarily stores it in the partial summing block 51, so it is called a partial summing area. Block 51 and summing buffer unit 5, or simply referred to as PSUM unit and PSUM BUFFER module. On the other hand, the pooling operation of the pooling unit 52 of the present embodiment can obtain the statistical feature output by using the above-described average pooling calculation method. After all the data to be input are processed by the convolution operation module 3 and the interleave summation unit 4, the total buffer unit 5 outputs the final data processing result, and the result can be restored to the memory through the buffer device 2 as well. Body 1, or re-transmission through memory 1 to other components. At the same time, the convolution operation module 3 and the interleaving summation unit 4 continue to perform data feature acquisition and calculation to improve the volume. The processing performance of the product computing device.

In the case of the above average pooling, the filter coefficients of the convolution layer originally in the memory need to be adjusted, and the actual input to the convolution operation module 3 is an adjusted factor, and the factor may be the foregoing. The factors used in the operation of integrating the pooling layer and the next convolution layer have been described in the foregoing embodiments since the generation of the factors has been described, and therefore will not be described. When the convolution operation device is processing the convolution layer and the pooling layer of the current layer, the pooling unit 52 may not process the average pooled division portion in the current layer pooling layer, and the convolution operation device processes the next convolution layer. The convolution operation module 3 then integrates the average pooled division portion of the previous pooling unit 52 that has not been processed into the convolution multiplication operation. On the other hand, when the convolution operation device processes the convolution layer of the current layer and the pooling layer, the pooling unit 52 can use the shift operation as a partial division, but leaves a division portion that has not been completely averaged, When the convolution operation device processes the next convolution layer, the convolution operation module 3 integrates the average pooled division portion that has not been processed by the previous pooling unit 52 into the convolution multiplication operation.

The convolution operation device may include a plurality of convolution operation modules 3, and the convolution unit of the convolution operation module 3 and the interleave summation unit 4 are capable of selectively operating in a low-scale convolution mode and a high-scale convolution mode. In the low-scale convolution mode, the interleaving summing unit 4 is configured to interleave the results of the respective convolution operations in the corresponding order in the convolution operation module 3 to output a total result. In the high-scale convolution mode, the interleaving summing unit 4 interleaves the results of the convolution operations of the respective convolution units as outputs.

In summary, in the arithmetic device and the calculation method of the present invention, the average pooling operation is performed in two stages, and the pooling unit performs only the addition operation, and is matched with the bit shift operation to avoid the data overflow caused by the accumulation process. Then, the output result of the pooling unit is weighted to obtain the final average result. Since the pooling unit does not perform the division operation, the processor can be prevented from using more performance, thereby achieving the effect of improving the performance of the pooling operation.

The above-mentioned embodiments are not intended to limit the invention, and any equivalent modifications and variations of the present invention are intended to be included within the scope of the appended claims.

Claims (10)

A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data to output an accumulated data; performing a one-bit shift operation on the accumulated data to output a shift data; and shifting the shift data; The data performs a weighting operation to output a weighted data, wherein a factor of the weighting operation is based on the number of the input data, the number of bits shifted to the right in the bit shift operation, and a subsequent step of the convolutional neural network. a scaling weight of the layer; wherein the factor of the weighting operation is proportional to the scaling weight and the number of bits shifted to the right in the bit shifting operation, the factor of the weighting operation being associated with the input The amount of data is inversely proportional to the weighted data multiplied by the factor. A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data to output an accumulated data; performing a one-bit shift operation on the accumulated data to output a shift data; and shifting the shift data; The data performs a weighting operation to output a weighted data, wherein a factor of the weighting operation is based on the number of the input data, the number of bits shifted to the right in the bit shift operation, and a subsequent step of the convolutional neural network. The weight of the layer depends on a scale weight; wherein the number of bits shifted to the right in the bit shift operation depends on the size of a pooled window, and the number of the input data depends on the size of the pool window. . A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data to output an accumulated data; performing a one-bit shift operation on the accumulated data to output a shift data; and shifting the shift data; The data performs a weighting operation to output a weighted data, wherein a factor of the weighting operation is based on the number of the input data, the number of bits shifted to the right in the bit shift operation, and a subsequent step of the convolutional neural network. a scaling weight of the layer; wherein the subsequent layer is a primary convolution layer of the convolutional neural network, the scaling weight is a filter coefficient of the convolution layer of the next layer, the addition operation and the bit shift The operation is an operation in a pooled layer of the convolutional neural network. The arithmetic method according to claim 3, wherein the division operation in the pooling layer is performed in a multiplication operation of the next layer convolution layer. A method for computing a convolutional neural network, comprising: performing an addition operation on a plurality of input data in a pooling layer to output an accumulated data; and performing a weighting operation on the accumulated data in a subsequent layer to output a Weighting data, wherein a factor of the weighting operation is determined according to the number of the input data and a scaling weight of the subsequent layer, the weighting data being equal to the accumulated data multiplied by the factor. The operation method of claim 5, wherein the subsequent layer is a layer-by-one convolution layer, the scaling weight is a filter coefficient, and the weighting operation is a convolution operation, and the factor of the weighting operation is equal to the filtering The factor is divided by the number of such input data. The method of claim 5, wherein the amount of the input data is determined according to the size of the pooling window. A method for computing a convolutional neural network, comprising: multiplying a scaled weight by an original filter coefficient to generate a weighted filter coefficient; and pairing a convolution with an input data and the weighted filter coefficient Perform a convolution operation. The operation method of claim 8, wherein the operation method further comprises: performing a one-bit shift operation on the input data; and inputting the input data after the bit shift operation to the convolution layer, wherein The scaling weight is based on an original scaling weight and the number of bits shifted to the right in the bit shift operation. An arithmetic unit of a convolutional neural network, which performs the arithmetic method according to any one of the first to ninth aspects of the patent application.