JP6561877B2 - Arithmetic processing unit - Google Patents

Description

  The present invention relates to an arithmetic processing device.

  2. Description of the Related Art Conventionally, there has been considered an arithmetic processing device that executes arithmetic operations using a neural network in which a plurality of processing layers are hierarchically connected. Particularly in arithmetic processing devices that perform image recognition, a so-called convolutional neural network (CNN) is at the core.

Japanese Patent No. 5184824

  In a conventional convolutional neural network, a convolution operation process is executed on a plurality of different calculation result data obtained by the previous hierarchy, that is, feature value extraction result data, an activation process is executed, and a pooling process is executed. Therefore, higher-dimensional feature amounts are extracted. Further, by performing normalization processing on the processing result data by the pooling processing, the feature amount recognition rate can be improved, and the feature amount extraction processing can be performed more preferentially.

  By the way, when performing normalization processing on processing result data, conventionally, first, data is read from the memory at the time of convolution operation processing, convolution operation processing and pooling processing are performed on the read data, and the operation result data is Store in memory. After that, in the same processing hierarchy, the data stored in the memory is read, and the read data is normalized. That is, in the prior art, it is necessary to read data from the memory twice in the same processing hierarchy, which causes a large latency.

  Accordingly, an object of the present invention is to suppress the occurrence of latency by improving the configuration for performing normalization processing in an arithmetic processing device that realizes arithmetic processing using a neural network.

  An arithmetic processing apparatus according to the present invention is an arithmetic processing apparatus that performs an arithmetic operation using a neural network in which a plurality of processing layers are hierarchically connected, and includes a convolution arithmetic processing unit, a pooling processing unit, an integration processing unit, and a statistical processing unit. And a normalization processing unit. The convolution operation processing unit performs a convolution operation process on the input data input from the previous layer. The pooling processing unit performs a pooling process on the processing result data by the convolution operation unit. The integration processing unit integrates the processing result data by the pooling processing unit before the calculation of the next layer is started. The statistical processing unit calculates an average value and a standard deviation of the data integrated by the integration processing unit. When the calculation of the next layer is started, the normalization processing unit uses the average value and the standard deviation calculated by the statistical processing unit in the previous layer, and the processing result data output by the pooling processing unit in the previous layer Normalization processing is executed for.

  According to this configuration, the normalization process can be performed using the statistical value obtained during the calculation process of the previous layer in the input part of the calculation process in the next layer. Therefore, it is not necessary to read data from the memory twice in the same processing hierarchy, and the occurrence of latency can be suppressed.

A diagram conceptually showing a configuration example of a convolutional neural network The figure which shows the flow of arithmetic processing in the middle layer visually (the 1) A diagram visually showing the flow of arithmetic processing in the intermediate layer (Part 2) Diagram showing general arithmetic expressions and functions used for feature extraction processing 1 is a block diagram schematically showing a configuration example of an arithmetic processing device according to a first embodiment. Diagram showing an example of a normalization function The figure which shows visually the flow of the arithmetic processing by the arithmetic processing unit The block diagram which shows roughly the structural example of the arithmetic processing unit which concerns on 2nd Embodiment. The figure which shows an example of the subtraction type | formula and division formula which comprise a normalization process The block diagram which shows roughly the structural example of the arithmetic processing unit which concerns on 3rd Embodiment. The figure which shows the sampling example of the process result data of a pooling process part The block diagram which shows roughly the structural example of the arithmetic processing apparatus which concerns on 4th Embodiment. The figure which shows an example of the processing result data of a pooling process part by a binary number Block diagram (part 1) schematically showing a configuration example of an arithmetic processing unit according to a fifth embodiment Block diagram schematically showing a configuration example of an arithmetic processing unit according to the fifth embodiment (No. 2) Diagram showing an example of a nonlinear activation function FIG. 6 is a block diagram schematically showing a configuration example of an arithmetic processing unit according to the sixth embodiment.

Hereinafter, a plurality of embodiments of an arithmetic processing device will be described with reference to the drawings. In each embodiment, substantially the same elements are denoted by the same reference numerals, and description thereof is omitted.
(neural network)
FIG. 1 conceptually shows the configuration of a neural network, in this case a convolutional neural network, applied to arithmetic processing units 10, 20, 30, 40, 50, and 60, which will be described in detail later. That is, the convolutional neural network N is applied to an image recognition technique for recognizing a predetermined shape or pattern from the image data D1 that is input data, and includes an intermediate layer Na and a total coupling layer Nb. The intermediate layer Na has a configuration in which a plurality of feature quantity extraction processing layers Na1, Na2,. Each feature amount extraction processing layer Na1, Na2,... Includes a convolution layer C and a pooling layer P, respectively.

  Next, the flow of processing in the intermediate layer Na will be described. As illustrated in FIG. 2, in the first feature amount extraction processing layer Na1, the arithmetic processing unit scans the input image data D1 for each predetermined size by, for example, raster scanning. A plurality of feature amounts included in the input image are extracted by performing a known feature amount extraction process on the scanned data. Note that the first feature amount extraction processing layer Na1 extracts relatively simple single feature amounts such as a linear feature amount extending in the horizontal direction and a linear feature amount extending in the oblique direction. At this time, the arithmetic processing device generates a plurality of feature maps respectively corresponding to the plurality of features included in the input image.

  In the second feature amount extraction processing layer Na2, the arithmetic processing unit scans the input data input from the preceding feature amount extraction processing layer Na1 for each predetermined size by, for example, raster scanning. A plurality of feature amounts included in the input image are extracted by performing a known feature amount extraction process on the scanned data. In addition, in the feature amount extraction processing layer Na2 of the second layer, by integrating the spatial positional relationship of a plurality of feature amounts extracted by the feature amount extraction processing layer Na1 of the first layer, Extract higher-dimensional composite features. At this time, the arithmetic processing device generates a plurality of feature maps respectively corresponding to the plurality of features included in the input image.

  In the third feature quantity extraction processing layer Na3, the arithmetic processing unit scans the input data input from the previous feature quantity extraction processing layer Na2 for each predetermined size by, for example, raster scanning. A plurality of feature amounts included in the input image are extracted by performing a known feature amount extraction process on the scanned data. The feature extraction processing layer Na3 of the third layer is integrated by considering the spatial positional relationship of a plurality of feature amounts extracted by the feature extraction processing layer Na2 of the second layer, Extract higher-dimensional composite features. At this time, the arithmetic processing device generates a plurality of feature maps respectively corresponding to the plurality of features included in the input image. In this way, by repeating the feature amount extraction processing by the plurality of feature amount extraction processing layers, the arithmetic processing device performs image recognition of the detection target object included in the image data D1.

  The arithmetic processing unit extracts various feature amounts included in the input image data D1 in a high dimension by repeating the processing by the plurality of feature amount extraction processing layers Na1, Na2, Na3... In the intermediate layer Na. Then, the arithmetic processing unit outputs the result obtained by the processing of the intermediate layer Na to the all coupling layer Nb as intermediate operation result data.

  The total coupling layer Nb combines a plurality of intermediate calculation result data obtained from the intermediate layer Na and outputs final calculation result data. That is, the total connection layer Nb combines a plurality of intermediate operation result data obtained from the intermediate layer Na, and further performs a sum-of-products operation while varying the weighting coefficient for the combined result, thereby obtaining a final operation. Result data, that is, image data in which the detection target included in the image data D1 as input data is recognized is output. At this time, the part where the value of the result of the product-sum operation is large is recognized as a part or all of the detection target.

  Next, a flow of feature amount extraction processing by the arithmetic processing device will be described. As illustrated in FIG. 3, the arithmetic processing device uses a predetermined size for the input data Dn input from the feature extraction processing layer in the previous hierarchy, in this case, according to the filter size for each 3 × 3 pixel indicated by hatching in the figure. Scan. Note that the pixel size is not limited to 3 × 3 pixels, and can be appropriately changed, for example, 5 × 5 pixels.

  The arithmetic processing unit performs a known convolution operation on the scanned data. Then, the arithmetic processing device performs a well-known activation process on the data after the convolution operation, and outputs the result to the convolution layer C. Then, the arithmetic processing unit performs a well-known pooling process on the output data Cn of the convolution layer C at a predetermined size, in this case, 2 × 2 pixels, and outputs the result to the pooling layer P. Then, the arithmetic processing device outputs the output data Pn of the pooling layer P to the feature amount extraction processing layer of the next layer. The pixel size is not limited to 2 × 2 pixels and can be changed as appropriate.

  Here, the arithmetic processing unit performs a well-known normalization process on the pooled data Pn1, Pn2,. ... and then output to the next layer. Thereby, the pooling data Pn can be output to the next layer in a more unified format. Therefore, the recognition rate of the feature amount can be improved, and the feature amount extraction process can be performed more preferentially. In each embodiment described later, the arithmetic processing device is improved in the configuration for performing this normalization process.

  FIG. 4 shows general examples of a convolution function used for convolution operation processing, a function used for activation processing, and a function used for pooling processing. That is, the convolution function Yij is a function that accumulates values obtained by multiplying the output Xij of the immediately preceding layer by the weighting factors Wp, q obtained by learning. Note that “N” indicates a pixel size to be processed by one cycle of convolution operation processing. That is, for example, when the pixel size of one calculation cycle is “3 × 3” pixels, the value of N is “2”. Further, the convolution function Yij may be a function for adding a predetermined bias value to the accumulated value. Various functions can be adopted as the convolution function as long as it is a function capable of multiply-accumulate operation that can cope with all-join processing. For the activation process, a well-known logistic sigmoid function, ReLU function (Rectified Linear Units), or the like is used. For the pooling process, a known maximum pooling function that outputs a maximum value of input data, a known average pooling function that outputs an average value of input data, or the like is used.

  According to the convolutional neural network N described above, the processing by the convolution layer C and the processing by the pooling layer P are repeated, so that higher-dimensional feature amounts can be extracted. Next, a plurality of embodiments according to an arithmetic processing device to which the convolutional neural network N is applied will be described. In the drawings according to the embodiments, the arithmetic processing device that performs the processing of the nth layer is indicated by a solid line, and the arithmetic processing device that performs the processing of the (n + 1) th layer that is the next layer is indicated by a two-dot chain line. It shows by. In addition, a calculation block that performs processing of the nth layer is indicated by a solid line, and a calculation block that performs processing of the (n + 1) th layer that is the next layer is indicated by a two-dot chain line.

(First embodiment)
The arithmetic processing device 10 illustrated in FIG. 5 includes a convolution arithmetic processing unit 11, a pooling processing unit 12, an integration processing unit 13, a statistical processing unit 14, and a normalization processing unit 15. The convolution operation processing unit 11 performs a well-known convolution operation process on the input data input from the previous layer, and outputs the processing result data to the pooling processing unit 12. The arithmetic processing device 10 performs a known activation process on the processing result data of the convolution arithmetic processing unit 11 by an activation processing unit (not shown) and outputs the result to the pooling processing unit 12. The pooling processing unit 12 performs a well-known pooling process on the processing result data input from the convolution operation processing unit 11, and outputs the processing result data to the normalization processing unit 15 at the next layer arithmetic processing. It is supposed to be.

  The integration processing unit 13 integrates the processing result data output from the pooling processing unit 12 before the calculation processing in the next layer is started. The statistical processing unit 14 calculates the average value and standard deviation of the data integrated by the integration processing unit 13, that is, the processing result data output by the pooling processing unit 12. The normalization processing unit 15 includes a subtractor, a divider, and the like (not shown), and when arithmetic processing in the next layer is started, the average value and standard deviation calculated by the statistical processing unit 14 in the previous layer are used. A known normalization process is performed on the processing result data output by the pooling processing unit 12 in the previous hierarchy. The normalization processing unit 15 performs normalization processing based on, for example, the normalization function shown in FIG.

  According to the arithmetic processing unit 10, as illustrated in FIG. 7, the normalization processing is performed using the statistical value obtained during the arithmetic processing in the previous layer in the input portion of the arithmetic processing in the next layer. Therefore, it is not necessary to read out data from the memory twice in the same processing hierarchy, and normalization processing can be performed in a pipeline over two consecutive layers. Therefore, the processing speed can be increased and the occurrence of latency can be suppressed.

(Second Embodiment)
The arithmetic processing device 20 illustrated in FIG. 8 includes a convolution arithmetic processing unit 21, a pooling processing unit 22, an integration processing unit 23, a statistical processing unit 24, and a normalization processing unit 25. The normalization processing unit 25 includes a subtraction processing unit 25a and a division processing unit 25b.

  The subtraction processing unit 25a calculates the average value calculated by the statistical processing unit 24 in the previous hierarchy based on the subtraction formula (1) illustrated in FIG. 9 from the processing result data output by the pooling processing unit 22 in the previous hierarchy. Subtract. The convolution operation processing unit 21 performs a well-known convolution operation process on the processing result data output from the subtraction processing unit 25a. The division processing unit 25b divides the processing result data output from the convolution operation processing unit 21 by the standard deviation calculated by the statistical processing unit 24 in the previous hierarchy based on the division formula (2) illustrated in FIG.

  In a configuration in which convolution operation processing is performed after performing normalization processing including subtraction and division, a division circuit corresponding to the number of data input to the normalization processing unit is necessary, and there is a problem that the circuit scale increases. is there. According to the arithmetic processing unit 20, a normalization process including subtraction is performed, a convolution operation process is performed on the processing result data, and a normalization process including division is performed on the processing result data. In the convolution operation process, there are a plurality of input data, but only one data is output. Therefore, according to the configuration of the arithmetic processing unit 20, since the division is performed after the convolution arithmetic processing, the number of division circuits can be reduced.

(Third embodiment)
The arithmetic processing device 30 illustrated in FIG. 10 includes a convolution arithmetic processing unit 31, a pooling processing unit 32, an integration processing unit 33, a statistical processing unit 34, a normalization processing unit 35, and a sampling processing unit 36. As illustrated in FIG. 11, the sampling processing unit 36 samples a part of the processing result data output from the pooling processing unit 32. Then, the sampling processing unit 36 outputs the sampled data to the integration processing unit 33. Then, the integration processing unit 33 integrates only part of the data sampled by the sampling processing unit 36, that is, the processing result data by the pooling processing unit 32. Then, the statistical processing unit 34 calculates the average value and the standard deviation based only on a part of the processing result data by the pooling processing unit 32 accumulated by the accumulation processing unit 33.

  According to the arithmetic processing unit 30, the average value and the standard deviation are calculated based on only a part rather than all of the processing result data by the pooling processing unit 32 and used for the normalization process. In this way, it is possible to perform normalization processing with sufficiently high accuracy even with statistical values based only on part of the processing result data by the pooling processing unit 32. Moreover, according to the arithmetic processing unit 30, the processing load of the integration processing unit 33 can be reduced.

(Fourth embodiment)
The arithmetic processing device 40 illustrated in FIG. 12 includes a convolution arithmetic processing unit 41, a pooling processing unit 42, an integration processing unit 43, a statistical processing unit 44, a normalization processing unit 45, a right shift processing unit 46, and a left shift processing unit 47. Prepare. The right shift processing unit 46 is provided between the pooling processing unit 42 and the integration processing unit 43, shifts the processing result data by the pooling processing unit 42 to the right by a predetermined bit, and outputs it to the integration processing unit 43. The left shift processing unit 47 is provided between the integration processing unit 43 and the statistical processing unit 44, shifts the processing result data by the integration processing unit 43 to the left by a predetermined bit, and outputs it to the statistical processing unit 44.

  The number of bits when the right shift processing unit 46 shifts the data to the right is the same as the number of bits when the left shift processing unit 47 shifts the data to the left. Therefore, as illustrated in FIG. 13, the integration processing unit 43 is configured to integrate only the upper predetermined bits of the processing result data by the pooling processing unit 42. According to the arithmetic processing unit 40, the integration processing unit 43 integrates not only all bits of the processing result data by the pooling processing unit 42 but only the upper predetermined bits. According to this configuration, the processing load of the integration processing unit 43 can be reduced.

(Fifth embodiment)
The arithmetic processing device 50 illustrated in FIG. 14 includes a plurality of arithmetic blocks 500 each including a convolution arithmetic processing unit 51, a pooling processing unit 52, an integration processing unit 53, a statistical processing unit 54, and a normalization processing unit 55. FIG. 14 shows only one calculation block 500. Each calculation block 500 includes a weight adjustment processing unit 56. The weight adjustment processing unit 56 specifies a weighting coefficient indicating the maximum value among the weighting coefficients used when the convolution operation processing unit 51 of each operation block 500 executes the convolution operation processing. Then, the weight adjustment processing unit 56 divides the weighting factor used by the convolution operation processing unit 51 by the absolute value of the maximum value of the identified weighting factor. As a result, the weight adjustment processing unit 56 adjusts the weighting coefficient so that it falls within the range of −1 to 1.

  The arithmetic processing device 50 specifies the maximum value of the weighting coefficient used when each of the plurality of convolution arithmetic processing units 51 executes the convolution arithmetic processing, and uses the absolute value of the specified maximum value to determine the plurality of convolution arithmetic processing units 51. Is divided to adjust the weighting factor to a range of −1 to 1. By keeping the weighting factor within the predetermined range in this way, the processing result data of the convolution calculation process calculated using the weighting factor can be within the predetermined range, and blurring of the processing result data is suppressed. Can do. Therefore, the processing result data can be expressed by a fixed-point number whose expression range is narrower than that of the floating-point number, and the calculation by the fixed-point number enables faster and more accurate calculation.

  As illustrated in FIG. 15, the arithmetic processing device 50 may further include a storage unit 57 and a multiplication processing unit 58. In this configuration, the storage unit 57 stores the absolute value of the maximum value of the weighting coefficient specified by the weight adjustment processing unit 56. The multiplication processing unit 58 is constituted by, for example, a multiplier, and multiplies the processing result data output from the convolution operation processing unit 51 by the absolute value of the maximum value of the weighting coefficient stored in the storage unit 57 to activate the multiplication result. To the processing unit 59. The activation processing unit 59 performs a well-known activation process on the input processing result data, and outputs the processing result data to the pooling processing unit 52.

  This configuration is particularly effective when the activation function used for the activation process is non-linear. That is, in the processing result data by the non-linear activation function illustrated in FIG. 16, when the amount of change in the processing result near the processing result R1 when the weighting factor of the convolution operation is not adjusted and the weighting factor of the convolution operation are adjusted. The amount of change in the processing result in the vicinity of the processing result R2 is different. Therefore, the arithmetic processing unit 50 multiplies the processing result data R2 when the weighting factor is adjusted by the absolute value of the maximum value used when the weighting factor is adjusted, as shown by the arrow r. The processing result data of the arithmetic processing is returned to the processing result data R1 when the weighting coefficient is not adjusted. As a result, it is possible to approximate the processing result data to the processing result data when the weighting factor is not adjusted while performing the fast and accurate convolution operation with the adjusted weighting factor, and particularly the activation function. Even if is non-linear, the influence can be suppressed.

(Sixth embodiment)
An arithmetic processing device 60 illustrated in FIG. 17 includes a plurality of arithmetic blocks 600 each including a convolution arithmetic processing unit 61, a pooling processing unit 62, an integration processing unit 63, and a normalization processing unit 65. The calculation block 600 includes an addition processing unit 66 and a storage unit 67, respectively. In addition, the statistical processing unit 64 is provided in any one of the plurality of calculation blocks 600, in this case, the most upstream calculation block 600. For convenience of explanation, the lower side of the figure is defined as the downstream side, and the upper side of the figure is defined as the upstream side.

  The addition processing unit 66 adds the data input from the calculation block 600 downstream of itself to the data input from the integration processing unit 63 that constitutes the same calculation block 600 as that of the addition processing unit 66 and outputs the result to the storage unit 67. To do. The storage unit 67 stores the data input from the addition processing unit 66 and outputs the data to the addition processing unit 66 of the arithmetic block 600 on the upstream side of itself. The storage unit 67 of the most upstream calculation block 600 outputs the stored data to the statistical processing unit 64. The statistical processing unit 64 calculates an average value and a standard deviation for the input data and outputs them to the normalization processing unit 65 of each calculation block 600. That is, the statistical processing unit 64 calculates the average value and standard deviation of the data calculated by the plurality of integration processing units 63 and provides them to the plurality of normalization processing units 65, respectively.

  According to the arithmetic processing device 60, in a configuration including a plurality of arithmetic blocks 600, the statistical processing unit 64 is not provided in each arithmetic block 600, but the statistical processing unit 64 is provided only in one arithmetic block 600, and the statistical The statistical value output from the processing unit 64 is shared by the plurality of normalization processing units 65. According to this configuration, even if a plurality of operation blocks 600 are provided, the statistical processing unit 64 having a relatively large circuit scale can be suppressed to one, and the entire apparatus can be made compact and low in cost. Can be planned.

(Other embodiments)
Note that the present invention is not limited to the above-described embodiments, and can be applied to various embodiments without departing from the gist thereof. For example, you may implement combining several embodiment mentioned above suitably.

  In the drawings, 10, 20, 30, 40, 50, 60 are arithmetic processing units, 11, 21, 31, 41, 51, 61 are convolution arithmetic processing units, 12, 22, 32, 42, 52, 62 are pooling processes. , 13, 23, 33, 43, 53, 63 are integration processing units, 14, 24, 34, 44, 54, 64 are statistical processing units, and 15, 25, 35, 45, 55, 65 are normalization processing units. , 25a denotes a subtraction processing unit, 25b denotes a division processing unit, 56 denotes a weight adjustment processing unit, 500 denotes a calculation block, and 600 denotes a calculation block.

Claims (7)

An arithmetic processing device (10, 20, 30, 40, 50, 60) that executes an arithmetic operation using a neural network in which a plurality of processing layers are hierarchically connected,
A convolution operation processing unit (11, 21, 31, 41, 51, 61) for executing convolution operation processing on input data input from the previous layer;
A pooling processing unit (12, 22, 32, 42, 52, 62) for performing a pooling process on the processing result data by the convolution operation processing unit;
An integration processing unit (13, 23, 33, 43, 53, 63) for integrating the processing result data by the pooling processing unit before the calculation of the next layer is started;
A statistical processing unit (14, 24, 34, 44, 54, 64) for calculating an average value and a standard deviation of data integrated by the integration processing unit;
When calculation of the next layer is started, normalization processing is performed on the processing result data output by the pooling processing unit in the previous layer, using the average value and the standard deviation calculated by the statistical processing unit in the previous layer A normalization processing unit (15, 25, 35, 45, 55, 65) for executing
An arithmetic processing device comprising: The normalization processing unit (25) includes a subtraction processing unit (25a) and a division processing unit (25b).
The subtraction processing unit subtracts the average value calculated by the statistical processing unit (24) in the previous hierarchy from the processing result data output by the pooling processing unit (22) in the previous hierarchy,
The convolution operation processing unit (21) performs a convolution operation processing on the processing result data output from the subtraction processing unit,
2. The arithmetic processing device according to claim 1, wherein the division processing unit divides the processing result data output from the convolution arithmetic processing unit by a standard deviation calculated by the statistical processing unit (24) in a previous hierarchy.   The arithmetic processing unit according to claim 1 or 2, wherein the integration processing unit (33) integrates a part of processing result data obtained by the pooling processing unit (32).   The arithmetic processing unit according to any one of claims 1 to 3, wherein the integration processing unit (43) integrates predetermined high-order bits of processing result data by the pooling processing unit (42). A plurality of calculation blocks (600) including the convolution calculation processing unit (61), the pooling processing unit (62), the integration processing unit (63), and the normalization processing unit (65) are provided.
The arithmetic processing unit according to claim 1, wherein the statistical processing unit (64) calculates an average value and a standard deviation of data calculated by the plurality of integration processing units (63). A plurality of calculation blocks (500) including the convolution calculation processing unit (51), the pooling processing unit (52), the integration processing unit (53), and the normalization processing unit (55) are provided,
The plurality of convolution arithmetic processing units respectively specify the maximum value of the weighting coefficient used when executing the convolution arithmetic processing, and the weighting factor used by the plurality of convolution arithmetic processing units is divided by the absolute value of the specified maximum value The arithmetic processing unit according to any one of claims 1 to 5, further comprising a weight adjustment processing unit (56) for performing the processing.   A multiplication processing unit (58) for multiplying the calculation result data calculated by the convolution calculation processing unit (51) using the weighting coefficient adjusted by the weight adjustment processing unit by the absolute value of the maximum value of the weighting coefficient. The arithmetic processing apparatus according to claim 6.

Images