JP6414458B2 - Arithmetic processing unit - Google Patents

Description

  The present invention relates to an arithmetic processing device.

  2. Description of the Related Art Conventionally, there has been considered an arithmetic processing device that executes arithmetic operations using a neural network in which a plurality of processing layers are hierarchically connected. Particularly in arithmetic processing devices that perform image recognition, a so-called convolutional neural network (CNN) is at the core.

Japanese Patent No. 5184824

  This type of convolutional neural network has a structure in which convolution layers and pooling layers are alternately connected. In the convolution layer, so-called convolution calculation processing using a learned filter is performed on the input data input from the previous layer, and thereby the feature amount included in the input data is extracted. In the pooling layer following the convolution layer, the maximum or average value is output from the neighborhood of several pixels in the feature map obtained by the convolution calculation process, thereby obtaining the invariance of the response to a minute change in the feature amount. . That is, a minute change in the feature amount is absorbed.

  By the way, according to the convolutional neural network, it is possible to extract higher-dimensional feature values by repeating the process using the convolution layer and the pooling layer. However, the pooling process is a resolution reduction process in which the resolution of output data gradually decreases as the process is repeated. Therefore, as the resolution reduction process is repeated, the resolution of the output data may gradually become insufficient, and there is a concern that the position of the detection target in the input image cannot be accurately identified.

  Therefore, the present invention provides an arithmetic processing device that realizes arithmetic processing using a neural network, and accurately positions the detection target in an input image even when the resolution of data decreases as the resolution reduction processing is repeated. The purpose is to identify.

An arithmetic processing apparatus according to the present invention is an arithmetic processing apparatus that executes an operation by a neural network in which a plurality of processing layers are hierarchically connected, and includes an arithmetic processing means, a storage means, and a restoration means. The arithmetic processing means executes at least a convolution operation process and a resolution reduction process on the input data input from the previous layer. The storage unit stores position information of pixel data indicating a maximum value among pixel data indicating characteristics during the resolution reduction processing. The restoration unit specifies the position of the detection target in the input data based on the position information stored in the storage unit. When a plurality of the input data is input from the previous layer, the storage unit specifies pixel data indicating a maximum value as the characteristic pixel data at the time of the resolution reduction processing for each of the input data, The position information of the feature pixel data indicating the largest value among the plurality of feature pixel data is stored. Then, the convolution calculation process and the resolution reduction process are repeatedly executed over multiple layers while storing the position information of the pixel data indicating the maximum value in the resolution reduction process.

  According to this configuration, the position of the detection target in the input image can be accurately determined based on the position information stored in the storage unit even when the resolution of the data decreases as the resolution reduction process is repeated. Can be identified.

A diagram conceptually showing a configuration example of a convolutional neural network The figure which shows the flow of arithmetic processing in the middle layer visually (the 1) A diagram visually showing the flow of arithmetic processing in the intermediate layer (Part 2) The figure which shows an example of the general arithmetic expression and function used for the feature-value extraction process 1 is a block diagram schematically showing a configuration example of an arithmetic processing apparatus according to an embodiment. The figure which shows an example of the resolution of the data in each pooling layer The figure which shows visually an example of the memory | storage process of the positional information by a memory | storage process part The figure which shows visually an example of the memory | storage process of the positional information by a memory | storage process part over many layers The figure which shows visually an example of the correction process of the rectangular frame by a correction process part The figure which shows an example of the process which memorize | stores positional information from a some feature map visually The figure which shows visually an example of the correction process of the rectangular frame based on the positional information obtained from the several feature map

Hereinafter, an embodiment according to an arithmetic processing device will be described with reference to the drawings.
(neural network)
FIG. 1 conceptually shows the configuration of a neural network, in this case, a convolutional neural network, which is applied to an arithmetic processing unit 100 described later in detail. That is, the convolutional neural network N is applied to an image recognition technique for recognizing a predetermined shape or pattern from the image data D1 that is input data, and includes an intermediate layer Na and a total coupling layer Nb. The intermediate layer Na has a configuration in which a plurality of feature quantity extraction processing layers Na1, Na2, Na3,. Each feature amount extraction processing layer Na1, Na2, Na3,... Includes a convolution layer C and a pooling layer P, respectively.

  Next, the flow of processing in the intermediate layer Na will be described. As illustrated in FIG. 2, in the first feature amount extraction processing layer Na1, the arithmetic processing unit scans the input image data D1 for each predetermined size by, for example, raster scanning. And the feature-value contained in an input image is extracted by performing the known feature-value extraction process with respect to the scanned data. Note that the first feature amount extraction processing layer Na1 extracts relatively simple single feature amounts such as a linear feature amount extending in the horizontal direction and a linear feature amount extending in the oblique direction.

  In the second feature amount extraction processing layer Na2, the arithmetic processing unit scans the input data input from the preceding feature amount extraction processing layer Na1 for each predetermined size by, for example, raster scanning. And the feature-value contained in an input image is extracted by performing the known feature-value extraction process with respect to the scanned data. In addition, in the feature amount extraction processing layer Na2 of the second layer, by integrating the spatial positional relationship of a plurality of feature amounts extracted by the feature amount extraction processing layer Na1 of the first layer, Extract higher-dimensional composite features.

  In the third feature quantity extraction processing layer Na3, the arithmetic processing unit scans the input data input from the previous feature quantity extraction processing layer Na2 for each predetermined size by, for example, raster scanning. And the feature-value contained in an input image is extracted by performing the known feature-value extraction process with respect to the scanned data. The feature extraction processing layer Na3 of the third layer is integrated by considering the spatial positional relationship of a plurality of feature amounts extracted by the feature extraction processing layer Na2 of the second layer, Extract higher-dimensional composite features. In this way, by repeating the feature amount extraction processing by the plurality of feature amount extraction processing layers, the arithmetic processing device performs image recognition of the detection target object included in the image data D1.

  The arithmetic processing unit extracts various feature amounts included in the input image data D1 in a high dimension by repeating the processing by the plurality of feature amount extraction processing layers Na1, Na2, Na3,... In the intermediate layer Na. . Then, the arithmetic processing unit outputs the result obtained by the processing of the intermediate layer Na to the all coupling layer Nb as intermediate operation result data.

  The total coupling layer Nb combines a plurality of intermediate calculation result data obtained from the intermediate layer Na and outputs final calculation result data. That is, the total connection layer Nb combines a plurality of intermediate operation result data obtained from the intermediate layer Na, and further performs a sum-of-products operation while varying the weighting coefficient for the combined result, thereby obtaining a final operation. Result data, that is, image data in which the detection target included in the image data D1 as input data is recognized is output. At this time, the part where the value of the result of the product-sum operation is large is recognized as a part or all of the detection target.

  Next, a flow of feature amount extraction processing by the arithmetic processing device will be described. As illustrated in FIG. 3, the arithmetic processing device is configured to include a plurality of arithmetic blocks, and executes arithmetic processing by each arithmetic block. That is, the arithmetic processing unit scans the input data Dn input from the feature amount extraction processing layer of the previous hierarchy, in this case, every 5 × 5 pixels indicated by hatching in the drawing. The pixel size is not limited to 5 × 5 pixels and can be changed as appropriate.

  The arithmetic processing unit performs a known convolution operation on the scanned data. Then, the arithmetic processing unit performs a well-known activation process on the data after the convolution calculation, and outputs the result of the convolution layer C. Then, the arithmetic processing unit performs a well-known pooling process on the output data Cn1, Cn2, Cn3,... Of the convolution layer C for each predetermined size, in this case, every 2 × 2 pixels. Output. Then, the arithmetic processing unit outputs the output data Pn1, Pn2, Pn3,... Of the pooling layer P to the feature amount extraction processing layer of the next layer. The pixel size is not limited to 2 × 2 pixels and can be changed as appropriate.

  FIG. 4 shows, as a reference, general examples of a convolution function used for convolution operation processing, a function used for activation processing, and a function used for pooling processing. That is, the convolution function yj is a function that adds a predetermined bias value Bj to the sum of values obtained by multiplying the output yi of the immediately previous layer by the weighting coefficient wij obtained by learning. For the activation process, a well-known logistic sigmoid function, ReLU function (Rectified Linear Units), or other nonlinear functions are used. For the pooling process, a known maximum pooling function that outputs a maximum value of input data, a known average pooling function that outputs an average value of input data, or the like is used.

(One embodiment)
An arithmetic processing device 100 illustrated in FIG. 5 includes a convolution arithmetic processing unit 101, an activation processing unit 102, and a pooling processing unit 103. Although not shown, the arithmetic processing device 100 includes a plurality of arithmetic blocks 104 including the convolution arithmetic processing unit 101, the activation processing unit 102, and the pooling processing unit 103. The arithmetic block 104 is an example of arithmetic processing means. The arithmetic processing apparatus 100 includes a storage processing unit 105, a restoration processing unit 106, and a correction processing unit 107. The arithmetic processing device 100 may have a configuration in which each processing unit is virtually realized by software, a configuration realized by hardware, or a configuration in which a configuration by software and a configuration by hardware are mixed.

  The convolution operation processing unit 101 performs a well-known convolution operation process on the input data input from the previous layer, and outputs the processing result data to the activation processing unit 102. The activation processing unit 102 performs a well-known activation process on the processing result data by the convolution operation processing unit 101, and outputs the processing result data to the pooling processing unit 103. The pooling processing unit 103 performs a well-known pooling process on the processing result data from the activation processing unit 102 and outputs the processing result data. That is, the arithmetic block 104 is configured to perform pooling processing as the resolution reduction processing in this case.

  The storage processing unit 105 is an example of a storage unit, and stores position information of pixel data indicating characteristics during pooling processing in a storage medium (not shown). In this case, the storage processing unit 105 stores the position information of the pixel data indicating the “maximum value” during the pooling process. As the storage medium, for example, a semiconductor memory can be considered. The restoration processing unit 106 is an example of a restoration unit, and specifies the position of the detection target in the input data, that is, the input image, based on the position information stored in the storage processing unit 105. The position of the detection target specified by the correction processing unit 107 and the restoration processing unit 106 is corrected.

  As illustrated in FIG. 6, according to the convolutional neural network N, the processing by the convolution layer C and the processing by the pooling layer P are repeated, whereby higher-dimensional feature amounts can be extracted. However, as the pooling process is repeated, the data resolution gradually decreases. That is, in the example shown in FIG. 6, the resolution of the data in the second pooling layer P2 is reduced to ¼ of the resolution of the data in the first pooling layer P1, and the third pooling layer The data resolution in P3 is reduced to 1/6 of the data resolution in the first pooling layer P1. For this reason, the information for specifying the position of the detection target in the input image data D1 becomes insufficient as it advances to the subsequent layer, and there is a concern that the position of the detection target in the input image cannot be specified with high accuracy.

Therefore, according to the arithmetic processing device 100, even when the resolution of data is reduced as the pooling process is repeated, a device is provided for accurately identifying the position of the detection target in the input image.
That is, as illustrated in FIG. 7, the arithmetic processing apparatus 100 causes the storage processing unit 105 to store the position information of the pixel data indicating the maximum value during the pooling process. Specifically, the storage processing unit 105 sets position information “1” to “4” for each pixel group G processed by one pooling process, in this case, 2 × 2 four pixels. And the memory | storage process part 105 memorize | stores the positional information on the pixel data which shows the maximum value among these four pixels in the case of a pooling process. In this case, it is assumed that the pixel of the position information “4” among the four pixels in the first layer shows the maximum value. For this reason, the storage processing unit 105 stores that the position information of the pixel having the maximum value in the first layer is “4”, and transmits it to the next second layer.

  In the second layer, the position information “1” to “4” is set for every four pixels processed by one pooling process in a state where the resolution is lower than that in the first layer. In this case, information “2 [4]” indicating that the position information of the pixel having the maximum value in the previous layer is “4” is displayed in the pixel of the position information “2” in the second layer. . It should be noted that the numerical value indicated in parentheses in the information “2 [4]” is pixel position information indicating the maximum value in the previous layer. In this way, as illustrated in FIG. 8, the arithmetic processing device 100 stores the position information of the pixel data indicating the maximum value during the pooling process, and the process by the convolution layer C and the process by the pooling layer P. Is repeatedly executed over multiple layers.

  Then, the arithmetic processing device 100 specifies the position of the detection target object in the input image data D1, in other words, the input image, by the restoration processing unit 106 based on the position information stored in the processing of each layer. That is, as illustrated in FIG. 9, for example, the arithmetic processing device 100 sets a rectangular frame W3 that surrounds an area where it is estimated that a detection target exists in the input image based on the position information J3 stored in the third layer. Set. The rectangular frame W3 is a frame having a minimum size including a position information group including a plurality of position information J3 stored in the third layer.

  However, the rectangular frame W3 based on the position information J3 in the third layer includes a detection target object, but also includes many areas where the detection target object does not exist. Therefore, the accuracy of specifying the position where the detection target exists in the input image is not good. Therefore, the arithmetic processing apparatus 100 corrects the rectangular frame W2 set by the restoration processing unit 106 by the correction processing unit 107. That is, the arithmetic processing device 100 corrects the rectangular frame W3 to a rectangular frame W2 smaller than the rectangular frame W3 based on the position information J2 stored in the second layer. The rectangular frame W2 is a frame having a minimum size including a position information group including a plurality of position information J2 stored in the second layer. According to the rectangular frame W2, compared with the rectangular frame W3, it is possible to reduce the area where the detection target does not exist. Therefore, it is possible to improve the accuracy of specifying the position where the detection target exists in the input image.

  However, even in the rectangular frame W2 based on the position information J2 of the second layer, a region where no detection target exists remains. Therefore, the arithmetic processing apparatus 100 further corrects the rectangular frame W2 by the correction processing unit 107. That is, the arithmetic processing device 100 corrects the rectangular frame W2 to a rectangular frame W1 smaller than the rectangular frame W2 based on the position information J1 stored in the first layer. The rectangular frame W1 is a frame having a minimum size including a position information group including a plurality of pieces of position information J1 stored in the first layer. According to the rectangular frame W1, it is possible to further reduce the area where the detection target does not exist as compared to the rectangular frame W2. Therefore, it is possible to further improve the accuracy of specifying the position where the detection target exists in the input image.

  By the way, according to the processing by the actual convolutional neural network, a plurality of input data is input as a feature map from the previous layer. Therefore, the arithmetic processing unit 100 is configured to specify, as the feature pixel data, the pixel data indicating the maximum value in the pooling process for each feature map by the storage processing unit 105. The storage processing unit 105 is configured to store position information of feature pixel data indicating the largest value among the plurality of feature pixel data.

  That is, FIG. 10 shows a case where two feature maps M1 and M2 are input as input data for the second layer. In the feature map M1, the pixel value of the feature pixel data Ds indicating the maximum value among the image groups included in the region R1 is “2.7”, and in the feature map M2, the maximum value among the pixel groups included in the region R1. The pixel value of the characteristic pixel data Ds indicating “0.3” is “0.3”. Therefore, the storage processing unit 105 stores “2.7”, which is the largest value among the pixel values of the feature pixel data Ds, and transmits it to the subsequent layer.

  In the feature map M1, the pixel value of the feature pixel data Ds indicating the maximum value among the image groups included in the region R2 is “0.4”. In the feature map M2, the pixel value included in the region R2 The pixel value of the feature pixel data Ds indicating the maximum value is “3.4”. Therefore, the storage processing unit 105 stores “3.4”, which is the largest value among the pixel values of the feature pixel data Ds, and transmits it to the subsequent layer.

  In this way, the arithmetic processing device 100 compares feature amounts among a plurality of feature maps, stores the position information of the feature map from which features are most prominently extracted, and transmits them to the subsequent layer. Yes. Note that the feature map M1 is an example of a feature map obtained by a filter learned to extract features of an object mainly extending in the horizontal direction in the input image. The feature map M2 is an example of a feature map obtained by a filter learned to extract features of an object that mainly extends in the vertical direction in the input image.

  Further, the feature map processed by the arithmetic processing device 100 is not limited to the one illustrated in FIG. In addition, it is desirable that the arithmetic processing device 100 be configured to normalize the feature map by a normalization processing unit (not shown) when comparing feature amounts between a plurality of feature maps. That is, the arithmetic processing unit 100 performs a known normalization process on the data after the pooling process, thereby converting the data into normalized data having a predetermined reference format, and then comparing the feature values. Good. According to this configuration, the feature amounts can be compared with higher accuracy.

  FIG. 11 shows an example of correcting a rectangular frame surrounding an area where a detection target is estimated to exist in an input image when position information obtained from a plurality of feature maps coexists. That is, the arithmetic processing device 100 sets the rectangular frame W3 based on the position information J3a from the feature map M1 and the position information J3b from the feature map M2 stored in the third layer, for example. The rectangular frame W3 is a frame having a minimum size including a position information group including a plurality of pieces of position information J3a and J3b stored in the third layer. Furthermore, the arithmetic processing unit 100 can correct the rectangular frame W3 to the rectangular frame W2 based on the position information J2a from the feature map M1 and the position information J2b from the feature map M2 stored in the second layer. . The rectangular frame W2 is a frame having a minimum size including a position information group including a plurality of pieces of position information J2a and J2b stored in the second layer. Furthermore, the arithmetic processing device 100 can correct the rectangular frame W2 to the rectangular frame W1 based on the position information J1a from the feature map M1 and the position information J1b from the feature map M2 stored in the first layer. . The rectangular frame W1 is a frame having a minimum size including a position information group including a plurality of pieces of position information J1a and J1b stored in the first layer.

According to the arithmetic processing device 100, even if the resolution of data decreases as the pooling process is repeated, the detection object in the input image is detected based on the position information stored by the storage processing unit 105 in the processing of each layer. The position can be specified with high accuracy.
Further, according to the arithmetic processing device 100, when a plurality of feature maps are input from the previous hierarchy, the pixel data indicating the maximum value is specified as the feature pixel data in the pooling process for each feature map, The position information of the feature pixel data indicating the largest value among the plurality of feature pixel data is stored. As a result, the position of the detection target can be specified based on the position information obtained from the feature map from which the feature is most prominently extracted, and the specification accuracy can be further improved.

Further, according to the arithmetic processing device 100, it is possible to further correct the rectangular frame W once identified, that is, the position of the detection target, based on the position information stored hierarchically over multiple hierarchies. Thereby, the specific accuracy of the position of a detection target object can be improved further.
Note that the present invention is not limited to the above-described embodiments, and can be applied to various embodiments without departing from the gist thereof. For example, the calculation block 104 may be configured to perform a sub-sampling process similar to the pooling process as the resolution reduction process. The sub-sampling process is a process for reducing output data by skipping a plurality of pixel data, that is, not reading. Therefore, this is a resolution reduction process in which the resolution of the output data decreases with the repetition of the process.

  In the drawing, 100 is an arithmetic processing unit, 104 is an arithmetic processing block (arithmetic processing means), 105 is a storage processing section (storage means), 106 is a restoration processing section (restoration means), and 107 is a correction processing section (correction means). Show.

Claims (2)

An arithmetic processing apparatus (100) for executing an arithmetic operation using a neural network in which a plurality of processing layers are hierarchically connected,
Arithmetic processing means (104) for executing at least convolution arithmetic processing and resolution reduction processing on input data input from the previous layer;
Storage means (105) for storing position information of pixel data indicating a maximum value among pixel data indicating characteristics during the resolution reduction processing;
A restoring means (106) for identifying the position of the detection object in the input data based on the position information stored in the storage means;
Equipped with a,
When a plurality of the input data is input from the previous layer, the storage unit specifies pixel data indicating a maximum value as the characteristic pixel data at the time of the resolution reduction processing for each of the input data, The position information of the feature pixel data indicating the largest value among the plurality of feature pixel data is stored,
An arithmetic processing apparatus , wherein the convolution calculation processing and the resolution reduction processing are repeatedly executed over a plurality of layers while storing position information of pixel data indicating a maximum value in the resolution reduction processing . The arithmetic processing apparatus according to claim 1 , further comprising a correction unit that corrects the position of the detection target specified by the restoration unit.

Images