JP7071215B2 - Recognition device - Google Patents

Description

The present disclosure relates to a recognition device and a recognition method.

Conventionally, as a technique for recognizing an object or a region from an image obtained by capturing an object, a method of first detecting an object candidate region from a parallax image and then recognizing an object in the image using a luminance image has been disclosed. (For example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-197378

However, the technique described in Patent Document 1 has a problem that it is difficult to recognize an object that is difficult to recognize with a parallax image, such as a distant object. Therefore, other techniques with high recognition accuracy have been desired.

The present invention can be realized as the following forms.

According to one embodiment of the present invention, there is provided a recognition device (100) that recognizes a region and an object by using a trained neural network. The recognition device (100) includes a captured image feature map extraction unit (104) for extracting a feature map of a captured image in which the region and the object are included in the image, and the region and the object in the image. The distance image feature map extraction unit (105) that extracts the feature map of the distance image, and the feature map connection unit (106) that connects the feature map extracted from the captured image and the feature map extracted from the distance image. ), The linked feature map, the feature map extracted from the captured image, and the feature map extracted from the distance image, the region segmentation unit that generates the feature map used for the segmentation of the region. (107), a region output unit (109) that performs semantic segmentation that associates the image with the region using the feature map used for segmentation of the region, the linked feature map, and extraction from the captured image. Using the created feature map and the feature map extracted from the distance image, the object segmentation unit (108) that generates the feature map used for the segmentation of the object, and the feature map used for the segmentation of the object are used. An object output unit (110) that performs semantic segmentation that associates the image with the object, the captured image feature map extraction unit, the distance image feature map extraction unit, the feature map connection unit, and the above. The area segmentation section and the object segmentation section are configured by the neural network.

According to this form of the recognition device, semantic segmentation is performed from the connected feature map, the feature map extracted from the captured image, and the feature map extracted from the distance image, so that the recognition accuracy is improved.

It is a functional block diagram of a vehicle equipped with a recognition device. It is a block diagram which shows the structure of a recognition device. It is a figure which shows the flowchart of the recognition process. It is a figure explaining the captured image feature map extraction unit 104. It is a figure explaining the distance image feature map extraction unit 105. It is a figure explaining the area segmentation part 107. It is a figure explaining the object segmentation part 108. It is a figure which shows the example of the luminance image acquired by the image acquisition part, and the area image and the object image after semantic segmentation. It is a figure which shows the comparative example using FuseNet. It is a figure which shows the comparative example using U-Net. It is a block diagram which shows the structure of the recognition device of a modification.

A. First Embodiment As shown in FIG. 1, in this embodiment, the recognition device 100 is mounted on the vehicle 10. The recognition device 100 may be mounted on an object other than a vehicle such as a ship or a drone.

The vehicle 10 further includes a captured image acquisition unit 21 and a distance image acquisition unit 22. In the present embodiment, the captured image acquisition unit 21 and the distance image acquisition unit 22 are mounted so that the front of the vehicle 10 is the imaging range, and a monocular camera is used as the captured image acquisition unit 21 to obtain the distance image acquisition unit 22. A stereo camera is used as. In the present embodiment, the recognition device 100 acquires a captured image from the captured image acquisition unit 21 and acquires a distance image from the distance image acquisition unit 22.

The recognition device 100 is configured as a well-known computer including a CPU 11 and a memory 12 such as a ROM or RAM. It is desirable that the recognition device 100 uses a chip dedicated to the convolution operation in the neural network. The recognition device 100 uses the CPU 11 and the memory 12 to execute a program stored in the memory 12 to perform the recognition process described later. Specifically, the recognition device 100 recognizes an object and a region in the image from the captured image and the distance image acquired by the captured image acquisition unit 21 and the distance image acquisition unit 22 by using the trained neural network. The recognition result obtained by the recognition process is input to the control unit 30 of the vehicle 10 by the recognition device 100. The control unit 30 controls the operation of the vehicle 10 by using the input recognition result. In this embodiment, a convolutional neural network (CNN) is used as the neural network, but other types of neural networks may be used. Here, examples of the object of the present embodiment include a vehicle existing in front of the vehicle 10 and an obstacle such as a stone. Further, as an area of the present embodiment, for example, a travelable area of the vehicle 10 can be mentioned. In the present embodiment, the neural network is pre-learned using an arbitrary number (for example, 9000) of data tagged about the region and the object in the captured image and the distance image.

As shown in FIG. 2, the recognition device 100 includes a captured image input unit 101, a distance image input unit 102, a captured image feature map extraction unit 104, a distance image feature map extraction unit 105, and a feature map connecting unit 106. The region segmentation unit 107, the object segmentation unit 108, the region output unit 109, and the object output unit 110 are provided. Of these, the captured image feature map extraction unit 104, the distance image feature map extraction unit 105, the feature map connection unit 106, the region segmentation unit 107, and the object segmentation unit 108 are configured as the neural network 103. Each of the above parts is actually executed by the CPU 11 executing a program (mainly a matrix operation or a convolution operation) stored in advance. The processing contents of each part are shown in FIG. Since each process is executed by sending data along the arrow, it is described as assuming that there is a block that executes each process centering on the data flow, not as a sequential process as in a flowchart.

As shown in FIG. 3, when the recognition process is started, the captured image input unit 101 of the recognition device 100 inputs the captured image acquired from the captured image acquisition unit 21 to the neural network 103. The acquired image may be resized, corrected for distortion, or the like before being input to the neural network 103.

In the present embodiment, a luminance image, which is an image converted into luminance (0 to 255) for each pixel, is used as the captured image, but a color image may be used instead. By using the luminance image, the influence of the difference in illuminance can be reduced, and the amount of information is smaller than that of the color image, so that the processing can be performed quickly. Further, it is preferable to use a parallax image as a distance image and a luminance image as a captured image because it is easy to detect a distant object.

Further, the distance image input unit 102 of the recognition device 100 inputs the distance image acquired from the distance image acquisition unit 22 to the neural network 103. The acquired distance image may be subjected to processing such as size change and distortion correction before being input to the neural network 103.

In the present embodiment, the parallax image is used as the distance image, but the present invention is not limited to this, and for example, a depth image acquired from a depth camera or a distance image acquired from a lidar or a millimeter wave radar may be used. The parallax image is an image in which a value (0 to 255) corresponding to the parallax is given for each pixel, and in the present embodiment, the larger the parallax, the brighter the image is expressed. By using the parallax image, the area of the object and the boundary of the area become clearer as a feature than the luminance image, so that the accuracy of the boundary in the output result can be improved. For example, when a black vehicle traveling on a black road surface such as asphalt is traveling, it is difficult to distinguish the boundary between the road surface and the vehicle in a luminance image or a color image. The boundary of is clear. Further, in the luminance image and the color image, the appearance of the vehicle also affects the output result, and this influence can be mitigated by using the distance image. Further, by using a distance image, the influence of brightness at the time of imaging can be mitigated. As described above, in the present embodiment, the characteristics of each other can be complemented by using the captured image and the distance image as the input image.

Next, the captured image feature map extraction unit 104 of the recognition device 100 extracts the feature map of the captured image. That is, the captured image feature map extraction unit 104 extracts a feature map useful for segmentation of an object and a region captured in the captured image from the captured image according to a command from the learned model stored in the memory 12.

As shown in FIG. 4, in the present embodiment, the captured image feature map extraction unit 104 has a plurality of imaging blocks SB1 to SB5 including a convolutional layer and a pooling layer (hereinafter, also simply referred to as “imaging block SB”). To prepare for. In the present embodiment, the image pickup block SB is provided with two convolutional layers and one pooling layer so as to process data in this order. Here, in the drawings after FIG. 3, the convoluted layer is described as "Conv.", And the pooling layer is described as "Pooling". The number of the image pickup block SB is one more than the number of traveling blocks described later, and in the present embodiment, the image pickup image feature map extraction unit 104 includes five image pickup block SBs. As shown in FIG. 3, the feature map extracted by the image pickup block SB is output to the region segmentation unit 107 and the object segmentation unit 108.

The captured image feature map extraction unit 104 further includes an imaging upsampling block SBU (hereinafter, also referred to as “imaging US block SBU”) on the downstream side of the imaging block SB. The imaging US block SBU includes a convolutional layer and an upsampling layer. In the figures after FIG. 3, the upsampling layer is referred to as "US". In the present embodiment, the imaging US block SBU is provided with two convolutional layers and one upsampling layer so as to process data in this order. The feature map extracted by the imaging US block SBU is output to the feature map connecting unit 106.

The distance image feature map extraction unit 105 of the recognition device 100 extracts the feature map of the distance image. That is, the distance image feature map extraction unit 105 extracts a feature map useful for segmentation of an object and a region captured in the distance image from the distance image according to a command from the trained model stored in the memory 12.

As shown in FIG. 5, in the present embodiment, the distance image feature map extraction unit 105 includes a plurality of distance blocks KB1 to KB5 (hereinafter, also simply referred to as “distance block KB”) including a folding layer and a pooling layer. To prepare for. In the present embodiment, the distance block KB is provided with two convolutional layers and one pooling layer so as to process data in this order. In this embodiment, the number of distance blocks KB is the same as the number of image pickup blocks. In the present embodiment, the distance image feature map extraction unit 105 includes five distance blocks KB. The feature map extracted by the distance block KB is output to the area segmentation unit 107 and the object segmentation unit 108.

The distance image feature map extraction unit 105 further includes a distance upsampling block KBU (hereinafter, also referred to as “distance US block KBU”) on the downstream side of the distance block. The distance US block KBU comprises a convolutional layer and an upsampling layer. In the present embodiment, the distance US block KBU is provided with two convolutional layers and one upsampling layer to process data in this order. The feature map extracted by the distance US block KBU is output to the feature map connecting unit 106.

As shown in FIGS. 3 to 5, the feature map connecting portion 106 of the recognition device 100 connects the feature map extracted from the captured image and the feature map extracted from the distance image. Specifically, the feature map connecting portion 106 was extracted by the feature map extracted by the most downstream image pickup block SB5, the feature map extracted by the most downstream distance block KB5, and the image pickup US block SBU. The feature map and the feature map extracted by the distance US block KBU are connected. More specifically, the feature map connecting portion 106 includes (i) a feature map extracted by the upsampling layer of the imaging US block SBU, and (ii) a feature map extracted by the upsampling layer of the distance US block KBU. , (Iii) Feature map extracted by the convolutional layer in front of the pooling layer in the 5th imaging block SB5, and (iv) Features extracted by the convolutional layer in front of the pooling layer in the 5th distance block KB5. Connect with the map.

After that, the recognition device 100 uses the connected feature map, the feature map extracted from the captured image, and the feature map extracted from the distance image to generate a feature map to be used for segmentation of the region, and also to generate an object. The feature map used for the segmentation of is generated. Here, the segmentation of an area means that a specific area in an image is specified in pixel units, and the segmentation of an object means that a specific object in an image is specified in pixel units.

In the present embodiment, as shown in FIG. 6, the region segmentation unit 107 includes a plurality of region blocks RB1 to RB4 (hereinafter, also simply referred to as “region block RB”). In the present embodiment, the region segmentation unit 107 includes four region blocks. The region block RB includes a reverse convolution layer, an upsampling layer, and a coupling layer. In addition, in the figure after FIG. 6, the reverse convolution layer is described as "Deconv.", And the binding layer is described as "Concat". In the present embodiment, the region block RB is provided with two reverse convolution layers, an upsampling layer, and a coupling layer so as to process data in this order. The connection layer of the region block is a combination of the feature map extracted by the upsampling layer and the feature map extracted by the image pickup block of the captured image feature map extraction unit 104 and the distance block of the distance image feature map extraction unit 105, respectively. conduct.

The region segmentation unit 107 further includes a region dropout block RBD (hereinafter, also referred to as “region DO block RBD”) on the downstream side of the region block. The region DO block RBD comprises a reverse convolution layer and a dropout layer. In the present embodiment, the region DO block RBD is provided with two layers alternately of a reverse convolution layer and a dropout layer, so as to process data in this order. The feature map extracted by the area DO block RBD is output to the area output unit 109. The present embodiment can avoid overfitting by providing a dropout layer.

The object segmentation unit 108 of the recognition device 100 generates a feature map used for object segmentation by using the connected feature map, the feature map extracted from the captured image, and the feature map extracted from the distance image. ..

In the present embodiment, as shown in FIG. 7, the object segmentation unit 108 includes a plurality of object blocks BB1 to BB4 (hereinafter, also simply referred to as “object block BB”). In the present embodiment, the object segmentation unit 108 includes four object blocks BB. The object block BB includes a reverse convolution layer, an upsampling layer, and a coupling layer. In the present embodiment, the object block BB is provided with two reverse convolution layers, an upsampling layer, and a coupling layer so as to process data in this order. The coupling layer of the object block BB is a combination of the feature map extracted by the upsampling layer and the feature map extracted by the imaging block of the captured image feature map extraction unit 104 and the distance block of the distance image feature map extraction unit 105, respectively. I do.

The object segmentation unit 108 further includes an object dropout block BBD (hereinafter, also referred to as “object DO block BBD”) on the downstream side of the object block. The object DO block BBD includes a reverse convolution layer and a dropout layer. In the present embodiment, the object DO block BBD is provided with two layers alternately of a reverse convolution layer and a dropout layer, and the data is processed in this order. The feature map extracted by the object DO block BBD is output to the object output unit 110. The present embodiment can avoid overfitting by providing a dropout layer.

Then, the area output unit 109 of the recognition device 100 performs semantic segmentation that associates the image with the area from the feature map extracted by the area segmentation unit 107. In the present embodiment, the region output unit 109 performs semantic segmentation by performing conversion by using a sigmoid activation function and a binary cross entropy error function.

The object output unit 110 of the recognition device 100 performs semantic segmentation regarding an object that associates an image with an object from a feature map extracted by the object segmentation unit 108. In the present embodiment, the object output unit 110 performs semantic segmentation by performing conversion by using a sigmoid activation function and a binary cross entropy error function.

As described above, the image data obtained by the captured image acquisition unit 21 and the distance image acquisition unit 22 is processed by the recognition device 100, so that a set of images captured by the captured image acquisition unit 21 and the distance image acquisition unit 22. The recognition process for the image ends. The recognition result obtained by the recognition process is input to the control unit 30 of the vehicle 10 by the recognition device 100. The above-mentioned processing is repeated as long as the imaging by the captured image acquisition unit 21 and the distance image acquisition unit 22 continues.

FIG. 8 shows an example of a luminance image acquired by the captured image acquisition unit 21 and a region image and an object image after semantic segmentation. In FIG. 8, the vehicle in front is recognized as an object, and the area in which the vehicle can travel is recognized as an area. As can be seen from FIG. 8, it can be seen that the boundary between the object and the boundary of the area are clearly separated.

In the present embodiment, the feature map extracted by the image pickup block SB is output to the image pickup block SB on the downstream side, and is also output to the area block RB and the object block BB. Here, assuming that the number of region blocks RB is K and n is an arbitrary integer (n = 1 to K), the nth image pickup block SB counting from the upstream is a feature map to the n + 1th image pickup block SB. Is output, and the feature map is output to the Kn + 1st area block RB and the Kn + 1st object block BB counting from the upstream. Here, the feature map extracted from the pooling layer of the nth imaging block SB is output to the n + 1th imaging block SB, but the Kn + 1st region block RB and the Kn + 1st object. The feature map extracted by the convolutional layer in front of the pooling layer of the nth imaging block SB is output to the block BB. In the present embodiment, the number of region blocks is 4, so for example, the first (n = 1) image pickup block SB1 counting from the upstream is output to the second image pickup block SB2 and is output from the upstream. It is calculated and output to the fourth area block RB4 and the fourth object block BB4.

Further, in the present embodiment, the feature map extracted by the distance block KB is output to the distance block KB on the downstream side, and is also output to the area block RB and the object block BB. Assuming that K is the number of area blocks and n is an arbitrary integer (n = 1 to K), the nth distance block KB counting from the upstream is output to the n + 1th distance block KB and is output from the upstream. It is calculated and output to the Kn + 1st area block RB and the Kn + 1st object block BB. Here, the feature map extracted from the pooling layer of the nth distance block KB is output to the n + 1th distance block KB, but the Kn + 1st region block RB and the Kn + 1st object. The feature map extracted by the convolutional layer before the pooling layer of the nth distance block KB is output to the block BB. In the present embodiment, the number of distance blocks is 4, so for example, the first distance block KB1 (n = 1) counting from the upstream is output to the second distance block KB2 and is output from the upstream. It is calculated and output to the fourth area block RB4 and the fourth object block BB4.

That is, the neural network of the present embodiment is further extracted from the feature map extracted from the captured image feature map extraction unit 104 and the distance image feature map extraction unit 105 in addition to the feature map connected by the feature map connection unit 106. The created feature map is directly output to the area segmentation section 107 and the object segmentation section 108 without passing through the feature map connecting section 106. For this reason, in a general neural network, an error is less likely to be transmitted as the number of layers increases, so that learning efficiency decreases and the boundaries of objects and regions are blurred. However, according to the present embodiment, the region that is the output layer of the network. Since the segmentation unit 107 and the object segmentation unit 108 collate the information from the captured image feature map extraction unit 104 and the distance image feature map extraction unit 105, which are input layers with a large amount of boundary information, the boundaries of the object are not blurred. The accuracy can be improved.

Here, the structure of the neural network of this embodiment is different from other known structures. Comparative Example 100Y using the FaceNet shown in FIG. 9 includes (i) a segmentation unit 107Y instead of the region segmentation unit 107 and the object segmentation unit 108, and (ii) the region output unit 109 and The difference is that the output unit 109Y is provided instead of the object output unit 110. Due to this difference, the structure of the neural network of the present embodiment has a clearer boundary between an object and a region as compared with Comparative Example 100Y.

Further, in Comparative Example 100Z using U-Net shown in FIG. 10, the distance image input unit 102, the distance image feature map extraction unit 105, the object segmentation unit 108, and the object output unit 110 are compared with the present embodiment. The difference is that the segmentation unit 107Z is provided in place of the area segmentation unit 107, and the output unit 109Z is provided in place of the area output unit 109. Due to this difference, the structure of the neural network of the present embodiment is specialized in extracting the feature map of the captured image in the captured image feature map extraction unit 104, and the feature map of the distance image in the distance image feature map extraction unit 105. It differs from Comparative Example 100Z in that it specializes in the extraction of. As a result, in the structure of the neural network of the present embodiment, the boundary between the boundary of the object and the boundary of the region becomes clearer as compared with Comparative Example 100Z.

B. Modification example The recognition device 100A of the modification shown in FIG. 11 is compared with the above-mentioned recognition device 100, and further includes a color image input unit 102A for inputting a color image into a neural network and a color for extracting a feature map of the color image. It differs in that it includes an image feature map extraction unit 105A, a pedestrian segmentation unit 108A that generates a feature map used for pedestrian segmentation, and a pedestrian output unit 110A that performs semantic segmentation that associates an image with a pedestrian. Further, in the recognition device 100A of the modified example, as compared with the above-mentioned recognition device 100, the feature map extracted by (i) the color image feature map extraction unit 105A has the feature map connecting unit 106, the area segmentation unit 107, and the feature map. In addition to being output to the object segmentation unit 108, (ii) the feature map extracted by the captured image feature map extraction unit 104, the distance image feature map extraction unit 105, and the feature map connection unit 106 is output to the pedestrian segmentation unit 108A. The point is different.

As in this modification, the images input in the present disclosure may be three or more types instead of two types, and the output images may be three or more types instead of two types.

The present disclosure is not limited to the above-described embodiments and modifications, and can be realized with various configurations within a range not deviating from the gist thereof. For example, the technical features in the present embodiment and modifications corresponding to the technical features in each of the embodiments described in the column of the outline of the invention may be used to solve some or all of the above-mentioned problems, or the above-mentioned ones. It is possible to replace or combine them as appropriate in order to achieve some or all of the effects of. Further, if the technical feature is not described as essential in the present specification, it can be appropriately deleted.

10 vehicle, 11 CPU, 12 memory, 21 captured image acquisition unit, 22 distance image acquisition unit, 30 control unit, 100 recognition device, 100A recognition device, 100Y, 100Z comparative example, 101 image capture image input unit, 102 distance image input unit , 102A color image input section, 104 captured image feature map extraction section, 105 distance image feature map extraction section, 105A color image feature map extraction section, 106 feature map connection section, 107 area segmentation section, 107Y segmentation section, 108 object segmentation section. , 108A pedestrian segmentation section, 109 area output section, 109Y output section, 110A pedestrian output section, 110 object output section, BB object block, BBD object DO block, KB distance block, KBU distance US block, RB area block, RBD Area DO block, SB imaging block, SBU imaging US block,

Claims (4)

A recognition device (100) that recognizes an area and an object using a trained neural network.
An image capture image feature map extraction unit (104) that extracts a feature map of an image captured image in which the region and the object are included in the image, and
A distance image feature map extraction unit (105) that extracts a feature map of a distance image in which the region and the object are included in the image, and
A feature map connecting unit (106) that connects the feature map extracted from the captured image and the feature map extracted from the distance image, and
A region segmentation unit (107) that generates a feature map to be used for segmentation of the region by using the linked feature map, the feature map extracted from the captured image, and the feature map extracted from the distance image. When,
A region output unit (109) that performs semantic segmentation that associates the image with the region using a feature map used for segmentation of the region.
An object segmentation unit (108) that generates a feature map to be used for segmentation of the object by using the connected feature map, the feature map extracted from the captured image, and the feature map extracted from the distance image. When,
An object output unit (110) that performs semantic segmentation that associates the image with the object by using the feature map used for the segmentation of the object is provided.
A recognition device in which the captured image feature map extraction unit, the distance image feature map extraction unit, the feature map connection unit, the region segmentation unit, and the object segmentation unit are configured by the neural network. The recognition device according to claim 1.
The neural network is a recognition device which is a convolutional neural network. The recognition device according to claim 1 or 2.
The captured image is a recognition device which is a luminance image. The recognition device according to any one of claims 1 to 3.
The distance image is a parallax image, a recognition device.

Images