JP6830052B2 - Spatial information generator and spatial information generation method - Google Patents

Description

The present invention relates to a spatial information generation device and a spatial information generation method for generating three-dimensional spatial information using a machine learning model.

A device that generates a three-dimensional object model showing the surface position of a subject in a three-dimensional space based on a plurality of images is known. Patent Document 1 discloses a technique for generating a three-dimensional object model by extracting a subject from images captured by a plurality of cameras.

JP-A-2010-122725

In order to realize a free-viewpoint video system that reproduces an image viewed from an arbitrary position based on moving images taken by a plurality of cameras, information indicating the position of the subject represented in each image, that is, three-dimensional It is necessary to generate model information as accurately as possible. By using the conventional technique, it is possible to estimate the surface position of the subject in the image based on a plurality of images. However, there is a possibility that different subjects among a plurality of images are extracted as the same subject. In this case, accurate 3D model information cannot be generated.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide a spatial information generation device and a spatial information generation method capable of improving the accuracy of estimating the surface position of a subject in a three-dimensional space. And.

The spatial information generator according to the first aspect of the present invention is generated by the first image captured by the first image pickup device and the second image pickup device installed at a position different from the first image pickup device. Machine learning that can output the type of subject included in the second captured image, the image acquisition unit that acquires the second captured image, the first captured image, and the second captured image based on the input image. Common to both the propagation control unit that propagates the plurality of processing layers included in the model, the post-stage processing layer selected from the plurality of processing layers, and the pre-stage processing layer that is the processing layer immediately before the post-stage processing layer. One or more first captured image outputs output from the post-processed layer and the pre-processed layer based on the first captured image, and the post-processed layer and the post-processed layer based on the second captured image. An extraction unit that extracts one or more second captured image outputs output from the pre-stage processing layer, and one or more first captured image feature points are detected based on the one or more first captured image outputs, and the above A feature point detection unit that detects one or more second captured image feature points based on one or more second captured image outputs, the one or more first captured image feature points, and the one or more second captured image feature points. Based on the above, it has a spatial information generation unit that generates spatial information indicating the surface position of the subject in the three-dimensional space.

The extraction unit outputs a plurality of post-stage first captured images output from the post-stage processing layer because the first captured image propagates in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. , And one or more post-stage first images that are commonly activated from the plurality of post-stage second captured image outputs output from the post-stage processing layer because the second captured image propagated in the order of the pre-stage processing layer and the post-stage processing layer. It is a factor that activates the post-stage extraction unit that extracts one captured image output and one or more post-stage second captured image outputs, the one or more post-stage first captured image output, and the one or more post-stage second captured image output. Of the plurality of pre-stage first captured image outputs output from the pre-stage processing layer and the plurality of pre-stage second captured image outputs output from the pre-stage processing layer, one or more pre-stage first images that are commonly activated. It may have a pre-stage extraction unit that extracts a captured image output and one or more pre-stage second captured image outputs.

The pre-stage extraction unit has one or more pre-stage first captured image outputs and the above-mentioned one or more pre-stage first captured image outputs based on the activated size of the plurality of pre-stage first captured image outputs and the plurality of pre-stage second captured image outputs. One or more pre-stage second captured image outputs may be extracted.

The machine learning model includes a convolutional neural network, and the post-processing layer may be any one of an output layer, a fully connected layer, a regularized layer, a pooling layer, and a convolutional layer.
The pretreatment layer may be any one of a fully bonded layer, a regularized layer, a pooling layer, a convolution layer, and an input layer.

When the last layer, which is the last treatment layer, is selected as the subsequent treatment layer among the plurality of treatment layers, the extraction unit is one or more of the first ones that are commonly activated in the last treatment layer. When there is no captured image output and one or more second captured image outputs, the one or more first captured image outputs and the one or more second captured images that are commonly activated in the processing layer before the rearmost layer. The image output may be extracted.

The spatial information generator is a part of the first image based on the mutual correspondence between the one or more first captured image feature points and the one or more second captured image feature points specified by the feature point detection unit. It further has a selection unit for selecting a captured image feature point and a part of the second captured image feature point, and the spatial information generation unit includes the first captured image feature point of the part and the second captured image of the part. The spatial information may be generated based on the feature points.

The spatial information generation unit has a relationship between the positions of the first captured image feature points in the first captured image and the positions of the second captured image feature points in the second captured image, and The distance to the subject may be estimated as the spatial information based on the difference between the first image pickup device and the second image pickup device.

The spatial information generation device further includes an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers, and the extraction unit receives the instruction receiving unit. The treated layer indicated by the instruction may be used as the post-processed layer.

The extraction unit selects one of the plurality of processing layers as the post-processing layer, extracts the one or more first captured image outputs and the one or more second captured image outputs, and then extracts the one or more second captured image outputs. The processing layer selected as the processing layer may be selected as the post-stage processing layer to extract another one or more first captured image outputs and one or more second captured image outputs.

The spatial information generation method according to the second aspect of the present invention is generated by the first image captured by the first image pickup device and the second image pickup device installed at a position different from the first image pickup device. A machine learning model capable of outputting the type of the subject included in the second captured image, the step of acquiring the second captured image, and the type of the subject included in the first captured image and the second captured image based on the input image. It is commonly activated in both the step of propagating the plurality of included treatment layers, the post-stage treatment layer selected from the plurality of treatment layers, and the pre-stage treatment layer which is the treatment layer immediately before the post-stage treatment layer. One or more first captured image outputs output from the post-stage processing layer and the pre-stage processing layer based on the first captured image, and from the post-stage processing layer and the pre-stage processing layer based on the second captured image. One or more first captured image feature points are detected based on the step of extracting one or more output second captured image outputs and the one or more first captured image outputs, and the one or more second captured image features. A three-dimensional space based on the step of detecting one or more second captured image feature points based on the captured image output, the one or more first captured image feature points, and the one or more second captured image feature points. It has a step of generating spatial information indicating the surface position of the subject in the above.

In the extraction step, a plurality of post-stage first captured images output from the post-stage processing layer due to the propagation of the first captured image in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. One or more post-stages that are commonly activated from the output and the plurality of post-stage second captured image outputs output from the post-stage processing layer due to the propagation of the second-stage image in the order of the pre-stage processing layer and the post-stage processing layer. Factors that activate the first-stage extraction step for extracting the first captured image output and one or more subsequent-stage second captured image outputs, the one or more subsequent-stage first captured image output, and the one or more subsequent-stage second captured image output. Of the plurality of pre-stage first captured image outputs output from the pre-stage processing layer and the plurality of pre-stage second captured image outputs output from the pre-stage processing layer, one or more pre-stage first images that are commonly activated. It may have one captured image output and one or more pre-stage second captured image outputs.

In the spatial information generation method, after executing the pre-stage extraction step, one or more pre-stage first captured image outputs and one or more pre-stage second captured image outputs are combined with the plurality of rear-stage first captured image outputs and the said. The latter-stage extraction step may be executed as a plurality of second-stage second captured image outputs.

In the spatial information generation method, the latter-stage extraction step and the first-stage extraction step may be executed for each of the plurality of processing layers.

According to the present invention, there is an effect that the accuracy of estimating the surface position of the subject in the three-dimensional space can be improved.

It is a figure for demonstrating the outline of the process of generating spatial information. It is a figure which shows an example of the structure of a machine learning model. It is a figure which shows the structure of the spatial information generation apparatus. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a flowchart which shows the flow of processing performed by a spatial information generator. It is a flowchart which shows the flow of the process performed by the extraction part.

[Overview of spatial information generator 1]
FIG. 1 is a diagram for explaining an outline of a process for generating spatial information. The spatial information generation device 1 is, for example, a PC (Personal Computer). The spatial information generation device 1 uses a machine learning model M based on a plurality of images generated by a plurality of imaging devices installed at different positions, and uses a machine learning model M to position the surface of the subject in the image in a three-dimensional space. It is a device that generates spatial information indicating. In the present specification, it is assumed that the imaging parameters of the imaging device (for example, the imaging position and the orientation of the imaging device) are known. The image is a still image or a moving image. When the image is a moving image, the spatial information generation device 1 generates spatial information for each frame included in the moving image.

The spatial information generation device 1 acquires the first captured image A generated by the first imaging device ((1) in FIG. 1). The first imaging device is, for example, a camera installed on the right side of a plurality of imaging devices provided in front of the vehicle. The first captured image A illustrated in FIG. 1 shows a truck T1 traveling ahead and a pedestrian T2 crossing a road.

Subsequently, the spatial information generation device 1 acquires the second captured image B generated by the second imaging device installed at a position different from that of the first imaging device ((2) in FIG. 1). The second image pickup device is, for example, a camera installed on the left side of a plurality of image pickup devices provided in front of the vehicle. Similar to the first captured image A, the second captured image B shown in FIG. 1 shows the truck T1 and the pedestrian T2.

The spatial information generation device 1 inputs each of the acquired first captured image A and second captured image B into the machine learning model M, and propagates a plurality of processing layers included in the machine learning model M ((1) in FIG. 3)). The machine learning model M is a model learned to output the type of the subject included in the image based on the input image.

FIG. 2 is a diagram showing an example of the configuration of the machine learning model M. The machine learning model M includes a convolutional neural network (CNN). In this case, the machine learning model M is an input layer M1, a first convolution layer M2, a second convolution layer M3, a first pooling layer M4, a normalization layer M5, a third convolution layer M6, and a second. It has a pooling layer M7, a first fully connected layer M8, a second fully connected layer M9, and an output layer M10. In the present specification, of the two adjacent processing layers, the processing layer on the upstream side when the first captured image A and the second captured image B propagate is referred to as a pre-stage processing layer, and the processing layer on the downstream side is referred to as a rear-stage processing layer. It is called a processing layer.

The treatment layers that can be the subsequent treatment layers are the first convolution layer M2, the second convolution layer M3, the first pooling layer M4, the normalization layer M5, the third convolution layer M6, the second pooling layer M7, and the second. It is one of the fully bonded layer M8 of 1, the second fully bonded layer M9, and the output layer M10. The processing layers that can be the pre-stage processing layers are the input layer M1, the first convolution layer M2, the second convolution layer M3, the first pooling layer M4, the normalization layer M5, the third convolution layer M6, and the second. It is any one of the pooling layer M7, the first fully bonded layer M8, and the second fully bonded layer M9. The spatial information generation device 1 inputs the acquired image to the machine learning model M, and forward-propagates each processing layer from the input layer M1 to the output layer M10, that is, infers the subject in the image. Output the type.

Returning to FIG. 1, the spatial information generation device 1 uses the calculation results in each processing layer that led to the machine learning model M outputting the type of subject, that is, the features with a high degree of abstraction by deep learning. A feature point common to the captured image A and the second captured image B is detected ((4) in FIG. 1). Here, the spatial information generation device 1 detects the common feature points in the reverse order of the propagation order. By doing so, the spatial information generation device 1 can detect feature points based on features with a high degree of abstraction.

By detecting common feature points, the spatial information generation device 1 includes the truck T1 and the pedestrian T2 shown in the first captured image A and the truck T1 and the pedestrian T2 shown in the second captured image B. Detects that there is a correspondence between them. The correspondence relationship is a relationship in which the pixels included in the first captured image indicated by the feature points of the first captured image and the pixels included in the second captured image indicated by the feature points of the second captured image B match or approximate. ..

Then, the spatial information generation device 1 is a space indicating the surface positions of the track T1 and the pedestrian T2 in the three-dimensional space based on the feature points of the first captured image A and the second captured image B which are in the detected corresponding relationship. Information is generated ((5) in FIG. 1). By doing so, the spatial information generation device 1 can improve the accuracy of estimating the surface position of the subject in the three-dimensional space.
Hereinafter, the details of the spatial information generation device 1 will be described.

[Configuration of spatial information generator 1]
FIG. 3 is a diagram showing the configuration of the spatial information generation device 1. The spatial information generation device 1 has an operation unit 11, a storage unit 12, and a control unit 13.

The operation unit 11 is an input device that accepts user operations.
The storage unit 12 is, for example, a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 12 stores various programs executed by the control unit 13. The storage unit 12 stores the first captured image and the second captured image.

The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 controls the function related to the spatial information generation device 1 by executing the program stored in the storage unit 12. By executing the program, the control unit 13 functions as an image acquisition unit 131, a propagation control unit 132, an extraction unit 133, an instruction reception unit 136, a feature point detection unit 137, a selection unit 138, and a spatial information generation unit 139. ..

The image acquisition unit 131 acquires the first captured image and the second captured image stored in the storage unit 12. The image acquisition unit 131 inputs the acquired first captured image and the second captured image to the propagation control unit 132.

The propagation control unit 132 propagates a plurality of processing layers included in the machine learning model M to each of the first captured image and the second captured image. In the example shown in FIG. 2, the propagation control unit 132 propagates each of the processing layers from the input layer M1 to the output layer M10 included in the machine learning model M to each of the first captured image and the second captured image in order. ..

The extraction unit 133 is based on the first captured image that is commonly activated in both the processing layer of the post-stage processing layer selected from the plurality of processing layers and the processing layer of the pre-stage processing layer that is the processing layer immediately before the post-stage processing layer. Extract one or more first captured image outputs output from the post-stage processing layer and the pre-stage processing layer and one or more second captured image outputs output from the post-stage processing layer and the pre-stage processing layer based on the second captured image. .. The details of the extraction process performed by the extraction unit 133 will be described later, but the extraction unit 133 is a part of the first captured image output that is commonly activated in the subsequent processing layer, and is a part of the first captured image output and the second imaging. The post-stage extraction unit 134 that extracts the post-stage second captured image output that is a part of the image output, and the pre-stage first captured image output and the first that are a part of the first-stage captured image output that are commonly activated in the pre-stage processing layer. 2. It has a pre-stage extraction unit 135 that extracts a pre-stage second captured image output that is a part of the captured image output.

The first captured image output and the second captured image output extracted by the extraction unit 133 are information indicating the activated unit among the plurality of units included in the processing layer. A unit is one or more pixels contained in an image. The definition of activation may be, for example, when the product of the output value of the unit or the output value of the unit and the weight of the connection of the unit exceeds a predetermined threshold value, or a predetermined number or a predetermined number in descending order of output. It may be included in the ratio. Further, in the processing layer other than the fully connected layer, for example, it may be included in a predetermined number or a predetermined ratio in descending order of output for each channel. The channel is the output calculated by convolution for each filter.

It is preferable that the extraction unit 133 selects the last treatment layer, which is the last treatment layer, as the subsequent treatment layer among the plurality of treatment layers. However, there are cases where there is no first captured image output and second captured image output that are commonly activated in the rearmost layer. Therefore, when the rearmost layer is selected as the subsequent processing layer among the plurality of processing layers, the extraction unit 133 outputs one or more first captured images and one or more first images that are commonly activated in the rearmost layer. 2 When there is no captured image output, one or more first captured image outputs and one or more second captured image outputs that are commonly activated in the processing layer before the rearmost layer may be extracted.

For example, when the extraction unit 133 selects the output layer M10, which is the rearmost layer, as the post-processing layer, one or more first captured image outputs and one or more second imaging that are commonly activated in the output layer M10. Suppose there is no image output. In this case, the extraction unit 133 repeatedly searches for one or more first captured image outputs and one or more second captured image outputs that are commonly activated for each processing layer before the output layer M10.

The extraction unit 133 has, for example, one or more first captured image outputs and one or more second captured image outputs that are commonly activated in the second fully connected layer M9, which is the processing layer immediately before the output layer M10. If so, the second fully bonded layer M9 is selected as the post-treatment layer. Then, the extraction unit 133 extracts one or more first captured image outputs and one or more second captured image outputs that are commonly activated in the second fully connected layer M9 selected as the post-processing layer. By doing so, the extraction unit 133 can associate the subject to be captured in each of the first captured image and the second captured image even when there are few matching regions.

The extraction unit 133 may select the processing layer specified by the user as the subsequent processing layer. Specifically, the instruction receiving unit 136 receives an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 11. Then, the extraction unit 133 uses the processing layer indicated by the instruction received by the instruction reception unit 136 as the subsequent processing layer. In the example shown in FIG. 2, the extraction unit 133 uses the second fully connected layer M9 indicated by the instruction received by the instruction receiving unit 136 as the subsequent processing layer when the user selects the second fully connected layer M9. use. The extraction unit 133 inputs the extracted first captured image output and the second captured image output to the feature point detection unit 137.

The feature point detection unit 137 detects one or more first captured image feature points based on one or more first captured image outputs, and one or more second captured images based on one or more second captured image outputs. Detect feature points. Specifically, the feature point detection unit 137 first searches for a corresponding feature point based on one or more first captured image outputs and one or more second captured image outputs. Then, the feature point detection unit 137 has one or more first captured image feature points based on one or more first captured image outputs and one or more second captured images based on one or more second captured image outputs. Detects image feature points. The feature point detection unit 137 inputs the detected first captured image feature point and the second captured image feature point to the selection unit 138.

The selection unit 138 is a part of the first captured image feature points based on the mutual correspondence from one or more first captured image feature points and one or more second captured image feature points specified by the feature point detection unit 137. And some second captured image feature points are selected. Specifically, the selection unit 138 removes the erroneously detected correspondence, and selects one or more first captured image feature points and one or more second captured image feature points based on the correspondence after the removal. The erroneous detection of the correspondence relationship is a state in which the correspondence relationship between the first captured image feature point and the second captured image feature point is inconsistent. False detection of correspondence is, for example, the distance to the subject calculated to estimate the surface position of the subject, that is, when the depth value becomes a negative number and does not fall within the normal range, or three or more captured images. In the case of generating spatial information based on the above, the depth value of the same feature point is significantly different depending on the combination of captured images. The selection unit 138 removes the correspondence by, for example, narrowing down based on the RANSAC (Random Sampling Consensus) method or the least squares median (LMedS: Least Median of Square) method.

Further, the selection unit 138 may remove the correspondence by using the multi-linear constraint of the multifocal tensor that can be calculated based on the imaging parameters of the imaging device. When the selection unit 138 generates spatial information based on, for example, two captured images, the selection unit 138 removes a correspondence relationship that does not satisfy the double linear constraint for the two captured images. Further, when the selection unit 138 generates spatial information based on, for example, three captured images, the selection unit 138 removes a correspondence relationship that does not satisfy the triple linear constraint with respect to the three captured images. Further, when the selection unit 138 generates spatial information based on, for example, four captured images, the selection unit 138 removes a correspondence relationship that does not satisfy the quadruple linear constraint with respect to the four captured images.

Further, the selection unit 138 may narrow down the captured image output extracted by the extraction process performed by the extraction unit 133 based on the multifocal tensor. Specifically, the selection unit 138 first deconvolves the captured image output from the extracted processing layer based on the captured image output extracted by the extraction unit 133, and performs imaging corresponding to the captured image output. Identify the area of the image. Then, when the specified region does not satisfy the multi-linear constraint, the selection unit 138 removes the captured image output corresponding to the region from the captured image output extracted by the extraction unit 133.

The selection unit 138 deconvolves the first captured image region, which is the region of the first captured image corresponding to the first first captured image output, in the processing layer from which the first captured image output is extracted. Identify by doing. Similarly, the selection unit 138 deconvolves the second captured image region, which is the region of the second captured image corresponding to the first second captured image output, in the processing layer from which the first second captured image output is extracted. To identify by doing.

The selection unit 138 determines whether or not the first captured image region converted on the second captured image has a part in common with the second captured image region by using the double linear constraint (for example, epipolar constraint). To do. Similarly, the selection unit 138 determines whether or not the second captured image region converted on the first captured image using the double linear constraint has a portion common to the first captured image region.

If the selection unit 138 determines that there is no common part in any of the determinations, it determines that the output should not be extracted, and the correspondence between the first captured image output and the second captured image output. To remove. In this case, the extraction unit 133 cancels the extraction process based on the one first captured image output and the first second captured image output that are in a corresponding relationship removed by the selection unit 138. On the other hand, when the selection unit 138 determines that there is a common part in the two determinations, the selection unit 138 maintains the correspondence between the one first captured image output and the first second captured image output, and then the extraction unit 133. The first captured image output and the second captured image output extracted by are narrowed down.

The spatial information generation unit 139 generates spatial information indicating the surface position of the subject in the three-dimensional space based on one or more first captured image feature points and one or more second captured image feature points. Specifically, the spatial information generation unit 139 is based on a part of the first captured image feature points and a part of the second captured image feature points based on the correspondence after the selection unit 138 removes the false detection. , Generate spatial information.

More specifically, the spatial information generation unit 139 first determines the positions of some first captured image feature points in the first captured image and the positions of some second captured image feature points in the second captured image. The distance to the subject is estimated as spatial information based on the relationship between the above and the difference between the first image pickup device and the second image pickup device. Then, the spatial information generation unit 139 generates spatial information indicating the surface position of the subject in the three-dimensional space based on the estimated distance to the subject. The spatial information generation unit 139 may generate, as spatial information, a disparity map showing the correspondence between pixels indicating the same position in the three-dimensional space in the first captured image and the second captured image. As a method of estimating the distance to the subject performed by the spatial information generation unit 139, a known technique can be used. The spatial information generation unit 139 stores the generated spatial information in the storage unit 12.

[Extraction process]
Subsequently, the extraction process performed by the extraction unit 133 will be described. As described above, the extraction unit 133 has a rear-stage extraction unit 134 and a front-stage extraction unit 135. The post-stage extraction unit 134 has a plurality of post-stage first captured image outputs and a plurality of post-stage first captured image outputs output from the post-stage processing layer because the first-stage image is propagated in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. One or more post-stage first captured image outputs that are commonly activated from the plurality of post-stage second captured image outputs output from the post-stage processing layer due to the propagation of the second captured image in the order of the pre-stage processing layer and the post-stage processing layer. And one or more subsequent second captured image outputs are extracted.

The pre-stage extraction unit 135 includes a plurality of pre-stage first captured image outputs output from the pre-stage processing layer that has been a factor in activating one or more rear-stage first captured image outputs and one or more rear-stage second captured image outputs, and Of the plurality of pre-stage second captured image outputs output from the pre-stage processing layer, one or more pre-stage first captured image outputs and one or more pre-stage second captured image outputs that are commonly activated are extracted.

4 to 9 are diagrams for explaining the extraction process performed by the extraction unit 133. 4 to 9 show a state of propagation from the pre-stage processing layer to the post-stage processing layer. In FIGS. 4 to 9, the connecting line connecting the units shown by the solid line indicates that there was an output from the connecting unit, and the connecting line indicated by the broken line indicates that there was no output from the connecting unit. In addition, the thickness of the line indicating the connecting line indicates the magnitude of the output from the unit to be connected.

In the case of FIG. 4, the post-stage treatment layer is a treatment layer (for example, an output layer or a fully connected layer) or a treatment layer before the last layer selected by the extraction unit 133 (such as a fully bonded layer or a pooling layer). Yes, the pre-stage treatment layer is a treatment layer immediately before the post-stage treatment layer (for example, a fully bonded layer or a pooling layer). In FIG. 4, it is assumed that the post-stage processing layer is the output layer M20 and the pre-stage processing layer is the fully connected layer M19.

FIG. 4A shows a state before extraction, and FIG. 4B shows a state after extraction. In the first captured image, units U5 and U8 are activated in the output layer M20, and units U2, U5, U6, U7 and U8 are activated in the fully connected layer M19. In the second captured image, the output layer M20 has the units U3 and U5 activated, and the fully connected layer M19 has the units U2, U4, U5 and U8 activated.

In this case, the post-stage extraction unit 134 is a rear-stage second captured image output output from the units U5 and U8, which are the rear-stage first captured image outputs, and the output layer M20, which are output from the output layer M20, which is the post-stage processing layer. Compare certain units U3 and U5. Then, the rear-stage extraction unit 134 extracts the unit U5 of the rear-stage first captured image output and the unit U5 of the rear-stage second captured image output that are commonly activated.

Subsequently, the front-stage extraction unit 135 includes a unit U2, which is a front-stage first captured image output, which is output from a fully connected layer M19, which is a front-stage processing layer, which is a factor that activates the unit U5 of the rear-stage first captured image output. The units U2, U5, and U8, which are the first-stage second captured image outputs, are compared with each other, which are the U5, U6, and the first-stage second captured image output units U5, which are the factors that activate the second-stage second captured image output unit U5. Then, the pre-stage extraction unit 135 extracts the units U2 and U5 of the pre-stage first captured image output and the units U2 and U5 of the pre-stage second captured image output that are commonly activated.

When the extraction unit 133 extracts the output from the output layer M20 to the fully connected layer M19, the extraction unit 133 extracts the output for the next processing layer. Specifically, the extraction unit 133 reverses the order in which the propagation control unit 132 propagates the process of extracting the first captured image output and the second captured image output that are commonly activated for each processing layer. Repeat in the order of. More specifically, the extraction unit 133 selects one of the plurality of processing layers as the post-processing layer and extracts one or more first captured image outputs and one or more second captured image outputs. The processing layer selected as the pre-stage processing layer is selected as the post-stage processing layer, and another one or more first captured image outputs and one or more second captured image outputs are extracted. By doing so, the extraction unit 133 can improve the accuracy of comparison with the first captured image and the second captured image.

FIG. 5 shows a state in which the image is propagated from the pre-stage processing layer to the post-stage processing layer based on the first captured image. FIG. 6 shows a state in which the image is propagated from the pre-stage processing layer to the post-stage processing layer based on the second captured image. In the case of FIGS. 5 and 6, the post-stage treatment layer is a fully bonded layer M18, and the front-stage treatment layer is a treatment layer other than the fully-bonded layer (for example, a pooling layer or a convolution layer). In FIGS. 5 and 6, the pretreatment layer will be described as the pooling layer M17. Further, in FIGS. 5 and 6, the pretreatment layer has three channels. The first channel in the upper row includes units U11, U12, U13, U14, and U15. The second channel in the middle stage includes units U21, U22, U23, U24, and U25. The lower third channel includes units U31, U32, U33, U34, and U35.

In the first captured image, in the pooling layer M17, the units U13 included in the first channel and the units U21 and U24 included in the second channel are activated. In the second captured image, in the fully connected layer M18, the units U22, U24, U25 included in the second channel and the units U32, U33 included in the third channel are activated.

The front-stage extraction unit 135 of the front-stage first captured image output and the rear-stage second captured image output output from the pooling layer M17, which is the front-stage processing layer, which is a factor that activates the unit U5 of the rear-stage first captured image output. The output of the second captured image in the previous stage output from the pooling layer M17, which is a factor that activates the unit U5, is compared. The pre-stage extraction unit 135 examines the presence or absence of the activated unit, and the unit U13 included in the first channel of the activated pre-stage first captured image output, U21 and U24 included in the second channel, and the pre-stage second Focus on the units U22, U24, U25 included in the second channel of the captured image output and U32, U33 included in the third channel.

Then, in the pre-stage extraction unit 135, since the channel in which the unit activated in both the pre-stage first captured image output and the pre-stage second captured image output exists is the second channel, the pre-stage first captured image The units U21, U24 included in the second channel of the output and the units U22, U24, U25 included in the second channel of the second captured image output in the previous stage are extracted.

In the case of FIG. 7, the post-stage treatment layer is the pooling layer M16, and the front-stage treatment layer is a treatment layer other than the pooling layer (for example, a convolution layer or a regularization layer). In FIG. 7, the pre-stage processing layer will be described as the convolution layer M15. Further, in FIG. 7, the pre-stage processing layer will be described assuming that it has one channel. In the first captured image, the pooling layer M16 has the unit U5 activated, and the convolutional layer M15 has the units U3 and U5 activated. In the second captured image, the pooling layer M16 has the unit U3 activated, and the convolution layer M15 has the units U3 and U4 activated.

Here, in the pooling layer that compresses the image, the extraction unit 133 extracts the output based on the degree of activation for each channel among the plurality of units bonded to the pooling layer from the immediately preceding processing layer. To do. Specifically, the pre-stage extraction unit 135 outputs one or more pre-stage first captured images based on the activated size of the plurality of pre-stage first captured image outputs and the plurality of pre-stage second captured image outputs. And one or more pre-stage second captured image outputs are extracted. The pre-stage extraction unit 135 is, for example, one or more pre-stage first captured image outputs and one or more pre-stage first captured image outputs that are most activated for each channel among the plurality of front-stage first captured image outputs and the plurality of front-stage second captured image outputs. The second captured image output of the first stage is extracted.

In this case, the post-stage extraction unit 134 selects the pooling layer M16 selected as the pre-stage processing layer in the immediately preceding extraction process, and the rear-stage first captured image output unit U5 and the pooling layer M16 output from the pooling layer M16. The unit U3 of the second-stage second captured image output output from is extracted. Then, the front-stage extraction unit 135 is the front-stage first captured image output that is most activated for each channel among the rear-stage first captured image output units U3 and U5 and the rear-stage second captured image output units U3 and U4. The unit U5 and the unit U4 of the second captured image output in the previous stage are extracted. By doing so, the pre-stage extraction unit 135 can specify a characteristic region in the image.

In the case of FIG. 8, the post-stage treatment layer is a convolution layer M14, and the front-stage treatment layer is another treatment layer including the convolution layer (for example, a regularization layer or a pooling layer). In FIG. 8, the pre-stage processing layer will be described as the normalization layer M13. Further, in FIG. 8, the pre-stage processing layer will be described as having one channel. In the first captured image, the convolution layer M14 has the unit U5 activated, and the regularization layer M13 has the units U3, U5, and U6 activated. In the second captured image, the convolution layer M14 has the unit U3 activated, and the regularization layer M13 has the units U3, U4, and U5 activated.

In this case, the post-stage extraction unit 134 selects the convolution layer M14 selected as the pre-stage processing layer in the immediately preceding extraction process, and the rear-stage first captured image output unit U5 and the convolution layer M14 output from the convolution layer M14. The unit U3 of the second-stage second captured image output output from is extracted.

Subsequently, the front-stage extraction unit 135 outputs the front-stage first captured image output from the normalization layer M13, which is the front-stage processing layer, which is a factor that activates the unit U5 of the rear-stage first captured image output, and the rear-stage second. The second pre-stage captured image output output from the normalization layer M13, which is the pre-stage processing layer that became a factor for activating the captured image output unit U3, is compared. Here, when the post-stage processing layer is a convolutional layer, the pre-stage extraction unit 135 outputs the first-stage captured image among the plurality of units of the pre-stage processing layer that are coupled to the units extracted from the post-stage processing layer by the post-stage extraction unit 134. And the second captured image output in the previous stage, the units having the same position and the same channel are extracted. In this case, the front-stage extraction unit 135 is the unit U5 of the front-stage first captured image output as a unit in which the positions of the front-stage first captured image output and the front-stage second captured image output are relatively the same and the channels are common. , U6, and units U3 and U4 of the second captured image output in the previous stage are extracted.

In the case of FIG. 9, the post-stage treatment layer is a normalization layer M12, and the front-stage treatment layer is a treatment layer other than the normalization layer (for example, a convolution layer or a pooling layer). In FIG. 9, the pretreatment layer will be described as the pooling layer M11. Further, in FIG. 9, the pre-stage processing layer will be described as having one channel. In the first captured image, the unit U5 is activated in the normalized layer M12. In the second captured image, the unit U3 is activated in the normalized layer M12.

Here, in the normalization layer that preprocesses the image, the extraction unit 133 is the center of the plurality of units included in the pretreatment layer that is bound to the unit that is activated in the post-processing layer. Extract the unit. In this case, the post-stage extraction unit 134 is the unit U5 of the post-stage first captured image output output from the normalized layer M12 selected as the post-stage processing layer, and the rear-stage second captured image output output from the normalized layer M12. Extract unit U3.

Then, the front-stage extraction unit 135 extracts the central unit U5 from the units of the pooling layer M11 coupled to the unit U5 of the rear-stage first captured image output output from the normalization layer M12. Similarly, the front-stage extraction unit 135 extracts the central unit U3 from the units of the pooling layer M11 coupled to the unit U3 of the rear-stage second captured image output output from the normalization layer M12.

It is preferable that the extraction unit 133 repeats the above-mentioned extraction process up to the input layer. However, the extraction unit 133 may end the extraction process at an intermediate process layer (for example, a pooling layer, a normalization layer, etc.) without performing the extraction process up to the first process layer. In this way, the extraction unit 133 can extract an output having a high degree of abstraction by performing the extraction process in the order opposite to the order in which the propagation control unit 132 propagates.

[Processing of spatial information generator 1]
Subsequently, the flow of processing performed by the spatial information generation device 1 will be described. FIG. 10 is a flowchart showing the flow of processing performed by the spatial information generation device 1. This flowchart starts when the first captured image and the second captured image stored in the storage unit 12 are acquired (S1).

The image acquisition unit 131 inputs the acquired first captured image and the second captured image to the propagation control unit 132. The propagation control unit 132 inputs a plurality of processing layers from the input layer M1 to the output layer M10 included in the machine learning model M to each of the first captured image and the second captured image input from the image acquisition unit 131. Propagate in order from layer M1 (S2).

The extraction unit 133 performs a process of extracting one or more first captured image outputs and one or more second captured image outputs that are commonly activated in both the processing layers of the post-stage processing layer and the front-stage processing layer (S3). .. FIG. 11 is a flowchart showing the flow of processing performed by the extraction unit 133. The extraction unit 133 determines whether or not the instruction receiving unit 136 has received an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 11 (S31).

When the extraction unit 133 determines that the instruction receiving unit 136 has received the instruction, the extraction unit 133 selects the processing layer indicated by the instruction received by the instruction receiving unit 136 as the subsequent processing layer (S32). When, for example, the extraction unit 133 determines that the instruction receiving unit 136 has received the instruction indicating the first fully connected layer M8, the extraction unit 133 processes the first fully connected layer M8 indicated by the instruction received by the instruction receiving unit 136 in a subsequent stage. Select as a layer. On the other hand, when the extraction unit 133 determines that the instruction reception unit 136 has not received the instruction, one or more first captured image outputs and one or more that are commonly activated in the rearmost layer (for example, the output layer M10). It is determined whether or not there is a second captured image output of (S33).

When the extraction unit 133 determines that there is one or more first captured image outputs and one or more second captured image outputs that are commonly activated in the output layer M10, the output layer M10, which is the rearmost layer, is moved to the subsequent stage. It is used as a processing layer (S34). On the other hand, when the extraction unit 133 determines that there is no one or more first captured image outputs and one or more second captured image outputs commonly activated in the output layer M10, each processing layer before the output layer M10 On the other hand, one or more first captured image outputs and one or more second captured image outputs that are commonly activated are repeatedly searched. Then, the extraction unit 133 removes a processing layer (for example, a second fully connected layer M9) having one or more first captured image outputs and one or more second captured image outputs that are commonly activated into a subsequent processing layer. It is used as (S35). The extraction unit 133 is one or more first layers output from the post-processing layer and the pre-processing layer based on the first captured image, which are commonly activated in both the selected post-processing layer and the pre-processing layer. Based on one captured image output and the second captured image, one or more second captured image outputs output from the post-stage processing layer and the front-stage processing layer are extracted.

Specifically, first, the post-stage extraction unit 134 starts with a plurality of post-stage first captured image outputs output from the selected post-stage processing layer and a plurality of post-stage second captured image outputs output from the selected post-stage processing layer. , One or more post-stage first captured image outputs and one or more post-stage second captured image outputs that are commonly activated are extracted (S36). The front-stage extraction unit 135 is a plurality of outputs from the front-stage processing layer that has been a factor in activating one or more rear-stage first captured image outputs and one or more rear-stage second captured image outputs extracted by the rear-stage extraction unit 134. Of the multiple pre-stage first captured image outputs and the plurality of pre-stage second captured image outputs output from the pre-stage processing layer, one or more pre-stage first captured image outputs and one or more pre-stage second captured images that are commonly activated. The image output is extracted (S37).

Subsequently, the extraction unit 133 determines whether or not there is another processing layer before the pre-stage processing layer (S38). When the extraction unit 133 determines that there is another treatment layer (for example, the third convolution layer M6) before the front treatment layer (for example, the second pooling layer M7), the extraction unit 133 puts the second pooling layer M7 in the second stage. It is selected as the processing layer (S39), and the processing is returned to S36. On the other hand, when the extraction unit 133 determines that there is no other processing layer before the pre-stage processing layer (for example, the input layer M1), the extraction unit 133 outputs one or more extracted first captured images and one or more second captured images. Is input to the feature point detection unit 137, and the extraction process is completed.

Returning to FIG. 10, the feature point detection unit 137 searches for the corresponding feature points based on one or more first captured image outputs and one or more second captured image outputs, and one or more first captured images having a corresponding relationship. One or more first captured image feature points based on the captured image output and one or more second captured image feature points based on one or more second captured image outputs are detected (S4). Subsequently, the selection unit 138 determines whether or not there are inappropriate feature points in the first captured image feature point and the second captured image feature point detected by the feature point detection unit 137 (S5). The selection unit 138 narrows down based on, for example, the RANSAC method.

When the selection unit 138 determines that the first captured image feature point and the second captured image feature point have inappropriate feature points, the selection unit 138 determines that the first captured image feature point and the second captured image feature point have inappropriate feature points, that is, the first captured image having a corresponding erroneous detection. The feature points and the second captured image feature points are removed (S6), and some first captured image feature points and some second captured image feature points based on the correspondence after the removal are selected. The spatial information generation unit 139 determines that there are no inappropriate feature points in the first captured image feature point and the second captured image feature point, or after removing the erroneously detected correspondence, one Spatial information is generated based on the above first captured image feature points and one or more second captured image feature points (S7).

Specifically, the spatial information generation unit 139 first determines the positions of some of the first captured image feature points in the first captured image and the positions of some of the second captured image feature points in the second captured image. The distance to the subject is estimated based on the relationship and the difference between the first image pickup device and the second image pickup device. Then, the spatial information generation unit 139 generates spatial information indicating the surface position of the subject in the three-dimensional space based on the estimated distance to the subject. The spatial information generation unit 139 stores the generated spatial information in the storage unit 12.

[Effect in Embodiment]
As described above, the spatial information generation device 1 propagates a plurality of processing layers included in the machine learning model M to each of the acquired first captured image and the second captured image. The spatial information generator 1 has one or more first captured image outputs and one or more second images that are commonly activated in both the processing layers of the post-stage processing layer and the front-stage processing layer in the reverse order of propagation. The captured image output is extracted for each processing layer, and the corresponding first captured image feature points and the second captured image feature points are detected, respectively. Then, the spatial information generation device 1 generates spatial information based on one or more first captured image feature points and one or more second captured image feature points from which the erroneously detected correspondence is removed.

By doing so, the spatial information generation device 1 can calculate the distance to the subject by using the machine learning model M including the convolutional neural network and obtaining the feature amount with a high degree of abstraction by deep learning. it can. As a result, the spatial information generation device 1 can improve the accuracy of estimating the surface position of the subject in the three-dimensional space.

The spatial information generator 1 avoids obstacles more safely by, for example, using multi-viewpoint images captured by a plurality of in-vehicle cameras to provide information indicating the distance to the obstacles to the automatic driving technology of the automobile. Can be made to. Further, the spatial information generation device 1 provides the automobile with information indicating the position of a pedestrian or the like that cannot be seen in the shadow from the automobile by using, for example, a multi-viewpoint image captured by using an in-vehicle camera and a road camera in combination. This makes it possible to warn the driver of pedestrians and the like jumping out. The spatial information generation device 1 can create an image as if a shadow is seen through by using spatial information indicating the position of a pedestrian or the like based on a multi-viewpoint image and a free-viewpoint image generation technique in combination.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist. is there. For example, the specific embodiment of the distribution / integration of the device is not limited to the above embodiment, and all or a part thereof may be functionally or physically dispersed / integrated in any unit. Can be done. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination has the effect of the original embodiment together.

1 Spatial information generation device 11 Operation unit 12 Storage unit 13 Control unit 131 Image acquisition unit 132 Propagation control unit 133 Extraction unit 134 Post-stage extraction unit 135 Front-stage extraction unit 136 Instruction reception unit 137 Feature point detection unit 138 Selection unit 139 Spatial information generation unit

Claims (14)

An image acquisition unit that acquires a first captured image generated by the first imaging device and a second captured image generated by a second imaging device installed at a position different from that of the first imaging device.
A propagation control unit that propagates a plurality of processing layers included in a machine learning model capable of outputting the type of the subject included in the image based on the input image to each of the first captured image and the second captured image. When,
The latter stage based on the first captured image, which is commonly activated in both the post-stage treatment layer selected from the plurality of treatment layers and the pre-stage treatment layer which is the treatment layer immediately before the post-stage treatment layer. One or more first captured image outputs output from the processing layer and the pre-stage processing layer, and one or more second captured image outputs output from the post-stage processing layer and the pre-stage processing layer based on the second captured image. Extraction part to extract
One or more first captured image feature points are detected based on the one or more first captured image outputs, and one or more second captured image feature points are detected based on the one or more second captured image outputs. Feature point detector and
A spatial information generation unit that generates spatial information indicating the surface position of the subject in a three-dimensional space based on the one or more first captured image feature points and the one or more second captured image feature points.
Spatial information generator with. The extraction unit
A plurality of post-stage first captured image outputs output from the post-stage processing layer due to the propagation of the first captured image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers, and the second-stage processing layer. One or more post-stage first captured image outputs and one or more post-stage first captured image outputs that are commonly activated from the plurality of post-stage second captured image outputs output from the post-stage processing layer because the captured images propagated in the order of the pre-stage processing layer and the post-stage processing layer. A post-stage extraction unit that extracts one or more post-stage second captured image outputs,
A plurality of pre-stage first captured image outputs output from the pre-stage processing layer, which are factors that activate the one or more post-stage first captured image outputs and the one or more post-stage second captured image outputs, and the pre-stage processing. Of the plurality of pre-stage second captured image outputs output from the layer, one or more pre-stage first captured image outputs and one or more pre-stage second captured image outputs that are commonly activated, and a pre-stage extraction unit that extracts one or more pre-stage second captured image outputs.
Have,
The spatial information generator according to claim 1. The pre-stage extraction unit has one or more pre-stage first captured image outputs and said Extract one or more previous second captured image outputs,
The spatial information generator according to claim 2. The machine learning model includes a convolutional neural network.
The post-treatment layer is any one of an output layer, a fully connected layer, a regularized layer, a pooling layer, and a convolution layer.
The spatial information generator according to claim 2 or 3. The pretreatment layer is any one of a fully bonded layer, a regularized layer, a pooling layer, a convolution layer, and an input layer.
The spatial information generator according to claim 4. When the last layer, which is the last treatment layer, is selected as the subsequent treatment layer among the plurality of treatment layers, the extraction unit is one or more of the first ones that are commonly activated in the last treatment layer. When there is no captured image output and one or more second captured image outputs, the one or more first captured image outputs and the one or more second captured images that are commonly activated in the processing layer before the rearmost layer. Extract image output,
The spatial information generator according to claim 1 or 5. From the one or more first captured image feature points and the one or more second captured image feature points specified by the feature point detection unit, a part of the first captured image feature points and a part based on the mutual correspondence relationship. Further has a selection unit for selecting the second captured image feature point of
The spatial information generation unit generates the spatial information based on a part of the first captured image feature points and the part of the second captured image feature points.
The spatial information generator according to any one of claims 1 to 6. The spatial information generation unit has a relationship between the positions of the first captured image feature points in the first captured image and the positions of the second captured image feature points in the second captured image, and The distance to the subject is estimated as the spatial information based on the difference between the first image pickup device and the second image pickup device.
The spatial information generator according to claim 7. Further, it has an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers.
The extraction unit uses the processing layer indicated by the instruction received by the instruction receiving unit as the subsequent processing layer.
The spatial information generator according to any one of claims 1 to 8. The extraction unit selects one of the plurality of processing layers as the post-processing layer, extracts the one or more first captured image outputs and the one or more second captured image outputs, and then extracts the one or more second captured image outputs. The processing layer selected as the processing layer is selected as the post-stage processing layer, and another one or more first captured image outputs and one or more second captured image outputs are extracted.
The spatial information generator according to any one of claims 1 to 9. A step of acquiring a first captured image generated by the first imaging device and a second captured image generated by a second imaging device installed at a position different from that of the first imaging device.
A step of propagating a plurality of processing layers included in a machine learning model capable of outputting the type of the subject included in the image based on the input image to each of the first captured image and the second captured image.
The latter stage based on the first captured image, which is commonly activated in both the post-stage treatment layer selected from the plurality of treatment layers and the pre-stage treatment layer which is the treatment layer immediately before the post-stage treatment layer. One or more first captured image outputs output from the processing layer and the pre-stage processing layer, and one or more second captured image outputs output from the post-stage processing layer and the pre-stage processing layer based on the second captured image. And the steps to extract
One or more first captured image feature points are detected based on the one or more first captured image outputs, and one or more second captured image feature points are detected based on the one or more second captured image outputs. Steps and
A step of generating spatial information indicating the surface position of the subject in a three-dimensional space based on the one or more first captured image feature points and the one or more second captured image feature points.
Spatial information generation method having. The extraction step
A plurality of post-stage first captured image outputs output from the post-stage processing layer due to the propagation of the first captured image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers, and the second-stage processing layer. One or more post-stage first captured image outputs and one or more post-stage first captured image outputs that are commonly activated from the plurality of post-stage second captured image outputs output from the post-stage processing layer because the captured images propagated in the order of the pre-stage processing layer and the post-stage processing layer. A post-stage extraction step for extracting one or more post-stage second captured image outputs, and
A plurality of pre-stage first captured image outputs output from the pre-stage processing layer, which are factors that activate the one or more post-stage first captured image outputs and the one or more post-stage second captured image outputs, and the pre-stage processing. among the plurality of front second image output provided from the layer, and front extraction step of extracting a common activation to which one or more of the preceding first image output and one or more of the preceding second image output,
11. The spatial information generation method according to claim 11. After executing the pre-stage extraction step, the one or more pre-stage first captured image outputs and the one or more pre-stage second captured image outputs are combined with the plurality of rear-stage first captured image outputs and the plurality of rear-stage second captured images. As an output, the latter-stage extraction step is executed.
The spatial information generation method according to claim 12. The latter-stage extraction step and the first-stage extraction step are executed for each of the plurality of processing layers.
The spatial information generation method according to claim 12 or 13.

Images