JP7373079B2 - Image recognition simulator device - Google Patents

Description

The present invention relates to an image recognition simulator device.
This application claims priority based on Japanese Patent Application No. 2020-149907 filed on September 7, 2020, the contents of which are incorporated herein.

BACKGROUND OF THE INVENTION Recently, testing of active safety systems that detect or avoid danger by installing various sensors in automobiles has been actively conducted. If an active safety system does not start as intended when it is needed, there is a risk of an accident, so it is necessary to test the system in many scenarios. However, there are limits to the ability to actually drive a vehicle and test whether the system will start up in a dangerous scene from a safety standpoint. For this reason, there is a need for a method of conducting tests using a vehicle and a driving environment that is simulated using CG (Computer Graphics) or the like.

As an example, Patent Document 1 proposes a method in which a plurality of pseudo driving scene patterns are created by superimposing images showing weather disturbances etc. on real image data and a test is performed.

Japanese Patent Application Publication No. 2010-33321

However, the method described in Patent Document 1 simply superimposes another image on the real image data, which causes a problem of lack of reality. For example, if an image of a pedestrian created using CG is simply superimposed on a live-action image, the sense of perspective will be disrupted, resulting in an unnatural-looking image.

The present invention was made to solve such technical problems, and an object of the present invention is to provide an image recognition simulator device that can create a realistic composite image.

The image recognition simulator device according to the present invention includes a distance calculation unit that calculates a distance by stereo matching based on brightness images captured by at least two cameras, and outputs a distance image representing the calculated result as an image; a region division calculation unit that performs region division on a luminance image to obtain a region division image; and a distance image error exclusion unit that excludes stereo matching errors from the distance image based on the result of region division by the region division calculation unit. a 3D space generation unit that generates a 3D space based on the range images excluded by the range image error exclusion unit; a virtual object installation unit installed in the virtual object installation unit, a virtual object synthesis unit that synthesizes the virtual object installed by the virtual object installation unit into the three-dimensional space generated by the three-dimensional space generation unit, and a virtual object synthesis unit configured to The present invention is characterized by comprising an image generation unit that generates brightness images of the two cameras based on the combined results.

In the image recognition simulator device according to the present invention, the distance image error exclusion section excludes the stereo matching error from the distance image based on the result of region division by the region division calculation section, so that the image recognition simulator device is not affected by the stereo matching error. A virtual object can be synthesized into a brightness image. Therefore, a realistic composite image can be created.

According to the present invention, a realistic composite image can be created.

1 is a schematic configuration diagram showing an image recognition simulator device according to an embodiment. FIG. 2 is a schematic configuration diagram showing a CG-natural image synthesis section of the image recognition simulator device. FIG. 3 is a schematic diagram for explaining the operation of a CG-natural image composition section. FIG. 3 is a schematic diagram for explaining the operation of a CG-natural image composition section. FIG. 3 is a schematic diagram for explaining the operation of a CG-natural image composition section. FIG. 3 is a schematic diagram for explaining the operation of a CG-natural image composition section.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments of an image recognition simulator device according to the present invention will be described with reference to the drawings.

FIG. 1 is a schematic configuration diagram showing an image recognition simulator device according to an embodiment. The image recognition simulator device 1 of this embodiment uses a natural image, CG to be synthesized, and its behavior to create a recognition application based on a plurality of time-series images collected by an in-vehicle image collection device and synchronized control signals. This is a device for performing simulations. Although not shown, the in-vehicle image collection device includes an image acquisition unit having at least two cameras (stereo cameras in this case).

A stereo camera is a vehicle-mounted camera, and includes, for example, a pair of left and right cameras arranged at a predetermined distance between optical axes (baseline length) so that their optical axes are parallel to each other, and captures images of the surroundings of the vehicle. The pair of left and right cameras each consist of an image sensor such as CMOS and an optical lens. The image captured by this stereo camera is the above-mentioned natural image.

As shown in FIG. 1, the image recognition simulator device 1 includes a CG-natural image composition section 10 and an image recognition section 20.

FIG. 2 is a schematic configuration diagram showing the CG-natural image synthesis section of the image recognition simulator device. As shown in FIG. 2, the CG-natural image synthesis section 10 includes a distance calculation section 11, a region division calculation section 12, a distance image error exclusion section 13, a three-dimensional space generation section 14, and a virtual object installation section 15. , a virtual object synthesis section 16 , and an image generation section 17 .

The distance calculation unit 11 calculates the distance by stereo matching based on the brightness image captured by the stereo camera, and outputs a distance image representing the calculated result as an image. More specifically, the distance calculation unit 11 first calculates the distance using stereo matching based on two brightness images captured by a stereo camera. At this time, the distance calculation unit 11 calculates the distance by acquiring parallax for each pixel, for example, based on the principle of triangulation. The acquired parallax can be converted into distance based on the specifications of the stereo camera used. For example, if the base length of the stereo camera is L, the CMOS size is μ, the focal length of the optical lens is V, and the parallax is d, the distance can be calculated as VL/dμ. Next, the distance calculation section 11 outputs a distance image representing the result of the calculation as described above to the distance image error exclusion section 13.

An example of stereo matching is a method performed based on local image information. In this method, a window is set near the image of interest, and the feature values within the window are obtained by calculating the similarity between the left and right images. Here, the window is assumed to have a width W and a height H, and is set to center around the pixel of interest. An example of calculating similarity is SAD (Sum of Absolute Difference).

Then, the coordinates and brightness of the right camera image are p _R =(x,y) ^T ,I(p _R ), and the coordinates and brightness of the left camera image are p _L =[x,y] ^T ,I(p _L ). When the disparity is D=[d,0] ^T and the amount of movement within the window is s=[w, h] ^T , the following formula is used as the similarity R(D) to obtain the disparity D=[d,0] ^T. It can be obtained using (1).

Furthermore, it must be noted that the parallax obtained for each pixel is determined by the window and evaluation function set above, and therefore is not a true value. A case where a parallax different from the original parallax is obtained (referred to as a false match) is called a distance error.

The region division calculation unit 12 obtains region divided images by performing region division on the luminance image captured by the stereo camera. Region segmentation refers to dividing an image into regions with similar characteristics such as edges and brightness, and labeling each divided region. At this time, for example, an algorithm applying Convolutional Neural Network (CNN) is used. Note that the labels here are, for example, roads, cars, pedestrians, grassy areas, etc., and it is preferable that IDs are set for each label.

Moreover, at this time, it is preferable that the region division calculation unit 12 divides the above-mentioned luminance image into regions more finely on the front side than on the back side. This is because the front side of the image is closer to the own vehicle than the rear side, so by dividing the image into smaller pieces, errors can be reduced and safety can be improved.

The distance image error exclusion section 13 excludes stereo matching errors from the distance image output by the distance calculation section 11 based on the results of the region division performed by the region division calculation section 12 . As described above, the distance image includes errors. For this reason, the distance image error exclusion unit 13 excludes the stereo matching error from the distance image and the region segmented image, and outputs the distance image excluding the error to the three-dimensional space generation unit 14.

Specifically, the distance image error exclusion unit 13 includes an image division unit 131 that divides the image based on the area divided image divided by the area division calculation unit 12, and an image division unit 131 that divides the image based on the area divided image divided by the area division calculation unit 12; and a distance acquisition unit 132 that acquires the distance and excludes errors in stereo matching. The image dividing unit 131 divides the image based on the area divided image, and further sets the ID specified in the area divided image to the divided image.

The distance acquisition unit 132 acquires a distance distribution based on the ID set in the region divided image, and further acquires a predetermined distance to exclude stereo matching errors. That is, the distance acquisition unit 132 acquires the distance characterized for each ID of the image, and then removes unnatural distances as errors compared to the characterized distance.

Specifically, the distance acquisition unit 132 has a mechanism for changing the distance to be acquired depending on the presence or absence of depth. An image ID is assigned as an integer, and the integer and type are internally associated with each other, for example, it is possible to have a correspondence relationship as shown in Table 1. The distance acquisition unit 132 has a distance acquisition method corresponding to each ID.

For example, in the case of ID=1, the type is a road, so the distance acquisition unit 132 implements a distance acquisition method suitable for acquiring on a road. Image collection is based on the premise that the vehicle is running on a road, and the distance on the road is set at 0 m from the vehicle's position and gradually moves farther away toward the vanishing point, reaching a maximum at infinity. The distance acquisition unit 132 corrects the distance set for each pixel based on this premise. Note that since the distance is similar in the X-axis direction and further attenuates in the Y-axis direction, an approximate curve can be drawn. Correcting the distance in this way eliminates stereo matching errors.

Furthermore, in the case of ID=2, 3, and 4, the types are automobiles (for example, preceding vehicles, oncoming vehicles), pedestrians, and two-wheeled vehicles, respectively. It is different from the road with ID=1 because cars, pedestrians, and two-wheeled vehicles move on their own. Furthermore, since the sizes and orientations vary, it can be expected that the distances of the segmented images will also include various values. At this time, since the purpose of the distance acquisition unit 132 is to acquire a distance suitable for compositing, it is not necessarily necessary to acquire the correct distance. Then, the distance acquisition unit 132 acquires the temporal and spatial distribution of distances by acquiring temporal changes in the distance image and the region-divided image, thereby making the fluctuations in distance uniform. By equalizing the variation in distance in this way, errors in stereo matching are eliminated. Note that the temporal or spatial distance distribution may be calculated for the entire image region, projected onto the X-axis or Y-axis, or weighted temporally.

Also, if ID=5, the type is grass. Grass and lawns have irregular patterns, and this is a case where local stereo matching using windows, as considered in this embodiment, is not suitable. Therefore, it is assumed that there will be a relatively large number of erroneous matchings, or that the distance will not be determined in many cases. In such a case, it is possible to estimate and use the distance of the road, which is relatively easy to estimate the correct distance, as the distance of the grass, rather than the grass itself. By doing so, errors in stereo matching are eliminated. In this case, if one image obtained by region division is divided according to changes in distance on the road surface, CG synthesis, which will be described later, will be easier to perform.

In this embodiment, the distance acquisition unit 132 acquires distances corresponding to each ID set in the region segmented image, and eliminates stereo matching errors based on the acquired distances. Exclusion accuracy can be increased.

The three-dimensional space generation section 14 generates a three-dimensional space based on the distance images excluded by the distance image error exclusion section 13. More specifically, the three-dimensional space generation unit 14 generates a three-dimensional space from the distance images excluded by the distance image error exclusion unit 13 and the brightness images captured by the stereo camera . Since the brightness image is assumed to be obtained by orthographic projection, coordinates in a three-dimensional space can be obtained by combining it with a distance image based on the use of a stereo camera.

The virtual object installation unit 15 installs a virtual object that is recognized as a target for the vehicle-mounted camera application at an arbitrary position and time. More specifically, the virtual object installation unit 15 determines the type of CG of the virtual object to be synthesized, and determines the time and position at which the virtual object is installed. At this time, the virtual object installation unit 15 determines the installation position and time of the virtual object based on information on how to move the virtual object and the travel distance of the own vehicle obtained from the control signal of the own vehicle. is preferred. In this way, it is possible to obtain a composite image that is closer to the real image, which has the effect of improving the reliability of the simulation. Further, the virtual object installation unit 15 installs the virtual object based on the determined result.

Note that the virtual object is an object that is recognized as a target for an in-vehicle camera application, and includes, for example, a car, a pedestrian, a two-wheeled vehicle, and the like. Furthermore, since the virtual object can be generated in a pseudo manner, its speed and size can be freely set.

The virtual object synthesis section 16 synthesizes the virtual object installed by the virtual object installation section 15 into the three-dimensional space generated by the three-dimensional space generation section 14. At this time, the virtual object synthesis section 16 synthesizes the virtual object installed by the virtual object installation section 15 at a predetermined position in the three-dimensional space.

The image generation unit 17 generates a brightness image of the stereo camera based on the result of synthesis by the virtual object synthesis unit 16. At this time, the image generation unit 17 generates brightness images of the left and right cameras obtained by the stereo cameras from the three-dimensional space in which the virtual objects are synthesized.

On the other hand, the image recognition unit 20 recognizes the brightness images of the left and right cameras generated by the image generation unit 17.

The operation of the CG-natural image composition section 10 will be described below with reference to FIGS. 3 to 6.

First, the distance calculation unit 11 calculates a distance by stereo matching based on a brightness image captured by a stereo camera (see FIG. 3). In FIG. 3, the left diagram is a simulated representation of a brightness image obtained by a stereo camera, and the right diagram is a representation of a distance image obtained by calculation by the distance calculation unit 11 using color shading. The color shading shown in the diagram on the right is set based on the distance of each object.

Furthermore, there are multiple irregular points in the distance image shown in the right figure. These irregular points represent distance errors caused by stereo matching errors. If such a distance error is included in the distance image, the front and back relationship with the object will change when CG is combined, resulting in an unnatural looking image. In other words, when a virtual object is synthesized with a distance image that includes these distance errors, a problem arises where, for example, a part of the lawn that is supposed to be in the back of the virtual object appears to be in the foreground, and the virtual object is partially hidden. Since it is occluded, the image becomes unnatural.

Subsequently, the region division calculation unit 12 performs region division on the brightness image captured by the stereo camera (see FIG. 4). As shown in FIG. 4, as a result of region segmentation performed on the brightness image acquired by the stereo camera, labels such as "car," "road," "lawn," and "other" are assigned, for example.

Subsequently, the distance image error exclusion unit 13 excludes stereo matching errors from the distance image based on the results of region division by the region division calculation unit 12 (see FIG. 5). Specifically, the distance acquisition unit 132 of the distance image error exclusion unit 13 recalculates the distance for each label based on the labeling obtained by the above-mentioned region division, and eliminates the distance error (i.e., the right side in FIG. 3). (Irregular points in the diagram) are removed. By recalculating the distance from the result of labeling by region division in this way, it is possible to effectively exclude errors caused by stereo matching, and it is possible to suppress the synthesis of unnatural images.

Next, the three-dimensional space generation unit 14 generates a three-dimensional space based on the distance image with distance errors removed, and the virtual object installation unit 15 installs a virtual object (in this case, a pedestrian) at an arbitrary position and time. do.

Next, the virtual object synthesis unit 16 places the pedestrian as a virtual object in the three-dimensional space generated by the three-dimensional space generation unit 14 and synthesizes CG (see the left diagram in FIG. 6). The image generation unit 17 generates a brightness image of the stereo camera based on the CG synthesized by the virtual object synthesis unit 16 (see the right diagram of FIG. 6).

In the image recognition simulator device 1 of this embodiment, the distance image error exclusion unit 13 excludes the stereo matching error from the distance image based on the result of region division by the region division calculation unit 12, which affects the stereo matching error. A virtual object can be synthesized into a brightness image without being Therefore, a realistic composite image can be created.

Furthermore, since natural images captured by a stereo camera are used, it is possible to further enhance the reality and improve the reliability of the simulation compared to the case where everything is created using CG. Furthermore, since CG images (i.e., virtual objects) of cars, pedestrians, motorcycles, etc. are combined with natural images, it is possible to easily increase the variety of images.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various designs can be made without departing from the spirit of the present invention as described in the claims. Changes can be made.

1 Image recognition simulator device 10 CG-natural image synthesis section 11 Distance calculation section 12 Area division calculation section 13 Distance image error exclusion section 14 3D space generation section 15 Virtual object installation section 16 Virtual object synthesis section 17 Image generation section 20 Image recognition Section 131 Image division section 132 Distance acquisition section

Claims (3)

a distance calculation unit that calculates a distance by stereo matching based on brightness images captured by at least two cameras, and outputs a distance image representing the calculated result as an image;
a region division calculation unit that performs region division on the luminance image to obtain a region divided image; and a distance image error that excludes stereo matching errors from the distance image based on the result of region division by the region division calculation unit. an exclusion section;
a three-dimensional space generation unit that generates a three-dimensional space based on the distance image excluded by the distance image error exclusion unit;
a virtual object installation unit that installs a virtual object recognized as a target for an in-vehicle camera application at an arbitrary position and time;
a virtual object synthesis unit that synthesizes the virtual object installed by the virtual object installation unit into the three-dimensional space generated by the three-dimensional space generation unit;
an image generation unit that generates brightness images of the two cameras based on the results of the synthesis by the virtual object synthesis unit;
Equipped with
The image recognition simulator device is characterized in that the region division calculation unit divides the brightness image into regions on a front side more finely than on a back side . The distance image error exclusion unit includes an image division unit that divides the image based on the region divided image divided by the region division calculation unit, and a stereo matching by acquiring distance for each image divided by the image division unit. The image recognition simulator device according to claim 1 , further comprising a distance acquisition unit that excludes an error in the image recognition simulator. 2. The virtual object installation unit determines the installation position and time of the virtual object based on the movement information of the virtual object and the travel distance of the vehicle, and installs the virtual object based on the determined result. 2. The image recognition simulator device according to 2 .

Images