JP2020012735A - On-vehicle environment recognition device - Google Patents

Abstract

To provide an on-vehicle environment recognition device which can easily correct the geometry of cameras even when one object is shown differently in images taken by a plurality of cameras.SOLUTION: The on-vehicle environment recognition device includes: a first camera 100 and a second camera 110; and a controller 10 for deforming at least one of a first image taken by the first camera and a second image taken by the second camera to perform a viewpoint conversion of converting the first image and the second image into images with the same viewpoint, and thereafter extracting a plurality of corresponding points and correcting the geometry of the first camera and the second camera, using the coordinates of the corresponding points in the first image and the second image before the conversion of the viewpoint.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an in-vehicle environment recognizing device that recognizes a surrounding environment of a vehicle using a camera installed in the vehicle.

  There is a technology related to an in-vehicle camera that recognizes a surrounding environment of a vehicle using two cameras installed in the vehicle. The commercialization of preventive safety technology is entering the spread period, and multifunctional and wide-field sensing is required at low cost.

  There is a method of estimating the position and orientation between two cameras installed in a vehicle and performing geometric calibration so that the images of the two cameras have a parallel positional relationship. At this time, a method of using corresponding points obtained from images of the two cameras is generally used as information for obtaining the geometric relationship between the positions and postures of the two cameras. The method of acquiring the corresponding point is to extract a unique point such as a corner (corner) on the image which is called a feature point from the left and right images, calculate a feature amount from a luminance change around the feature point, and calculate the feature amount. Feature points having similar amounts are searched for on the left and right images, and feature points found are set as corresponding points, and geometric calibration is performed based on the coordinates of the corresponding points in the left and right images.

  For example, Japanese Patent Application Laid-Open No. 2014-74632 (Patent Literature 1) describes that a distance to an object is calculated and a three-dimensional position of the object is estimated based on outputs of left and right cameras arranged so that their fields of view overlap. A three-dimensional coordinate estimating unit, an inter-camera parameter storage unit for storing inter-camera parameters used for estimation processing in the three-dimensional coordinate estimating unit, and a planar object whose size is known in advance based on the outputs of the left and right cameras A calibration device for an in-vehicle stereo camera including an object recognition and feature point collection unit for recognizing and acquiring feature point coordinates of an object, and an inter-camera parameter estimation unit for obtaining an inter-camera parameter based on the feature point coordinates is disclosed. Have been.

JP 2014-74632 A

  However, when the scenery or object between the two cameras is close to the camera or when the two cameras are far apart, the same object is observed from a greatly different viewpoint, and the two images are used. Of the same object may be greatly different. When the appearance of the same object is greatly different in this way, even if the feature point to be correlated with the two images is successfully extracted, the feature amount calculated based on the luminance change around the point between the two cameras Likely to be different. That is, since the feature values of the two feature points are calculated as different values, no corresponding point is found, an incorrect corresponding point (erroneous corresponding point) is found, or even if a corresponding point is found, the number is very small. And the like. The smaller the number of corresponding points or the more erroneous corresponding points, the lower the accuracy of the geometric calibration of the camera. Further, there may be a case where the position and orientation between the two cameras cannot be estimated by the convergence calculation.

  An object of the present invention is to provide an in-vehicle environment recognizing device that can easily perform a geometric calibration of a camera even when an image captured by a plurality of cameras includes the same object having a significantly different appearance.

  The present application includes a plurality of means for solving the above-mentioned problems. For example, a first camera and a second camera, a first image captured by the first camera, and an image captured by the second camera are provided. After performing viewpoint conversion for converting the first image and the second image to an image from a common viewpoint by deforming at least one of the second images, a plurality of corresponding points are extracted, and the plurality of corresponding points are extracted before the viewpoint conversion. A control device for performing geometric calibration of the first camera and the second camera using coordinates of the plurality of corresponding points in the first image and the second image.

  According to the present invention, dense corresponding points are extracted on an image by performing viewpoint conversion in a common visual field region of a plurality of cameras installed in a vehicle, and geometric calibration is performed based on the corresponding points. Thus, the position and orientation between the cameras can be estimated with high accuracy, and highly accurate parallelization of the two cameras can be realized. By performing stereo matching in a state where high-precision parallelization is performed, generation of a high-density parallax image is realized, and high-precision distance restoration is enabled from the parallax.

FIG. 1 is a configuration diagram of an in-vehicle environment recognition device according to a first embodiment. FIG. 3 is a functional block diagram of a viewpoint conversion unit. FIG. 3 is a functional block diagram of a conversion parameter generation unit. FIG. 3 is a functional block diagram of a corresponding point search unit. FIG. 3 is a functional block diagram of a camera geometric calibration unit. FIG. 3 is a functional block diagram of a parallax image generation unit (first embodiment). FIG. 9 is a functional block diagram of a parallax image generation unit (second embodiment). Description of the process of calibrating camera geometry from corresponding points. Explanation of the problem of acquiring corresponding points. Explanation of solution by viewpoint conversion. An example of viewpoint conversion of upper and lower area division. An example of viewpoint conversion by shearing (approximate deformation). 6 is an example of viewpoint conversion in six regions. Explanatory drawing about an inverse transformation. 3 is a processing flowchart of a control device according to the first embodiment. 4 is an example of viewpoint conversion for free area division. An example of viewpoint conversion for left and right cameras. 9 is a flowchart of a parallax image generation process performed by the control device according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

-First Embodiment <Fig. 1 In-vehicle environment recognition device>
FIG. 1 shows a configuration diagram of an in-vehicle environment recognition device 1 according to the present embodiment. The in-vehicle environment recognizing device 1 includes a left camera (first camera) 100 and a right camera (second camera) 110 arranged at intervals in the left and right directions in the horizontal direction, and imaging output from the two cameras 100 and 110. A process of performing geometric calibration of the two cameras 100 and 110 based on an image (an image obtained by the left camera 100 may be referred to as a first image and an image obtained by the right camera 110 may be referred to as a second image); A control device (computer) 10 that executes processing for creating a parallax image by performing stereo matching on two images captured by the two cameras 100 and 110 at the same timing is provided.

  The control device (computer) 10 includes an arithmetic processing device (eg, CPU) not shown, a storage device (eg, memory, hard disk, flash memory) for storing a program executed by the arithmetic processing device, and internal devices. And a communication device for performing communication with external devices. The control device 10 functions as the viewpoint conversion unit 200, the corresponding point search unit 300, the camera geometric calibration unit 400, and the parallax image generation unit 500 by executing a program stored in the storage device. Other functions can be implemented by adding a program.

  In general, a stereo camera composed of two cameras on the left and right recognizes the surrounding environment of the vehicle using a common field of view. After attaching the left and right cameras to the predetermined position of the support with high accuracy, the camera factory estimates the position and orientation of the left and right cameras while capturing the calibration chart with the left and right cameras, and uses the results to The parameters are corrected so that the captured images are parallel to each other. If stereo matching is performed in this state, it is possible to obtain a high-density parallax and a parallax image, and it is possible to measure a distance with high accuracy from the parallax image. However, high-precision camera manufacturing and parts and structures that suppress deformation of the camera due to temperature change, shock, vibration, aging, and the like are expensive.

  The present inventors can easily correct the deformation of the camera due to the positional shift of the camera, the temperature, the aging, etc. in order to suppress the cost of this kind, and realize the high-precision calibration during the traveling. As such, the in-vehicle environment recognition device 1 of the present embodiment was invented.

  First, in the present embodiment, similarly to the known technique, feature points (unique points such as corners (corners) of an object on the image) are extracted from the images of the left and right cameras 100 and 110, and the feature points are extracted. A feature amount is calculated from a change in surrounding brightness, a feature point having a similar feature amount is searched for on the left and right images, and a set of feature points having similar feature amounts on the left and right images is set as a set of corresponding points. I do. The image of FIG. 8 is an image in which a plurality of corresponding points on the searched left and right images are connected to each other with a line, and geometric calibration of the left and right cameras 100 and 110 is performed based on the plurality of corresponding points.

  In FIG. 1, the corresponding point search unit 300 extracts the above-described feature point from the images of the left and right cameras 100 and 110, calculates a feature amount from a luminance change around the feature point, and calculates the feature amount on the left and right images. A feature point (corresponding point) having a similar feature amount is searched.

  The camera geometric calibration unit 400 calculates geometric calibration parameters for making the left and right images parallel based on the plurality of corresponding points obtained by the corresponding point search unit 300. The image obtained by geometrically calibrating the images of the left and right cameras 100 and 110 is an image having a horizontal positional relationship and no lens distortion at all. This makes it possible to prepare left and right images that are easily geometrically matched.

  The parallax image generation unit 500 performs stereo matching on two images (parallelized images) captured at the same timing and corrected by the geometric calibration parameters of the camera geometric calibration unit 400, and performs two-dimensional matching on the two images. The distance information (parallax information) indicating the deviation of the position where the same object (the same pattern) is photographed is calculated by a known method to generate a parallax image (distance image). The parallax image generation unit 500 uses left and right images captured by the left camera 100 and the right camera 110. In the present embodiment, since the stereo matching is performed based on the right image (second image) from the right camera 110, the sensitivity, geometry, and the like are basically matched to the right reference. The parallax image generation unit 500 receives the images of the left and right cameras to which the geometric and sensitivity corrections have been applied, performs stereo matching, generates a parallax image, and finally performs noise removal to remove the noise-reduced parallax. An image is obtained.

  In the present embodiment, the processing performed by the viewpoint conversion unit 200 is the most characteristic part. The viewpoint conversion unit 200 deforms at least one of the left image (first image) captured by the left camera 100 and the right image (second image) captured by the right camera 110, thereby forming the left and right images (first image and first image). Viewpoint conversion for converting the second image) into an image from a common viewpoint. This makes it easier for the corresponding point search unit 300 to find corresponding points from the left and right images. The details of the image deformation method (viewpoint conversion method) will be described later. For example, the image (the left image, the first image) captured by the left camera 100 is made to approach the appearance from the viewpoint of the right camera 110. (For example, by deforming the image by affine transformation including scaling, rotation, translation, and shearing of the image), the whole or a part of the region on the image has a larger number than before the viewpoint transformation. The search result of the dense corresponding point can be obtained by the corresponding point search unit 300. By utilizing these dense corresponding points, high-precision geometric calibration is realized in the processing of the camera geometric calibration unit 400 performed in the subsequent stage. By using the paired right and left images accurately parallelized by the geometric calibration, the parallax image generation unit 500 can generate a high-density and high-precision parallax image.

  In the case of a stereo camera with a long base line in which the distance between the two cameras is relatively large, or when a short-distance object is imaged, the same object is imaged from completely different angles, and the right and left images are imaged. Since the appearance of the object is greatly different, it is difficult to obtain a corresponding point by a general corresponding point extraction method for searching for similar feature amounts on the left and right images. For example, in the left and right images shown in the upper part of FIG. 9, at a long distance, the deformation in the left and right images is relatively small, so that it is easy to obtain a corresponding point. However, at a short distance such as an object 91 placed on a road surface, The deformation is relatively large and it is difficult to obtain corresponding points densely. As a corresponding point search method, as shown in (2) and (3) of FIG. 10, the viewpoint (position on the image) of four points on the road surface in the left image is matched with or similar to the right image. After conversion, search for corresponding points. In this manner, in the image on the lower side of FIG. 9 before the deformation, the corresponding point could hardly be searched for from the road surface, but the corresponding points were densely obtained from the road surface after the deformation as shown in the lowermost part of FIG. Can be confirmed.

  In this way, by utilizing image deformation, for example, deformation by viewpoint transformation, dense correspondence points are obtained, high-precision geometric calibration is performed, parallelization is performed, and then stereo matching is performed. Achieve accurate parallax image generation and ranging.

<FIG. 2 Viewpoint conversion unit 200>
Next, an example of viewpoint transformation (image deformation) that can be performed by the viewpoint transformation unit 200 will be described. As illustrated in FIG. 2, the viewpoint conversion unit 200 includes an area setting unit 210 that sets an area where at least one of the left and right images is to be subjected to viewpoint conversion and an area that is not to be subjected to viewpoint conversion. A transformation parameter generation unit 220 that generates a matrix or a parameter of a function necessary for transforming the left and right images into an image from a common viewpoint by transforming, and a parameter (conversion parameter) generated by the conversion parameter generation unit 220 A viewpoint conversion image generation unit 240 that generates an image (viewpoint conversion image) in which at least one of the left and right images is viewpoint-converted by using a matrix or a function included therein, An inverse transformation parameter calculation unit 230 that generates parameters (inverse transformation parameters) of matrices and functions is provided.

  The viewpoint converted image generated by the viewpoint converted image generation unit 240 is used in the corresponding point search in the corresponding point search unit 300. Further, the inverse transformation parameter calculated by the inverse transformation parameter calculation unit 230 is used by the corresponding point inverse transformation correction unit 410 of the camera geometric calibration unit 400.

  Note that the region setting unit 210 may set all the regions in the camera image as regions where viewpoint conversion is performed or regions where viewpoint conversion is not performed. As a common viewpoint to which the left and right cameras 100 and 110 are converted, there is a viewpoint of either one of the left and right cameras 100 and 110 and an arbitrary viewpoint different from any of the left and right cameras 100 and 110. In the latter case, for example, a predetermined viewpoint (for example, the middle point between the left and right cameras 100 and 110) located between the left camera 100 and the right camera 110 can be set as a common viewpoint to perform viewpoint conversion.

(1) Simple viewpoint conversion As one of viewpoint conversions, a part (overlap area) where the viewpoint conversion unit 200 of the control device 10 is overlapped and captured in the left and right images (the first image and the second image). Is assumed to be a plane, the positions and orientations of the left camera 100 and the right camera 110 with respect to the plane are geometrically calculated based on the left and right images, and the left camera 100 and the right with respect to the calculated plane are calculated. Some cameras convert a left image (first image) into an image from the viewpoint of the right camera 110 based on the position and orientation of the camera 110.

  In the example of FIG. 10 regarding this method, assuming that a part captured in the image of the left camera 100 is a road surface (plane), the left and right cameras 100 and 110 and an arbitrary point on the road surface (plane) are assumed. Obtain dense corresponding points by calculating the relative position / posture camera geometrically and generating an image obtained by transforming the image of the left camera 100 into the position / posture of the right camera 110 based on the position / posture. are doing.

  However, here, in the image (original image) shown in the lower part of FIG. 9 before the deformation, many dense corresponding points are extracted on a distant three-dimensional object, and conversely, the image shown in the lower part of FIG. In the (viewpoint conversion image), many dense corresponding points are obtained from the road surface instead of extracting much corresponding points on a distant three-dimensional object. In FIG. 10, the left image is subjected to horizontal shear deformation parallel to the left and right directions so that the road surface at the lower left in the original image of the left camera 100 has the same appearance as the road surface in the image of the right camera 110. Thereby, as a result of the shape of the road surface being similar in the left and right images, corresponding points on the road surface were densely obtained. However, since this viewpoint conversion deforms the entire left image by focusing only on the road surface which is only a part of the left image, the viewpoint conversion is performed by focusing on the shape of a three-dimensional object that does not exist on the road surface (particularly, the building shown in FIG. 10). It can be confirmed that the image is greatly inclined after the conversion. When such a deformation is applied, the appearance (shape) of the building before and after the deformation is more similar on the left and right images, and the original image (the original image) is denser at a distance from the cameras 100 and 110. It can be understood that a corresponding point can be obtained.

(2) Perspective conversion by upper and lower two divisions In this embodiment, in order to obtain more corresponding points from the left and right images, the left image is divided into two types of regions, a region where viewpoint conversion is performed and a region where viewpoint conversion is not performed. The search is performed for the corresponding points by division. The former area in which the viewpoint conversion is performed can be said to be an area in which the number of corresponding points increases when the viewpoint conversion is performed. The latter area in which viewpoint conversion is not performed can be said to be an area in which dense corresponding points can be obtained even in the original image, and corresponds to, for example, an area far from the camera.

  As this kind of viewpoint conversion, as shown in FIG. 11, the left image is vertically divided into two by a horizontal boundary line based on a vanishing point VP in the image, and the upper region (upper region) of the two regions is divided. In some cases, the region (region) 111 is set as a region in which viewpoint conversion is not performed, and the lower region (lower region) 112 is set as a region in which viewpoint conversion is performed. Specifically, the viewpoint conversion unit 200 of the control device 10 includes an upper region (distant view) 111 including a vanishing point VP or having a boundary in contact with the vanishing point VP, and a lower region (below the upper region 111). The left image (first image) is divided vertically into two areas of two road areas 112, and it is assumed that at least a part of the lower area 112 is a plane, and the positions and postures of the left and right cameras 100 and 110 with respect to the plane. Is calculated, and the lower area 112 is converted into an image from the viewpoint of the right camera 110 based on the calculated position and orientation of the left and right cameras 100 and 110 with respect to the plane. (No viewpoint conversion) A combination of 111 and the left image (first image) after viewpoint conversion.

  This viewpoint conversion is one of the simplest and most effective ones in camera geometric calibration of an in-vehicle camera (in-vehicle environment recognition device). In addition, this method uses an image after viewpoint conversion (viewpoint conversion image) with hardware such as an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and a GPU (Graphics Processing Unit). Assuming, conversion is simplified with the restriction that the image is divided only into rectangles. A distant road surface can also be deformed on the assumption that it is flat, but if it is far away, the amount of deformation of the road surface due to viewpoint conversion is small. Further, since a three-dimensional object in a distant view is not significantly deformed by the viewpoint conversion, the upper region 111 including the vanishing point VP is set in the region setting unit 210 as a region where the viewpoint is not changed. In the case of an in-vehicle camera, since the road on which the host vehicle travels is basically captured in the area 112 on the lower side of the screen, it is very likely that a road surface in which many areas are flat is captured. Therefore, assuming that the lower region 112 is a region where the road surface is imaged, the lower region 112 is set by the region setting unit 210 as a region to be changed in viewpoint.

  The viewpoint conversion unit 200 performs the viewpoint conversion of the lower region 112 using the conversion parameters generated by the conversion parameter generation unit 220. As a simple viewpoint conversion method, there is a method in which conversion parameters are generated by the conversion parameter generation unit 220 using design values of the positions and postures of the cameras 100 and 110 mounted on the vehicle. When the positions and orientations of the cameras 100 and 110 are corrected from the design values by the image processing of the in-vehicle environment recognition device 1, the corrected values may be used. If the posture of the road surface is estimated at the same time as the position and posture of the camera based on the distance information of at least three points on the road surface, the posture information of the road surface may be used. Here, the reason that the design values can be used as the position and orientation of the left and right cameras is that it is important that the left and right images are more similar than before the viewpoint conversion, and the two do not need to be mathematically completely identical. Because. This is because the purpose of the viewpoint conversion is to obtain corresponding points densely in the image after the viewpoint conversion, and it suffices to generate an image similar to the degree that the corresponding points can be obtained. Further, the coordinates of the corresponding point after the viewpoint conversion by the conversion parameter are returned to the coordinates before the viewpoint conversion by the inverse conversion parameter, and are used for the geometric calibration of the camera. Therefore, a high-precision camera position / posture is not always necessary.

(2-1) Viewpoint transformation by upper and lower division (affine transformation including scaling, rotation, translation, and shearing of image)
As described above, the viewpoint conversion may be mathematically performed based on the relative positions and postures of the cameras 100 and 110 and the road surface. However, the main purpose of the viewpoint conversion is not accurate conversion. It is to obtain corresponding points of the left and right images densely. For this reason, even if the conversion is not mathematically accurate, it can be replaced by a method that has a certain effect. The conversion based on the mathematical operation in hardware may be omitted.

  For example, when the positions and orientations of the left and right cameras 100 and 110 are unknown (including the case where the calculation is not possible and the calculation is not possible) and the amount of deformation of the image is not determined, the conversion parameter generation unit 220 sets the center of the vanishing point VR as the center. The viewpoint transformation may be performed by transforming the image by affine transformation including the rotation of the image and the shear deformation of the road surface.

  As a specific method of viewpoint conversion in this case, the viewpoint conversion unit 200 of the control device 10 changes the parameters (matrix parameters for affine transformation) to the left image (first image) from the left camera 100 while changing the affine. Conversion is performed to generate a plurality of converted images, and corresponding points of each of the plurality of converted images and the right image (second image) are extracted. There is a process of using a large number of converted images that are equal to or more than a predetermined threshold (reference value) in a final viewpoint converted image to be used in subsequent processing (that is, an image obtained by converting a left image into an image from the viewpoint of the right camera 110). When generating a plurality of transformed images by this method, for example, affine transformation is performed for all combinations including at least one of scaling, rotation, translation, and shearing of the image while gradually changing parameters. As a result, converted images of substantially all patterns and a conversion matrix used for the converted images can be generated. At this time, the number of converted images can be adjusted at intervals at which parameters are changed. Then, a search for corresponding points with the right image is performed for all of the generated converted images, and the number of corresponding points obtained is compared, thereby transforming the left image into a shape most similar to the right image. You can get the image and the transformation matrix. The parameters of the transformation matrix thus obtained can be used as transformation parameters.

  A method of viewpoint transformation using shear deformation as affine transformation will be described with reference to FIG. In the example of FIG. 12, while generating a plurality of converted images by continuously increasing the amount of shear deformation (shear amount) of the road surface region of the left image at predetermined intervals, a corresponding point of each converted image with the right image is searched. It is determined that the shear amount of the converted image in which the number of corresponding points is the largest and which is larger than the reference (predetermined threshold) is used as the conversion parameter.

  Each time the shear amount is increased to Apixel, Bpixel, Cpixel (where A> B> C), the deformation of the road surface area (lower area) of the left image is increased, and each time, the correspondence between the road surface of the left image and the right image is increased. Record the score. Thereafter, when the deformation was continued up to Cpixel, it was found that there was a peak of the number of corresponding points near Bpixel. In such a case, it is indicated that when the shear amount is Bpixel, the deformation most similar to the right image has been performed.

  For this reason, when calculating the conversion parameters, it is necessary to correct the relationship between the road surface and the position and orientation of the camera in consideration of the occupant's riding and pitching while traveling. Dense corresponding points can be obtained by a simple method. Not only are the parameters fixed each time, but also a common deformation, it is easy to use and share hardware with relatively low-spec hardware. Although the case where shearing is used has been described here, the viewpoint conversion can be performed using the same approach as described above when using affine transformation such as rotation, enlargement / reduction, or translation.

  When the viewpoint conversion of the lower region 112 is completed, the viewpoint conversion image generation unit 240 combines the lower region 112 after the viewpoint conversion and the upper region 111 that has not been subjected to the viewpoint conversion into a viewpoint conversion image (the left side after the viewpoint conversion). Image). The viewpoint conversion image generated by the viewpoint conversion unit 200 is compared with the right image by the corresponding point search unit 300 to extract a corresponding point. The coordinates of the extracted corresponding points are returned to the coordinates in the original image by the inverse transformation by the camera geometric calibration unit 400 as shown in FIG. 14, and are used for camera geometric calibration.

  As described above, by dividing the image into the upper region 111 including a distant view or a three-dimensional object and the lower region 112 occupying the majority of the road surface, corresponding points are obtained. Is extracted in the lower area 112, and corresponding points are extracted from the image with the viewpoint conversion, and dense corresponding points can be obtained from above and below. The viewpoint conversion unit 200 performs processing from generation of a viewpoint conversion parameter to generation of a viewpoint conversion image and calculation of an inverse conversion parameter. The inverse transformation parameter is used in the camera geometric calibration unit 400 that integrates and uses feature points.

<Fig. 3 Conversion parameter generator>
As illustrated in FIG. 3, the conversion parameter generation unit 220 includes a parallax analysis unit 221, an attribute determination unit 222, and a conversion parameter calculation unit 223.

  The parallax analysis unit 221 acquires the parallax of the region set as the region for performing the viewpoint conversion by the region setting unit 210 from the parallax image of the previous frame (for example, one frame before) generated by the parallax image generation unit 500, By analyzing the parallax, it is determined whether or not a portion that can be approximated to a plane exists in the region.

  The attribute determining unit 222 determines the plane attribute of the area based on the analysis result of the parallax analyzing unit 221. The plane attributes include “plane” indicating that there is a portion that can be approximated to a plane in the area, and “non-plane” indicating that there is no part that can be approximated to the plane in the area. The former “plane” attribute further includes “road surface (ground)” and “wall surface (wall)” as attributes indicating the type of the area. The latter attribute of “non-planar” includes “infinity” as an attribute indicating the type of the area. In addition, each region may be given a plane attribute in advance. In this case, the attribute determination unit 222 determines whether the plane attribute given in advance is appropriate based on the analysis result of the parallax analysis unit 221, and the distance from the left and right cameras 100 and 110 in the area determined to be appropriate is determined. The plane attribute of the area smaller than the predetermined threshold is determined to be “plane”, and the area is determined as the target area of the viewpoint conversion. By applying the condition that the distance is less than the predetermined threshold, viewpoint conversion can be performed only when necessary, that is, only when the appearance of the same object is significantly different between the left and right images.

  The conversion parameter calculation unit 223 calculates a conversion parameter for performing viewpoint conversion on the area whose plane attribute is determined to be a plane by the attribute determination unit 222. The conversion parameter can be calculated by a known method. Here, an example will be described.

<Viewpoint conversion calculation>
First, the three-dimensional coordinates of the four corners of the plane in the conversion target area viewed from the left camera 100 are calculated. As a simple method, it is assumed that the road surface plane is at a height of 0 cm, the camera internal parameters (f, kn, Cn), the external parameters (rnn, tn), and the installation position / posture of the left camera 100 with respect to the world coordinate origin. Is known, the first and second matrices on the right side of the following equation (1) are known. Further, since the world coordinate of Yworld of a vector having four elements located at the right end of the right side of the equation is 0, unknown numbers can be represented only by Xworld and Zworld of world coordinates. The image coordinates can be known because the region to be transformed is set by itself. In this way, if the image coordinates of the four corners of the plane in the area to be transformed are input to equation (1), the three-dimensional world coordinates of the four corners can be calculated. Next, the three-dimensional coordinates of the four corners calculated by calculation are sequentially set to Xworld, Yworld, and Zworld at the right end of the right side. When the position and orientation of the right camera as viewed from the origin of the world coordinates are set in the matrix of external parameters, the positions of four points in the world coordinate system can be converted into image coordinates as viewed by the right camera. In this way, the image coordinates of the four corners of the plane in the region to be subjected to the viewpoint conversion are obtained to obtain the quadrangle conversion parameters. If the four corners can be calculated, all the coordinates inside the quadrangle can be calculated by interpolation.

  When generating the conversion parameters, first, it is necessary to determine a plane to be used for viewpoint conversion in the left image (view target conversion target image). For example, if the left image is divided into upper and lower parts as described above, the parallax analyzer 221 estimates that the lower region 112 where the road surface is often imaged is a portion that can be approximated to a plane. Whether or not each area includes a portion that can be approximated to a plane can be analyzed using parallax obtained from a stereo image. Assuming that the three-dimensional survey is performed with the left camera 100 as the center, the position of the road surface viewed from the left camera 100 can be analyzed using the parallax. A plane attribute “road surface” is assigned to the lower region 112 in advance, and when the attribute is determined to be valid from the analysis result, the lower region 112 is determined to be a viewpoint conversion target. Conversely, if it is determined that the attribute is not valid, the lower area 112 is excluded from viewpoint conversion. That is, erroneous viewpoint conversion is suppressed by searching for a corresponding point without performing viewpoint conversion. When an attribute such as “road surface” or “wall” is determined as the plane attribute of the area, the conversion parameter calculation unit 223 calculates the conversion parameter of the area.

<FIG. 4 Corresponding point search unit 300>
The corresponding point search unit 300 searches for a corresponding point between the left image (viewpoint converted image) subjected to viewpoint conversion by the viewpoint conversion unit 200 and the original image of the right image. In addition, when the viewpoint conversion is performed on the left and right images by the viewpoint conversion unit 200, a corresponding point is searched from the left and right images after the viewpoint conversion. As shown in FIG. 4, the corresponding point search unit 300 can function as a feature point extraction unit 310, a feature amount description unit 320, a maximum error setting unit 330, a corresponding point search unit 340, and a reliability calculation unit 350. .

  The feature point extracting unit 310 extracts a feature point from the left image. A feature point is a unique point such as a corner of an object on an image. Note that the feature point may be extracted from one of the left and right images, and the feature point may be extracted from the right image.

  The feature value description unit 320 describes a feature value obtained by quantifying a change in the surrounding luminance of the feature point extracted by the feature point extraction unit 310. When searching for the corresponding point of the right image based on the feature points / features on the left image, the search range of the right image may be set to an area expanded by parallax.

  The maximum error setting unit 330 sets a search range (vertical search range) in the vertical direction of the image in consideration of the maximum vertical error occurrence range of the left and right cameras 100 and 110 before performing the corresponding point search. For example, when obtaining feature points from the left image, the vertical search range is set to the right image. If the parallelization of the left and right images is perfect, the corresponding points of the feature points of the left image may be searched in the horizontal row of the same height in the right image. However, the range of the vertical error differs depending on the temperature characteristics and the assembling accuracy of the components to be used. In such a case, if the vertical maximum error is defined, it is sufficient to perform a corresponding point search for the corresponding point of the feature point extracted from the left image within the range of ± vertical maximum error from the same coordinate in the right image. By limiting the vertical search range and the area in the maximum error setting unit 330 in this way, the number of candidates for corresponding point search can be significantly reduced. The vertical search range is reduced by the maximum error setting unit 330, and the horizontal direction is set by the maximum parallax value of the divided region of the left image. Thereby, the feature point candidates to be searched by the corresponding point search unit 340 can be reduced.

  The corresponding point search unit 340 performs a corresponding point search on the right image while comparing the similarities of the feature amounts calculated for the feature points. Normally, a plurality of candidate corresponding points are found for one feature point. Among them, a candidate point having the highest similarity and having a similarity equal to or more than a predetermined threshold value is set as the corresponding point. In the example of FIG. 14, the left and right images are divided into an upper region and a lower region, and corresponding points are extracted from the image without viewpoint conversion in the upper region (infinity), and the viewpoint is extracted in the lower region (road surface). The corresponding points are extracted from the converted image. Thereby, compared with the conventional case in which the viewpoint conversion is not performed, it is possible to obtain a large number of dense corresponding points from the upper and lower regions.

  The reliability calculation unit 350 determines whether or not the area is an area that can be used for the camera geometry at the subsequent stage based on the similarity of the corresponding points obtained by the corresponding point search unit 340 and the number of corresponding points. Calculate the reliability, which is an index value. If the reliability is low, it is determined that the area cannot be used in the camera geometry. The reliability is calculated for all the regions to determine whether the region can be used for camera geometry. If it is determined that the area cannot be used in a number of areas equal to or greater than a certain threshold value, it is determined that calibration (geometric calibration) cannot be performed on the image acquired this time.

<Figure 5 Camera geometry calibration unit>
The camera geometric calibration unit 400 performs geometric calibration of the left and right cameras 100 and 110 based on the plurality of corresponding points obtained by the corresponding point search unit 300 so that the left and right images are parallel. As shown in FIG. 5, the camera geometric calibration unit 400 includes a corresponding point inverse transformation correction unit 410, a corresponding point aggregation unit 420, a noise corresponding point deletion unit 430, a geometric calibration parameter estimation unit 440, and an availability determination unit. 450, and may function as the geometric calibration reflection unit 460.

  First, the corresponding point inverse transformation correction unit 410 performs a calculation to return the coordinates of the corresponding point obtained by using the viewpoint change or the image deformation to the coordinate system on the original image. In the present embodiment, an inverse transformation parameter is used to return to the coordinate system of the original image. The parameters of the inverse transformation have already been obtained by the inverse transformation parameter calculation unit 230, and the coordinates of the corresponding points of the viewpoint-transformed image (left image) of the left and right images are converted to the coordinates of the original image using the inverse transformation parameters. Perform inverse conversion. FIG. 14 shows an inverse conversion method when the viewpoint conversion is performed on the lower area of the left image. In the example of this figure, the viewpoint is changed by modifying only the lower region of the left image as shown in the interruption image among the three stages. In this state, a corresponding point search is performed in both the upper region and the lower region. After the corresponding point search, in the lower area, inverse transformation (coordinate transformation) is performed to return the coordinates of the corresponding point (feature point) to the coordinates before the viewpoint transformation, and the position of the corresponding point in the image before transformation is calculated. . In other words, a number of corresponding points found based on the image after the viewpoint conversion are inversely transformed into coordinates on the original image and then used for geometric calibration.

  In this way, the corresponding point inverse transformation correction unit 410 inversely transforms the corresponding point coordinates of the area subjected to the viewpoint transformation (deformation) and moves the coordinate on the coordinate system of the original image. Are aggregated as corresponding points in the coordinate system of the original image.

  Next, it is desired to perform camera geometric calibration using the corresponding points collected by the corresponding point aggregating unit 420, but the corresponding points at the stage of aggregation include erroneous corresponding points. Therefore, the noise component of the corresponding point is removed. First, a predetermined number of sets of corresponding points (here, eight sets) corresponding to the obtained corresponding points are scattered throughout the screen, and the correspondence between the left and right images is mathematically determined based on the eight sets of corresponding points. Calculate the fundamental matrix shown in This is repeated a plurality of times while changing the corresponding points serving as the base matrix. As a result, many basic matrices can be obtained, but when the erroneous corresponding points are mixed in the eight corresponding points, the calculated basic matrices deviate from the true values. Conversely, if all eight sets of corresponding points are correct, the underlying matrices are aggregated into similar values. For this reason, the corresponding points that output similar basic matrices are regarded as reliable corresponding points, and the corresponding points that output fundamental matrices with dissimilar outliers are unclear which of the eight corresponding points are erroneous corresponding points. Do not use. However, when it is selected as another eight sets, the base matrix is aggregated into similar values, and is used when it is found that it is not an outlier.

  From the basic matrices other than the outliers obtained in this way, a certain evaluation scale is determined, and the basic matrix with the highest evaluation value is used as the initial value. The evaluation scale is, for example, a random selection of a pair of reliable corresponding points excluding the eight corresponding points not used for the generation of the basic matrix, and how much error the pair of corresponding points causes in the basic matrix. Is used as an evaluation scale. The basic matrix obtained in this manner is first set as an initial value of the basic matrix indicating the correspondence between the left and right cameras 100 and 110, and further, highly accurate geometric calibration is performed using the corresponding points determined to be reliable. Perform parameter optimization.

  Using the basic matrix obtained by the above method as an initial value, an optimization problem that minimizes the distance error between the corresponding point on the image and the estimated point calculated using the basic matrix as a cost function The geometric calibration parameter estimating unit 440 solves the problem. This makes it possible to estimate a basic matrix (geometric calibration parameter) with higher accuracy than the 8-point method.

  Next, the availability determining unit 450 first determines the number of corresponding points obtained from the corresponding point aggregating unit 420 (whether the number of corresponding points exceeds a predetermined number) and the outliers other than the outliers obtained from the noise corresponding point deleting unit 430. External information such as the number of pairs of corresponding points (whether the number of pairs exceeds a predetermined number) and the magnitude of the minimized distance error obtained from the geometric calibration parameter estimating unit 440 (whether the magnitude of the distance error is less than a predetermined value) Using the information, it is determined whether or not the result of the camera geometric calibration by the geometric calibration parameter estimation unit 440 can be used. Furthermore, when the left and right camera images are parallelized using the obtained geometric calibration parameters, there is a corresponding point pair in which a vertical error does not occur in the left and right image coordinates after the parallelization among the determined corresponding point pairs. The availability is determined based on whether the ratio exceeds a predetermined ratio (for example, 95%).

  In the geometric calibration reflection unit 460, when the availability is determined by the availability determination unit 450, parallelization is performed using the parameter base matrix indicating the geometry of the left and right cameras 100 and 110 estimated by the geometric calibration parameter estimation unit 440. Update the affine table of the image transformation for generating the image.

<FIG. 6 Parallax image generation unit 500>
The parallax image generation unit 500 generates a parallax image based on the left and right images captured by the right and left cameras 100 and 110 and the latest affine table updated in real time by the camera geometry correction unit 400. The parallax image generation unit 500 according to the present embodiment functions as a parallelized image generation unit 510, a stereo matching unit 520, and a distance calculation unit 530, as shown in FIG.

  The parallelized image generation unit 510 generates a left and right parallelized image using the affine table for generating a parallelized image updated by the camera geometric calibration unit 400. The stereo matching unit 520 performs stereo matching on the parallelized left and right images to generate a parallax image. The distance calculation unit 530 performs a three-dimensional distance conversion from the parallax image using the base line length of the left and right cameras 100 and 110 and the internal parameters (focal length and cell size) of the camera, thereby obtaining an arbitrary value on the left and right images. Calculate the distance to the object.

<FIG. 15 Processing flowchart of control device 10>
Here, a processing flow executed by the control device 10 when the left image is vertically divided into two as described above will be described. The control device 10 repeats a series of processes shown in FIG. 15 at a predetermined cycle.

  In step S01, first, the control device 10 (viewpoint conversion unit 200) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.

  In step S02, the control device 10 (viewpoint conversion unit 200) determines to divide the left image input in step S04 into upper and lower parts. Note that the right image is not divided.

  In step S03, the control device 10 (viewpoint conversion unit 200) divides the left image into two vertically and sets an area including the vanishing point VP as the upper area 111. The upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be imaged, and is used for the corresponding point search without performing the viewpoint conversion.

  In step S04, the control device 10 (viewpoint conversion unit 200) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as a lower region 112, and proceeds to step S05. Transition. The lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.

  In step S05, the control device 10 (viewpoint conversion unit 200) generates a conversion parameter for performing the viewpoint conversion of the lower region 112, and converts the corresponding point on the lower region 112 after the viewpoint conversion by the inverse conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to the coordinates of. The viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.

  In step S06, the control device 10 (viewpoint conversion unit 200) converts the viewpoint of the lower region 112 of the left image using the conversion parameters generated in step S05.

  In step S07, the control device 10 (viewpoint conversion unit 200) generates a viewpoint conversion image in which the upper region 111 divided in step S03 and the lower region 112 subjected to viewpoint conversion in step S06 are combined. Then, the control device 10 (corresponding point search unit 300) performs a corresponding point search on the viewpoint converted image and the right image input in step S01. That is, a process of searching for a corresponding point in the right image based on the feature points and feature amounts on the image of the upper region 111 that has not been transformed and the lower region 112 that has been transformed is executed. As a result, the first correspondence that is a set of a plurality of corresponding points based on the feature points and the feature amounts is obtained from the lower area 112 of the left image after the viewpoint conversion and the lower area 112 of the right image that has not been subjected to the viewpoint conversion. A point group is extracted, and from the upper region 111 of the left image before the viewpoint conversion and the upper region 111 of the right image without the viewpoint conversion, a second set of a plurality of corresponding points based on the feature points and the feature amounts is obtained. A corresponding point group is extracted.

  In step S08, the control device 10 (camera geometric calibration unit 400) straddles the upper and lower regions 111 and 112 of the plurality of corresponding points (first corresponding point group) found in the lower region 112 in step S07. The remaining corresponding points are excluded and the inverse transformation is performed using the inverse transformation parameter generated in step S05, and the coordinate values of the remaining corresponding points are corresponded to the original image (the left image input in step S01). Returns to the coordinate value when the point was taken.

  In step S09, the control device 10 (camera geometric calibration unit 400) inversely transforms the coordinate values of the plurality of corresponding points (second corresponding point group) on the upper region 111 found in step S07 in step S08. The coordinate values of a plurality of corresponding points (first corresponding point group) on the lower area 112 are collected. At this time, the coordinates of the corresponding point on the left image in the first corresponding point group are the coordinates obtained by inversely converting the coordinates before the viewpoint conversion, and the coordinates of the corresponding point on the right image in the first corresponding point group are the coordinates of the unconverted viewpoint. The original coordinates of the implementation. Further, the coordinates of the second corresponding point group are the coordinates before the viewpoint conversion for both the left and right images. As a result, the coordinate system of all the corresponding points in the upper and lower areas 111 and 112 can be treated uniformly with the coordinate system of the left and right original images.

  In step S10, the control device 10 (camera geometric calibration unit 400) performs noise removal. From the corresponding points aggregated in step S09, eight corresponding point pairs are selected at random so as to form a coordinate system scattered on the image, and a so-called eight corresponding point pair (input corresponding point) is selected based on the selected corresponding point pair (input corresponding point). Calculate the values of the fundamental matrix by the point method. Then, the input corresponding points of the basic matrix that did not become outliers based on the values of the basic matrix are distinguished from the input corresponding points that became outliers by setting a flag so that they can be used in subsequent processing.

  In step S11, the control device 10 (camera geometric calibration unit 400) uses the coordinates of the corresponding point not determined as noise in step S10 to perform geometric calibration parameter estimation. Using the basic matrix obtained by the above method as an initial value, an optimization problem that minimizes the distance error between the corresponding point on the image and the estimated point calculated using the basic matrix as a cost function solve. Thereby, it is possible to calculate a geometric calibration parameter with higher accuracy than the 8-point method.

  In step S12, the control device 10 (camera geometric calibration unit 400) uses information such as whether the magnitude of the distance error calculated in step S11 is less than a predetermined value or whether the number of corresponding points is equal to or more than a predetermined value. It is determined whether the geometric calibration parameters calculated in step S11 can be used. If it is determined that the geometric calibration parameters can be used, the process proceeds to step S13. On the other hand, if it is determined that the affine table cannot be used, the process proceeds to step S14 without updating the affine table.

  In step S13, the control device 10 (camera geometric calibration unit 400) updates the affine table for parallelizing the left and right images used in the previous frame based on the geometric calibration parameters calculated in step S11.

  In step S14, the control device 10 (the parallax image generation unit 500) generates a parallelized image of the left and right images using the stored affine table, performs stereo matching using the generated parallelized image, and generates a parallax image. I do.

  In the in-vehicle environment recognizing device of the present embodiment calibrated as described above, the left and right cameras 100 and 110 (left and right images) perform viewpoint conversion on a region (lower region 112) that is greatly different in appearance, thereby finding from the left and right images. The number of corresponding points to be significantly increased. That is, by deforming the lower area 112 of the left image captured by the camera 100 so as to approach the view from the viewpoint of the right camera 110, a search result of a denser corresponding point having a larger number than before the viewpoint conversion is performed. Can be obtained by the corresponding point search unit 300. The use of the close correspondence points enables highly accurate geometric calibration in the processing by the camera geometric calibration unit 400. By using the paired right and left images accurately parallelized by the geometric calibration, the parallax image generation unit 500 can generate a high-density and high-precision parallax image.

  In the above description, the left and right images are divided into an upper region and a lower region. For the lower region, the viewpoint conversion is performed on the left image, and then the corresponding point is searched. For the upper region, the viewpoint conversion is performed on both the left and right images. Although the corresponding points were searched without any problem, the configuration for dividing the left and right images into a plurality of regions is not essential. For example, a plurality of corresponding points (first corresponding point group) are extracted from the left image after the viewpoint conversion and the right image without the viewpoint conversion, and a plurality of correspondence points are extracted from the left image before the viewpoint conversion and the right image without the viewpoint conversion. The points (second corresponding point group) may be extracted and geometric calibration of the left and right cameras 100 and 110 may be performed.

<Modification of viewpoint conversion>
Another example of viewpoint conversion by the viewpoint conversion unit 200 will be described. In the above description, the method of dividing the camera image into two vertically has been described. However, a method of dividing the camera image into six or a method of dividing the camera image into free areas can be used.

(1) Viewpoint conversion by six divisions In the geometric calibration of the camera, the accuracy is improved when the corresponding points are obtained from the entire image. When corresponding points are obtained densely from a certain fixed point, if the corresponding points are used as they are, the geometric calibration of only the parts where the corresponding points are densely obtained is calculated so as to be small, so the entire image is evenly distributed. When the corresponding point is obtained, a value closer to the original value to be geometrically calibrated is often obtained. However, when a feature point is extracted, a feature amount is described, and a corresponding point is searched for in the entire image, the processing load becomes heavy.

  Therefore, as shown in FIG. 13, the image is divided into six regions, and a method of searching for corresponding points on the left and right images for the six regions is selected. In this method, the control device 10 divides the left image into six rectangular areas, and the six rectangular areas are given a plane attribute that is predicted to appear in each rectangular area while the host vehicle is traveling. It is determined based on the parallax image of the previous frame whether or not the plane attributes given to the six rectangular areas are valid. Of the six rectangular areas whose plane attributes are determined to be valid, the left and right cameras 100 , 110 are determined as conversion target regions, the positions and orientations of the left and right cameras 100 and 110 with respect to the conversion target region are estimated, and the estimated positions and orientations of the left and right cameras 100 and 110 are determined. The conversion target area is converted into an image from the viewpoint of the right camera based on the left image, and a left image after the viewpoint conversion is obtained by combining the remaining area excluding the conversion target area from the six areas and the conversion target area. To. The six rectangular regions are obtained by dividing the left image into two vertically and three horizontally, and each rectangular region is arranged in two rows and three columns. The image is divided into an upper stage and a lower stage, which are referred to as a first region, a second region, and a third region from the left side of the upper stage, and are referred to as a fourth region, a fifth region, and a sixth region from the left side of the lower stage. The planar attribute of the lower three rectangular regions (the fourth to sixth regions) in the six rectangular regions is “road surface”, and two rectangular regions located on the left and right of the upper three rectangular regions in the six rectangular regions. The plane attribute of the (first and third regions) is “wall”, and the plane attribute of the center rectangular region (second region) of the upper three rectangular regions in the six rectangular regions is “infinity”. It is.

This method is effective in extracting feature points, reducing the processing time of description, and narrowing down corresponding point search candidates.
At the time of viewpoint conversion by this method, if the plane attribute predicted to appear in each area while the own vehicle is running is determined, it is easy to select the deformation amount for each of the six areas. For example, as in the example shown in FIG. 13, considering the basics, the lower three regions (regions 4 to 6) perform the viewpoint conversion by the viewpoint conversion assuming the road surface (ground) as in the past. Since the middle area (second area) near the upper infinity (vanishing point) contains only a distant view or the sky, no deformation occurs. The upper left and right areas (the first and third areas) depend on the scenery of the running path, but when traveling in the city, buildings, trees, and the like are "walled" against the running path. There are many things that exist on the left and right. In such a case, conversion is performed assuming walls existing on the left and right of the travel path. The road surface or the wall may be converted assuming a certain fixed value, or only the lateral position and the rotation of the wall are estimated from the parallax image of the previous frame as shown in the middle diagram of FIG. A method that utilizes this may be used. Convert the disparity values in the divided area into distances and perform plane estimation, and determine the amount of outliers and how many percent of disparity points occupy within a certain distance from the final estimated plane. Judgment as to whether or not it may be approximated to a plane may be performed. If it is determined that the plane can be approximated, it is also calculated whether this plane is close to the two camera viewpoints and the difference in appearance due to the change of the viewpoint is large. If the difference in appearance is small, the need for viewpoint conversion is low in the first place. If viewpoint conversion is performed when the difference in appearance is large, the search performance of the corresponding point is greatly improved, so even if there is some error in the viewpoint conversion, it is better than performing the corresponding point search on the original image before conversion. Significantly closer correspondence points can be obtained.

  As shown in each rectangular area in FIG. 13, a plane attribute predicted to appear in each rectangular area during traveling of the own vehicle is added to each area in advance. For example, the upper left and right regions (first and third regions) are “walls”, the lower three regions (4th to 6th regions) are “road surfaces”, and the upper central region (second region) is “infinity”. The plane attribute “far” is attached, and it is determined whether or not the plane estimated from the parallax value of the previous frame is similar to the plane defined by these attributes. If the plane attribute is different from the pre-assigned plane attribute, it is assumed that the plane cannot be approximated well, and the viewpoint conversion is not performed. This makes it possible to avoid erroneous viewpoint conversion. In addition, since the plane attribute is determined in advance, it is relatively easy to remove an outlier that is an unstable element, and the parameters to be estimated are narrowed, so that the stability is enhanced.

  When divided into six rectangular areas as described above, the two areas (first and third areas) located on the left and right of the upper row are equivalent to walls when there are buildings and trees along the traveling path. It is possible to use viewpoint conversion assuming that there is a plane (a region whose plane attribute is “wall”). However, when there are few three-dimensional objects around the road on a rural road, it is better to assign a non-planar attribute, such as "infinity", to the two regions without assigning the attribute of "plane". The number of corresponding points may increase. For this reason, it is possible to understand the tendency of the three-dimensional structure of the scenery appearing in each region from the parallax image of the previous frame, and then determine whether or not to perform the viewpoint conversion and perform the corresponding point search.

(2) Viewpoint conversion by free region division When performing viewpoint conversion utilizing a GPU or the like, a camera image may be pasted on a three-dimensional plane and deformed. To split the camera image. Therefore, as shown in FIG. 16, a region division is performed using a diagonal line as a boundary line of each region. Based on information obtained from the parallax image of the previous frame (for example, a road surface area estimation result and a road edge area estimation result), it is determined whether or not a portion that can be approximated to a plane exists in the image. Then, a region including a portion determined to be a portion that can be approximated to a plane is estimated as a road surface plane region. At this time, it may be determined whether or not the distance from the area to the left and right cameras 100 and 110 is less than a threshold, and the area where the distance is less than the threshold may be set as a road surface area, that is, an area for performing viewpoint conversion. Similarly, it may be determined whether or not the same object is in a region where the left and right cameras 100 and 110 look greatly different.

  In addition, since buildings and trees are standing near the roadside area, area division based on the assumption that there is a wall along the traveling direction of the traveling path is performed by using both the image color and the three-dimensional area. Division may be performed. For each of the divided areas, whether or not it can be approximated to a three-dimensional plane is estimated from the parallax image of the previous frame, as in the case of the six areas. If the plane can be approximated, the viewpoint conversion is performed according to the plane.

  Since the image is divided by a more appropriate and flexible area than the above-described rectangular area, the entire screen can be used relatively effectively. However, if the background of the camera image is complicated, it is difficult to divide the area, and it is better to use a premise that knows what plane is to some extent, such as six areas, for stable determination. High stability.

(3) Perspective conversion of both left and right cameras In the above example, the case where the image of one camera (left camera 100) of the stereo camera is converted into an image from the viewpoint of the other camera (right camera 110) has been described. Images from both cameras (left and right cameras 100 and 110) may be converted to images from another viewpoint (for example, a predetermined viewpoint located between the left and right cameras 100 and 110). In the example of FIG. 17, the left and right images are divided into upper and lower parts. Then, each lower area is viewpoint-converted into an image from the viewpoint of another camera having an optical axis in the middle of the optical axes of the left and right cameras 100 and 110 (that is, the center of the baseline in the case of a stereo camera). .

  When the left and right images are viewpoint-converted in this manner, image quality degradation occurs due to viewpoint conversion in each of the left and right images, so that matching can be performed in the same image quality degradation state, and an improvement in the matching score can be expected. Furthermore, this method has an advantage in a case where the environment recognition apparatus includes three or more cameras and performs triangulation from a pair of multiple cameras, and it is not clear which camera mainly performs three-dimensional reconstruction. It is easy to use for three-dimensional surveying with a camera. As described above, the present invention does not need to be a three-dimensional restoration mainly performed by the right camera as shown in FIG.

-2nd Embodiment Although the viewpoint conversion image was not used for the generation of the parallax image in the parallax image generation unit 500 in the above embodiment, the viewpoint conversion image may be used for the generation of the parallax image. The control device 10 of the present embodiment includes a parallax image generation unit 500A. The other parts are the same as in the first embodiment, and a description thereof will be omitted.

<FIG. 7 Parallax image generation unit>
The parallax image generation unit 500A illustrated in FIG. 7 includes a region-based viewpoint conversion parallelized image generation unit 550, a region-based matching unit 560, a result integration unit 570, and a distance calculation unit 580.

  In a stereo camera with a long base line length, even in stereo matching at the time of generating a parallax image as in the corresponding point search, the appearance of a short-distance subject may differ greatly between the left and right cameras 100 and 110. For this reason, parallax matching may be difficult even in stereo matching. However, if viewpoint conversion is performed also during stereo matching as in the corresponding point search of the first embodiment, parallax of a plane such as a road surface can be obtained densely. Further, as with the corresponding points, the corresponding points can be obtained as appropriate for the distant scenery without deformation. For algorithms that analyze the road surface shape, analyze small irregularities, or mainly observe the road surface such as where the vehicle can travel, the corresponding point method is used. Similarly, it is preferable to convert the viewpoint of the left image and perform stereo matching of the left and right images.

  Therefore, in the present embodiment, first, the region-specific viewpoint-converted parallelized image generation unit 550 divides the left image into two regions (upper region and lower region) in the far and near regions as in the first embodiment. In the upper region, the viewpoint conversion is not performed, and in the lower region, the viewpoint conversion to the right camera viewpoint is performed simultaneously with the parallelization by the affine table. Note that viewpoint conversion may be performed by dividing an image after parallelization. As the conversion parameters at the time of viewpoint conversion and the affine table at the time of parallelization (at the time of geometric calibration), those calculated by the viewpoint conversion unit 200 and the camera geometric calibration unit 400 are used as in the first embodiment. Next, the region-specific matching unit 560 calculates parallax values individually for the two regions generated by the image generation unit 550, and generates parallax images individually. In the result integrating unit 570, the parallax values (parallax images) belonging to the lower region among the parallax values calculated by the region-specific matching unit 560 are corrected according to the viewpoint conversion, so that the upper region and the lower region are corrected. Correction is performed to match the meaning of the parallax values, and then the parallax values (parallax images) of the upper region and the lower region are integrated. Further, the distance calculation unit 580 calculates the distance from the parallax images integrated by the result integration unit 570, using the information on the base line length of the left and right cameras 100 and 110 and the information on the internal parameters of the left and right cameras 100 and 110. As a result, a parallax image based on a greater number of densely corresponding points can be obtained in a lower region (a road surface region at a short distance from the camera), which can greatly differ in the appearance of the left and right images, so that the accuracy of the parallax image is improved. improves.

  However, if there is a pedestrian at the edge of the lower area even if the majority is on the road, stereo matching using the viewpoint conversion image can obtain a dense parallax image for objects that are almost near the road. On the other hand, if the viewpoint is not changed in the pedestrian part, the parallax image may be obtained clearly. Therefore, as another embodiment, for the lower region, the left and right images (that is, the pair of the right image and the left image before the viewpoint conversion (first pair)) are matched without the viewpoint conversion after the region division and the parallelization. Both the result and the result of matching the left and right images (that is, the pair of the right image and the left image after the viewpoint conversion (second pair)) with the viewpoint conversion after the region division and parallelization are generated, and two matching results are generated. Among them, the parallax value and the parallax image may be generated by using the one with the higher matching score indicating how similar the left and right images are. At this time, it is preferable that the matching result with the viewpoint conversion is returned to the inversely converted state, and that the matching score indicating how similar the left and right images are similar can be referred to from each parallax value in both cases. According to this method, for example, a three-dimensional object such as a pedestrian is present in the lower region, and the stereo matching using the image after the viewpoint conversion can prevent the matching score from being reduced. Can be improved.

<FIG. 18 Flowchart of parallax image generation processing by control device 10>
Here, a processing flow executed by the control device 10 (the parallax image generation unit 500A) when the left image is vertically divided into two when generating the parallax image as described above will be described. The control device 10 repeats a series of processes illustrated in FIG. 18 based on the input of the parallax image request command. It should be noted that the processing of searching for corresponding points for calibration and estimating / updating geometric calibration parameters performed in steps DS02 to DS04 in the drawing may use any method, and the first processing shown in FIG. The method is not limited to the method of the embodiment, and a known method may be used. Here, an example of a method of applying the viewpoint transformation of the first embodiment to the generation of a parallax image, assuming that the calibration for parallelization has already been performed on the left and right images, will be described.

  In step DS01, first, the control device 10 (the parallax image generation unit 500A) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.

  In step DS02, the control device 10 (corresponding point search unit 300) searches for corresponding points in the left and right images.

  In step DS03, the control device 10 (camera geometric calibration unit 400) estimates geometric calibration parameters for performing parallelization of the left and right images.

  In step DS04, the control device 10 (camera geometric calibration unit 400) updates the geometric calibration parameters used when creating the parallelized image of the left and right images with the geometric calibration parameters calculated in step D02. . At this time, the estimated values of the relative positions and postures of the left and right cameras 100 and 110 and the parameters of the position and posture of the road surface and the stereo camera may be updated.

  In step DS05, the control device 10 (the parallax image generation unit 500A) determines to divide the left image input in step DS01 into upper and lower parts. Note that the right image is not divided.

  In step DS06, the control device 10 (the parallax image generation unit 500A) divides the left image into upper and lower portions, and sets an area including the vanishing point VP as the upper area 111. The upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be captured, and is used for stereo matching (step DS07) without performing viewpoint conversion. However, especially when the parallax of an object other than the road surface is regarded as important in the lower region 112 or when there is room for processing, the entire left image is set to the upper region, and the right image is set without performing viewpoint conversion. Stereo matching with the image may be performed (step DS07).

  In step DS08, the control device 10 (the parallax image generation unit 500A) sets an area obtained by removing the upper area 111 from the left image (an area located below the upper area 111) as the lower area 112. The lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.

  Then, the control device 10 (the parallax image generation unit 500A) generates a conversion parameter for performing the viewpoint conversion of the lower region 112 and performs the inverse conversion of the corresponding point on the lower region 112 after the viewpoint conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to coordinates. The viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.

  Further, the control device 10 (the parallax image generation unit 500A) performs viewpoint conversion of the lower region 112 of the left image using the generated conversion parameters.

  Note that the viewpoint conversion image generated in step DS08 may generate a viewpoint conversion image obtained by converting the viewpoint of the left image into the viewpoint of the right camera 110. Assuming that the cameras are located at the center positions of the left and right cameras 100 and 110, image conversion may be performed as if the images of the left and right cameras were brought to the position of the center of gravity of the stereo camera.

  In step DS07, the control device 10 (the parallax image generation unit 500A) performs parallax calculation and generates a parallax image by performing stereo matching on the upper region 111 in step DS06 and the corresponding right image region. At the same time, in step DS09, the control device 10 (disparity image generation unit 500A) calculates the disparity value by performing stereo matching on the lower region 112 converted in viewpoint in step DS08 and the corresponding right image region. Generate an image.

  In step DS10, the control device 10 (the parallax image generation unit 500A) inversely transforms the parallax value based on the viewpoint conversion image generated in step DS10 by using an inverse conversion parameter, thereby obtaining the parallax value of the parallax value. Perform conversion.

  In step DS11, the control device 10 (the parallax image generation unit 500A) determines that if two images (left and right images) on the parallelized image coordinates before the viewpoint conversion have a superimposed region, the lower region before the viewpoint conversion is performed. The matching score between the corresponding portion of the right image and the matching score of the corresponding portion of the right image is compared with the matching score of the lower region 112 and the corresponding portion of the right image after the viewpoint conversion, and the disparity value with the higher matching score is used as the disparity value of the lower region. select. As a result, the comparison of score matching acts to preferentially use the disparity when the viewpoint is changed on the road surface and preferentially use the disparity when the viewpoint is not changed when a three-dimensional object is present. Image accuracy is improved.

  In step DS12, the control device 10 (the parallax image generation unit 500A) combines the parallax image of the upper region 111 generated in step DS07 and the parallax image of the lower region 112 selected through the comparison in step DS11 into one sheet. By synthesizing a parallax image (combined parallax image), it is possible to generate a parallax image that is denser and has less noise than before.

  Note that, in the present embodiment, the case where the left image is vertically divided is described. However, if the left image and the right image are regions that are significantly different in appearance and include a portion that can be approximated to a plane, the disparity value obtained by the viewpoint conversion is used. Can be expected to improve the accuracy of That is, stereo matching may be performed by performing viewpoint conversion on an area different from the lower area described above, and this type of area includes an area on which viewpoint conversion is performed in the first embodiment.

<Others>
In the flowchart of FIG. 15, steps S10 and S12 can be omitted.

  The present invention is not limited to the above embodiments, and includes various modifications without departing from the gist thereof. For example, the present invention is not limited to those having all the configurations described in the above embodiments, but also includes those in which some of the configurations are deleted. Further, a part of the configuration according to one embodiment can be added to or replaced by the configuration according to another embodiment.

  In addition, the components of the control device 10 and the functions and execution processes of the components are partially or wholly realized by hardware (for example, a logic for executing each function is designed by an integrated circuit). You may. The configuration of the control device 10 may be a program (software) that is read and executed by an arithmetic processing device (for example, a CPU) to realize each function of the configuration of the device. Information related to the program can be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), a recording medium (magnetic disk, optical disk, etc.), and the like.

  Further, in the description of each of the above embodiments, the control lines and the information lines that are understood to be necessary for the description of the embodiment are shown, but all the control lines and the information lines related to the product are not necessarily used. It is not necessarily shown. In fact, it can be considered that almost all components are interconnected.

  100 left camera, 110 right camera, 111 upper region, 112 lower region, 200 viewpoint conversion unit, 210 region setting unit, 220 conversion parameter generation unit, 221 parallax analysis unit, 222 viewpoint conversion Attribute determination unit, 223: conversion parameter calculation unit, 230: inverse conversion parameter calculation unit, 240: viewpoint conversion image generation unit, 300: corresponding point search unit, 310: feature point extraction unit, 320: feature amount description unit, 330 ... Maximum error setting unit, 340: Corresponding point searching unit, 350: Reliability calculating unit, 400: Camera geometric calibration unit, 410: Corresponding point reverse conversion correcting unit, 420: Corresponding point aggregating unit, 430: Noise corresponding point deleting unit, 440: geometric calibration parameter estimating unit, 450: availability determining unit, 460: geometric calibration reflecting unit, 500: parallax image generating unit, 510: parallelized image generating unit, 520: stereo Etching unit, 530 ... distance calculator

Claims (12)

A first camera and a second camera,
The first image and the second image are converted into an image from a common viewpoint by deforming at least one of a first image captured by the first camera and a second image captured by the second camera. After performing the viewpoint conversion, a plurality of corresponding points are extracted, and the coordinates of the plurality of corresponding points in the first image and the second image before the viewpoint conversion are used for the first camera and the second camera. A vehicle environment recognition device comprising: a control device for performing geometric calibration. The in-vehicle environment recognition device according to claim 1,
In the viewpoint conversion, the control device may assume that at least a part of a portion captured in the first image and the second image in a redundant manner is a plane, and the first camera and the second camera with respect to the plane may be used. Viewpoint conversion for calculating the position and orientation of a camera and converting the first image into an image from the viewpoint of the second camera based on the calculated positions and orientations of the first camera and the second camera with respect to the plane; An in-vehicle environment recognition device, characterized in that: The in-vehicle environment recognition device according to claim 2,
The control device extracts a first corresponding point group, which is a set of a plurality of corresponding points, from the first image after the viewpoint conversion and the second image on which the viewpoint conversion has not been performed. A second corresponding point group, which is a set of a plurality of corresponding points, is extracted from the first image and the second image on which the viewpoint conversion has not been performed, and coordinates of the first corresponding point group are coordinates before the viewpoint conversion. The geometric calibration of the first camera and the second camera is performed using the coordinates converted into the coordinates and the coordinates of the second corresponding point group. The in-vehicle environment recognition device according to claim 1,
In the viewpoint change, the control device performs affine transformation while changing parameters on the first image to generate a plurality of transformed images, and extracts corresponding points between each of the plurality of transformed images and the second image. And the converted image having the largest number of the corresponding points among the plurality of converted images and equal to or greater than a predetermined threshold value is an image obtained by converting the first image into an image from the viewpoint of the second camera. An in-vehicle environment recognition device, which is a viewpoint conversion. The in-vehicle environment recognition device according to claim 1,
The viewpoint conversion includes:
The control device,
The first image is vertically divided into two regions, an upper region including a vanishing point or in contact with the vanishing point, and a lower region located below the upper region,
Assuming that at least a portion of the lower region is planar,
Estimating the position and orientation of the first camera and the second camera with respect to the plane,
Converting the lower area to an image from the viewpoint of the second camera based on the position and orientation of the first camera and the second camera with respect to the estimated plane,
An in-vehicle environment recognizing device, wherein viewpoint conversion is performed by combining the lower region and the upper region after the conversion with the first image after the viewpoint conversion. The in-vehicle environment recognition device according to claim 1,
The viewpoint conversion includes:
The control device,
Dividing the first image into a plurality of regions;
Determine whether there is a portion that can be approximated to a plane in the plurality of regions based on the parallax image of the previous frame,
Among the plurality of regions, a region having a portion that can be approximated to the plane exists, and a region whose distance from the first camera and the second camera is smaller than a predetermined threshold is determined as a conversion target region.
Estimating the position and orientation of the first camera and the second camera with respect to the conversion target area,
Converting the conversion target area into an image from the viewpoint of the second camera based on the estimated position and orientation of the first camera and the second camera;
An in-vehicle environment recognizing device, wherein the viewpoint conversion is a first image after a viewpoint conversion is performed by combining a remaining region excluding the conversion target region from the plurality of regions and the conversion target region. The in-vehicle environment recognition device according to claim 1,
The viewpoint conversion includes:
The control device,
Dividing the first image into a plurality of rectangular areas;
The plurality of rectangular areas are provided with an attribute of a plane that is predicted to appear in each rectangular area during traveling of the own vehicle,
Determine whether the attribute of the plane given to the plurality of rectangular areas is appropriate based on the parallax image of the previous frame,
In the plurality of rectangular regions, among the rectangular regions in which the attribute of the plane is determined to be appropriate, a rectangular region whose distance from the first camera and the second camera is smaller than a predetermined threshold is determined as a conversion target region,
Estimating the position and orientation of the first camera and the second camera with respect to the conversion target area,
Converting the conversion target area into an image from the viewpoint of the second camera based on the estimated position and orientation of the first camera and the second camera;
An in-vehicle environment recognizing device, wherein the viewpoint conversion is a first image after a viewpoint conversion is performed by combining a remaining region excluding the conversion target region from the plurality of regions and the conversion target region. The in-vehicle environment recognition device according to claim 7,
The plurality of rectangular areas are six rectangular areas obtained by dividing the first image into two vertically and three horizontally.
The attribute of the plane of the lower three rectangular areas in the six rectangular areas is a road surface,
The plane attribute of the left and right two rectangular areas of the upper three rectangular areas in the six rectangular areas is a wall,
An on-vehicle environment recognizing device, wherein the attribute of the plane of the central rectangular area among the upper three rectangular areas in the six rectangular areas is infinity. The in-vehicle environment recognition device according to claim 1,
The control device performs a viewpoint conversion for converting the first image and the second image into an image from a common viewpoint, obtains a parallax value by performing stereo matching, and performs the viewpoint conversion on the parallax value. An in-vehicle environment recognizing device that generates a parallax image by performing a corresponding correction. The in-vehicle environment recognition device according to claim 5,
The control device obtains a parallax value by performing stereo matching on the second image and the first image after the viewpoint conversion, and among the parallax values, those belonging to the lower region perform correction according to the viewpoint conversion. An in-vehicle environment recognizing device that generates a parallax image by performing a parallax image processing. The in-vehicle environment recognition device according to claim 10,
The control device performs stereo matching on two pairs of a second pair including the second image and the first image before the viewpoint conversion and a second pair including the second image and the first image after the viewpoint conversion. Wherein the parallax image is generated using the higher matching score of the stereo matching among the two pairs. The in-vehicle environment recognition device according to claim 1,
The viewpoint conversion includes:
The control device transforms the first image to convert the image to an image from a predetermined viewpoint located between the first camera and the second camera, and transforms the second image to the predetermined image. And a viewpoint conversion for converting a viewpoint into an image from the viewpoint.

Images