WO2020017334A1 - 車載環境認識装置 - Google Patents

車載環境認識装置 Download PDF

Info

Publication number
WO2020017334A1
WO2020017334A1 PCT/JP2019/026561 JP2019026561W WO2020017334A1 WO 2020017334 A1 WO2020017334 A1 WO 2020017334A1 JP 2019026561 W JP2019026561 W JP 2019026561W WO 2020017334 A1 WO2020017334 A1 WO 2020017334A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
camera
viewpoint
conversion
viewpoint conversion
Prior art date
Application number
PCT/JP2019/026561
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
雅幸 竹村
彰二 村松
Original Assignee
日立オートモティブシステムズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立オートモティブシステムズ株式会社 filed Critical 日立オートモティブシステムズ株式会社
Priority to CN201980046818.8A priority Critical patent/CN112424565B/zh
Publication of WO2020017334A1 publication Critical patent/WO2020017334A1/ja

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the present invention relates to an in-vehicle environment recognition device that recognizes a surrounding environment of a vehicle by a camera installed in the vehicle.
  • a method of estimating the position and orientation between two cameras installed in a vehicle and performing geometric calibration so that the images of the two cameras have a parallel positional relationship is generally used as information for obtaining the geometric relationship between the positions and postures of the two cameras.
  • the method of acquiring the corresponding point is to extract a unique point such as a corner (corner) on the image which is called a feature point from the left and right images, calculate a feature amount from a luminance change around the feature point, and calculate the feature amount.
  • Feature points having similar amounts are searched for on the left and right images, and feature points found are set as corresponding points, and geometric calibration is performed based on the coordinates of the corresponding points in the left and right images.
  • Patent Literature 1 calculates a distance to an object and estimates a three-dimensional position of the object based on outputs of left and right cameras arranged such that their fields of view overlap.
  • a calibration device for an in-vehicle stereo camera including an object recognition and feature point collection unit for recognizing and acquiring feature point coordinates of an object, and an inter-camera parameter estimation unit for obtaining an inter-camera parameter based on the feature point coordinates is disclosed. Have been.
  • the scenery or object between the two cameras is close to the camera or when the two cameras are far apart, the same object is observed from a greatly different viewpoint, and the two images are used. Of the same object may be greatly different.
  • the feature amount calculated based on the luminance change around the point between the two cameras Likely to be different. That is, since the feature values of the two feature points are calculated as different values, no corresponding point is found, an incorrect corresponding point (erroneous corresponding point) is found, or even if a corresponding point is found, the number is very small. And the like. The smaller the number of corresponding points or the more erroneous corresponding points, the lower the accuracy of the geometric calibration of the camera. Further, there may be a case where the position and orientation between the two cameras cannot be estimated by the convergence calculation.
  • An object of the present invention is to provide an in-vehicle environment recognizing device that can easily perform a geometric calibration of a camera even when an image captured by a plurality of cameras includes the same object having a significantly different appearance.
  • the present application includes a plurality of means for solving the above-mentioned problems.
  • a first camera and a second camera a first image captured by the first camera, and an image captured by the second camera are provided.
  • viewpoint conversion for converting the first image and the second image to an image from a common viewpoint by deforming at least one of the second images
  • a plurality of corresponding points are extracted, and the plurality of corresponding points are extracted before the viewpoint conversion.
  • dense corresponding points are extracted on an image by performing viewpoint conversion in a common visual field region of a plurality of cameras installed in a vehicle, and geometric calibration is performed based on the corresponding points.
  • viewpoint conversion in a common visual field region of a plurality of cameras installed in a vehicle
  • geometric calibration is performed based on the corresponding points.
  • the position and orientation between the cameras can be estimated with high accuracy, and highly accurate parallelization of the two cameras can be realized.
  • stereo matching in a state where high-precision parallelization is performed, generation of a high-density parallax image is realized, and high-precision distance restoration is enabled from the parallax.
  • FIG. 1 is a configuration diagram of an in-vehicle environment recognition device according to a first embodiment.
  • FIG. 3 is a functional block diagram of a viewpoint conversion unit.
  • FIG. 3 is a functional block diagram of a conversion parameter generation unit.
  • FIG. 3 is a functional block diagram of a corresponding point search unit.
  • FIG. 3 is a functional block diagram of a camera geometric calibration unit.
  • FIG. 3 is a functional block diagram of a parallax image generation unit (first embodiment).
  • FIG. 9 is a functional block diagram of a parallax image generation unit (second embodiment). Description of the process of calibrating camera geometry from corresponding points. Explanation of the problem of acquiring corresponding points. Explanation of solution by viewpoint conversion. An example of viewpoint conversion of upper and lower area division.
  • viewpoint conversion by shearing is an example of viewpoint conversion in six regions.
  • 3 is a processing flowchart of a control device according to the first embodiment.
  • 4 is an example of viewpoint conversion for free area division.
  • 9 is a flowchart of a parallax image generation process performed by the control device according to the second embodiment.
  • FIG. 1 shows a configuration diagram of an in-vehicle environment recognition device 1 according to the present embodiment.
  • the in-vehicle environment recognizing device 1 includes a left camera (first camera) 100 and a right camera (second camera) 110 arranged at intervals in the left and right directions in the horizontal direction, and imaging output from the two cameras 100 and 110.
  • a process of performing geometric calibration of the two cameras 100 and 110 based on an image (an image obtained by the left camera 100 may be referred to as a first image and an image obtained by the right camera 110 may be referred to as a second image);
  • a control device (computer) 10 that executes processing for creating a parallax image by performing stereo matching on two images captured by the two cameras 100 and 110 at the same timing is provided.
  • the control device (computer) 10 includes an arithmetic processing device (eg, CPU) not shown, a storage device (eg, memory, hard disk, flash memory) for storing a program executed by the arithmetic processing device, and internal devices. And a communication device for performing communication with external devices.
  • the control device 10 functions as the viewpoint conversion unit 200, the corresponding point search unit 300, the camera geometric calibration unit 400, and the parallax image generation unit 500 by executing a program stored in the storage device. Other functions can be implemented by adding a program.
  • a stereo camera consisting of two cameras on the left and right recognizes the environment around the vehicle using a common viewing area. After attaching the left and right cameras to the predetermined position of the support with high accuracy, the camera factory estimates the position and orientation of the left and right cameras while capturing the calibration chart with the left and right cameras, and uses the results to The parameters are corrected so that the captured images are parallel to each other. If stereo matching is performed in this state, it is possible to obtain a high-density parallax and a parallax image, and it is possible to measure a distance with high accuracy from the parallax image.
  • high-precision camera manufacturing and parts and structures that suppress deformation of the camera due to temperature change, shock, vibration, aging, and the like are expensive.
  • the present inventors can easily correct the deformation of the camera due to the positional shift of the camera, the temperature, the aging, etc. in order to suppress the cost of this kind, and realize the high-precision calibration during the traveling.
  • the in-vehicle environment recognition device 1 of the present embodiment has been invented.
  • feature points (unique points such as corners (corners) of an object on the image) are extracted from the images of the left and right cameras 100 and 110, and the feature points are extracted.
  • a feature amount is calculated from a change in surrounding brightness, a feature point having a similar feature amount is searched for on the left and right images, and a set of feature points having similar feature amounts on the left and right images is set as a set of corresponding points.
  • the image of FIG. 8 is an image in which a plurality of corresponding points on the searched left and right images are connected to each other with a line, and geometric calibration of the left and right cameras 100 and 110 is performed based on the plurality of corresponding points.
  • the corresponding point search unit 300 extracts the above-described feature point from the images of the left and right cameras 100 and 110, calculates a feature amount from a luminance change around the feature point, and calculates the feature amount on the left and right images. A feature point (corresponding point) having a similar feature amount is searched.
  • the camera geometric calibration unit 400 calculates geometric calibration parameters for making the left and right images parallel based on the plurality of corresponding points obtained by the corresponding point search unit 300.
  • the image obtained by geometrically calibrating the images of the left and right cameras 100 and 110 is an image having a horizontal positional relationship and no lens distortion at all. This makes it possible to prepare left and right images that are easily geometrically matched.
  • the parallax image generation unit 500 performs stereo matching on two images (parallelized images) captured at the same timing and corrected by the geometric calibration parameters of the camera geometric calibration unit 400, and performs two-dimensional matching on the two images.
  • the distance information (parallax information) indicating the deviation of the position where the same object (the same pattern) is photographed is calculated by a known method to generate a parallax image (distance image).
  • the parallax image generation unit 500 uses left and right images captured by the left camera 100 and the right camera 110. In the present embodiment, since the stereo matching is performed based on the right image (second image) from the right camera 110, the sensitivity, geometry, and the like are basically matched to the right reference.
  • the parallax image generation unit 500 receives the images of the left and right cameras to which the geometric and sensitivity corrections have been applied, performs stereo matching, generates a parallax image, and finally performs noise removal to remove the noise-reduced parallax. An image is obtained.
  • the processing performed by the viewpoint conversion unit 200 is the most characteristic part.
  • the viewpoint conversion unit 200 deforms at least one of the left image (first image) captured by the left camera 100 and the right image (second image) captured by the right camera 110, thereby forming the left and right images (first image and first image).
  • Viewpoint conversion for converting the second image) into an image from a common viewpoint This makes it easier for the corresponding point search unit 300 to find corresponding points from the left and right images.
  • the details of the image deformation method (viewpoint conversion method) will be described later.
  • the image (the left image, the first image) captured by the left camera 100 is made to approach the appearance from the viewpoint of the right camera 110.
  • the whole or a part of the region on the image has a larger number than before the viewpoint transformation.
  • the search result of the dense corresponding point can be obtained by the corresponding point search unit 300.
  • high-precision geometric calibration is realized in the processing of the camera geometric calibration unit 400 performed in the subsequent stage.
  • the parallax image generation unit 500 can generate a high-density and high-precision parallax image.
  • the viewpoint (position on the image) of four points on the road surface in the left image is matched with or similar to the right image. After conversion, search for corresponding points.
  • the corresponding point could hardly be searched for from the road surface, but the corresponding points were densely obtained from the road surface after the deformation as shown in the lowermost part of FIG. Can be confirmed.
  • the viewpoint conversion unit 200 includes an area setting unit 210 that sets an area where at least one of the left and right images is to be subjected to viewpoint conversion and an area that is not to be subjected to viewpoint conversion.
  • a transformation parameter generation unit 220 that generates a matrix or a parameter of a function necessary for transforming the left and right images into an image from a common viewpoint by transforming, and a parameter (conversion parameter) generated by the conversion parameter generation unit 220
  • a viewpoint conversion image generation unit 240 that generates an image (viewpoint conversion image) in which at least one of the left and right images is viewpoint-converted by using a matrix or a function included therein
  • An inverse transformation parameter calculation unit 230 that generates parameters (inverse transformation parameters) of matrices and functions is provided.
  • the viewpoint converted image generated by the viewpoint converted image generation unit 240 is used in the corresponding point search in the corresponding point search unit 300. Further, the inverse transformation parameter calculated by the inverse transformation parameter calculation unit 230 is used by the corresponding point inverse transformation correction unit 410 of the camera geometric calibration unit 400.
  • the region setting unit 210 may set all the regions in the camera image as regions where viewpoint conversion is performed or regions where viewpoint conversion is not performed.
  • a common viewpoint to which the left and right cameras 100 and 110 are converted there is a viewpoint of either one of the left and right cameras 100 and 110 and an arbitrary viewpoint different from any of the left and right cameras 100 and 110.
  • a predetermined viewpoint for example, the middle point between the left and right cameras 100 and 110 located between the left camera 100 and the right camera 110 can be set as a common viewpoint to perform viewpoint conversion.
  • viewpoint conversions a part (overlap area) where the viewpoint conversion unit 200 of the control device 10 is overlapped and captured in the left and right images (the first image and the second image). Is assumed to be a plane, the positions and orientations of the left camera 100 and the right camera 110 with respect to the plane are geometrically calculated based on the left and right images, and the left camera 100 and the right with respect to the calculated plane are calculated. Some cameras convert a left image (first image) into an image from the viewpoint of the right camera 110 based on the position and orientation of the camera 110.
  • this viewpoint conversion deforms the entire left image by focusing only on the road surface which is only a part of the left image
  • the viewpoint conversion is performed by focusing on the shape of a three-dimensional object that does not exist on the road surface (particularly, the building shown in FIG. 10). It can be confirmed that the image is greatly inclined after the conversion.
  • the appearance (shape) of the building before and after the deformation is more similar on the left and right images, and the original image (the original image) is denser at a distance from the cameras 100 and 110. It can be understood that a corresponding point can be obtained.
  • the left image is divided into two types of regions, a region where viewpoint conversion is performed and a region where viewpoint conversion is not performed.
  • the search is performed for the corresponding points by division.
  • the former area in which the viewpoint conversion is performed can be said to be an area in which the number of corresponding points increases when the viewpoint conversion is performed.
  • the latter area in which viewpoint conversion is not performed can be said to be an area in which dense corresponding points can be obtained even in the original image, and corresponds to, for example, an area far from the camera.
  • the left image is vertically divided into two by a horizontal boundary line based on a vanishing point VP in the image, and the upper region (upper region) of the two regions is divided.
  • the region (region) 111 is set as a region in which viewpoint conversion is not performed
  • the lower region (lower region) 112 is set as a region in which viewpoint conversion is performed.
  • the viewpoint conversion unit 200 of the control device 10 includes an upper region (distant view) 111 including a vanishing point VP or having a boundary in contact with the vanishing point VP, and a lower region (below the upper region 111).
  • the left image (first image) is divided vertically into two areas of two road areas 112, and it is assumed that at least a part of the lower area 112 is a plane, and the positions and postures of the left and right cameras 100 and 110 with respect to the plane. Is calculated, and the lower area 112 is converted into an image from the viewpoint of the right camera 110 based on the calculated position and orientation of the left and right cameras 100 and 110 with respect to the plane. (No viewpoint conversion) A combination of 111 and the left image (first image) after viewpoint conversion.
  • This viewpoint conversion is one of the simplest and most effective ones in camera geometric calibration of an in-vehicle camera (in-vehicle environment recognition device).
  • this method uses an image after viewpoint conversion (viewpoint conversion image) with hardware such as an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and a GPU (Graphics Processing Unit). Assuming, conversion is simplified with the restriction that the image is divided only into rectangles. A distant road surface can also be deformed on the assumption that it is flat, but if it is far away, the amount of deformation of the road surface due to viewpoint conversion is small.
  • FPGA Field-Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • GPU Graphics Processing Unit
  • the upper region 111 including the vanishing point VP is set in the region setting unit 210 as a region where the viewpoint is not changed.
  • the lower region 112 is a region where the road surface is imaged, the lower region 112 is set by the region setting unit 210 as a region to be changed in viewpoint.
  • the viewpoint conversion unit 200 performs the viewpoint conversion of the lower region 112 using the conversion parameters generated by the conversion parameter generation unit 220.
  • a simple viewpoint conversion method there is a method in which conversion parameters are generated by the conversion parameter generation unit 220 using design values of the positions and postures of the cameras 100 and 110 mounted on the vehicle.
  • the corrected values may be used. If the posture of the road surface is estimated at the same time as the position and posture of the camera based on the distance information of at least three points on the road surface, the posture information of the road surface may be used.
  • the reason that the design values can be used as the position and orientation of the left and right cameras is that it is important that the left and right images are more similar than before the viewpoint conversion, and the two do not need to be mathematically completely identical. Because. This is because the purpose of the viewpoint conversion is to obtain corresponding points densely in the image after the viewpoint conversion, and it suffices to generate an image similar to the degree that the corresponding points can be obtained. Further, the coordinates of the corresponding point after the viewpoint conversion by the conversion parameter are returned to the coordinates before the viewpoint conversion by the inverse conversion parameter, and are used for the geometric calibration of the camera. Therefore, a high-precision camera position / posture is not always necessary.
  • the viewpoint conversion may be mathematically performed based on the relative positions and postures of the cameras 100 and 110 and the road surface.
  • the main purpose of the viewpoint conversion is not accurate conversion. It is to obtain corresponding points of the left and right images densely. For this reason, even if the conversion is not mathematically accurate, it can be replaced by a method that has a certain effect.
  • the conversion based on the mathematical operation in hardware may be omitted.
  • the conversion parameter generation unit 220 sets the center of the vanishing point VR as the center.
  • the viewpoint transformation may be performed by transforming the image by affine transformation including the rotation of the image and the shear deformation of the road surface.
  • the viewpoint conversion unit 200 of the control device 10 changes the parameters (matrix parameters for affine transformation) to the left image (first image) from the left camera 100 while changing the affine. Conversion is performed to generate a plurality of converted images, and corresponding points of each of the plurality of converted images and the right image (second image) are extracted. There is a process of using a large number of converted images that are equal to or more than a predetermined threshold (reference value) in a final viewpoint converted image to be used in subsequent processing (that is, an image obtained by converting a left image into an image from the viewpoint of the right camera 110).
  • a predetermined threshold reference value
  • affine transformation is performed for all combinations including at least one of scaling, rotation, translation, and shearing of the image while gradually changing parameters.
  • converted images of substantially all patterns and a conversion matrix used for the converted images can be generated.
  • the number of converted images can be adjusted at intervals at which parameters are changed.
  • a search for corresponding points with the right image is performed for all of the generated converted images, and the number of corresponding points obtained is compared, thereby transforming the left image into a shape most similar to the right image. You can get the image and the transformation matrix.
  • the parameters of the transformation matrix thus obtained can be used as transformation parameters.
  • a method of viewpoint transformation using shear deformation as affine transformation will be described with reference to FIG.
  • a corresponding point of each converted image with the right image is searched. It is determined that the shear amount of the converted image in which the number of corresponding points is the largest and which is larger than the reference (predetermined threshold) is used as the conversion parameter.
  • the viewpoint conversion image generation unit 240 When the viewpoint conversion of the lower region 112 is completed, the viewpoint conversion image generation unit 240 combines the lower region 112 after the viewpoint conversion and the upper region 111 that has not been subjected to the viewpoint conversion into a viewpoint conversion image (the left side after the viewpoint conversion). Image).
  • the viewpoint conversion image generated by the viewpoint conversion unit 200 is compared with the right image by the corresponding point search unit 300 to extract a corresponding point.
  • the coordinates of the extracted corresponding points are returned to the coordinates in the original image by the inverse transformation by the camera geometric calibration unit 400 as shown in FIG. 14, and are used for camera geometric calibration.
  • the viewpoint conversion unit 200 performs processing from generation of a viewpoint conversion parameter to generation of a viewpoint conversion image and calculation of an inverse conversion parameter.
  • the inverse transformation parameter is used in the camera geometric calibration unit 400 that integrates and uses feature points.
  • the conversion parameter generation unit 220 includes a parallax analysis unit 221, an attribute determination unit 222, and a conversion parameter calculation unit 223.
  • the parallax analysis unit 221 acquires the parallax of the region set as the region for performing the viewpoint conversion by the region setting unit 210 from the parallax image of the previous frame (for example, one frame before) generated by the parallax image generation unit 500, By analyzing the parallax, it is determined whether or not a portion that can be approximated to a plane exists in the region.
  • the attribute determining unit 222 determines the plane attribute of the area based on the analysis result of the parallax analyzing unit 221.
  • the plane attributes include “plane” indicating that there is a portion that can be approximated to a plane in the area, and “non-plane” indicating that there is no part that can be approximated to the plane in the area.
  • the former “plane” attribute further includes “road surface (ground)” and “wall surface (wall)” as attributes indicating the type of the area.
  • the latter attribute of “non-planar” includes “infinity” as an attribute indicating the type of the area.
  • each region may be given a plane attribute in advance.
  • the attribute determination unit 222 determines whether the plane attribute given in advance is appropriate based on the analysis result of the parallax analysis unit 221, and the distance from the left and right cameras 100 and 110 in the area determined to be appropriate is determined.
  • the plane attribute of the area smaller than the predetermined threshold is determined to be “plane”, and the area is determined as the target area of the viewpoint conversion.
  • the conversion parameter calculation unit 223 calculates a conversion parameter for performing viewpoint conversion on an area whose plane attribute is determined to be a plane by the attribute determination unit 222.
  • the conversion parameter can be calculated by a known method. Here, an example will be described.
  • the image coordinates can be known because the region to be transformed is set by itself. In this way, if the image coordinates of the four corners of the plane in the area to be transformed are input to equation (1), the three-dimensional world coordinates of the four corners can be calculated. Next, the three-dimensional coordinates of the four corners calculated by calculation are sequentially set to Xworld, Yworld, and Zworld at the right end of the right side. When the position and orientation of the right camera as viewed from the origin of the world coordinates are set in the matrix of external parameters, the positions of four points in the world coordinate system can be converted into image coordinates as viewed by the right camera. In this way, the image coordinates of the four corners of the plane in the region to be subjected to the viewpoint conversion are obtained to obtain the quadrangle conversion parameters. If the four corners can be calculated, all the coordinates inside the quadrangle can be calculated by interpolation.
  • the parallax analyzer 221 estimates that the lower region 112 where the road surface is often imaged is a portion that can be approximated to a plane. Whether or not each area includes a portion that can be approximated to a plane can be analyzed using parallax obtained from a stereo image. Assuming that the three-dimensional survey is performed with the left camera 100 as the center, the position of the road surface viewed from the left camera 100 can be analyzed using the parallax.
  • a plane attribute “road surface” is assigned to the lower region 112 in advance, and when the attribute is determined to be valid from the analysis result, the lower region 112 is determined to be a viewpoint conversion target. Conversely, if it is determined that the attribute is not valid, the lower area 112 is excluded from viewpoint conversion. That is, erroneous viewpoint conversion is suppressed by searching for a corresponding point without performing viewpoint conversion.
  • the conversion parameter calculation unit 223 calculates the conversion parameter of the area.
  • the corresponding point search unit 300 searches for a corresponding point between the left image (viewpoint converted image) subjected to viewpoint conversion by the viewpoint conversion unit 200 and the original image of the right image. In addition, when the viewpoint conversion is performed on the left and right images by the viewpoint conversion unit 200, a corresponding point is searched from the left and right images after the viewpoint conversion. As shown in FIG. 4, the corresponding point search unit 300 can function as a feature point extraction unit 310, a feature amount description unit 320, a maximum error setting unit 330, a corresponding point search unit 340, and a reliability calculation unit 350. .
  • the feature point extracting unit 310 extracts a feature point from the left image.
  • a feature point is a unique point such as a corner of an object on an image. Note that the feature point may be extracted from one of the left and right images, and the feature point may be extracted from the right image.
  • the feature value description unit 320 describes a feature value obtained by quantifying a change in the surrounding luminance of the feature point extracted by the feature point extraction unit 310.
  • the search range of the right image may be set to an area expanded by parallax.
  • the maximum error setting unit 330 sets a search range (vertical search range) in the vertical direction of the image in consideration of the maximum vertical error occurrence range of the left and right cameras 100 and 110 before performing the corresponding point search. For example, when obtaining feature points from the left image, the vertical search range is set to the right image. If the parallelization of the left and right images is perfect, the corresponding points of the feature points of the left image may be searched in the horizontal row of the same height in the right image. However, the range of the vertical error differs depending on the temperature characteristics and the assembling accuracy of the components to be used.
  • the vertical maximum error is defined, it is sufficient to perform a corresponding point search for the corresponding point of the feature point extracted from the left image within the range of ⁇ vertical maximum error from the same coordinate in the right image.
  • the vertical search range is reduced by the maximum error setting unit 330, and the horizontal direction is set by the maximum parallax value of the divided region of the left image. Thereby, the feature point candidates to be searched by the corresponding point search unit 340 can be reduced.
  • the corresponding point search unit 340 performs a corresponding point search on the right image while comparing the similarities of the feature amounts calculated for the feature points. Normally, a plurality of candidate corresponding points are found for one feature point. Among them, a candidate point having the highest similarity and having a similarity equal to or more than a predetermined threshold value is set as the corresponding point.
  • the left and right images are divided into an upper region and a lower region, and corresponding points are extracted from the image without viewpoint conversion in the upper region (infinity), and the viewpoint is extracted in the lower region (road surface).
  • the corresponding points are extracted from the converted image. Thereby, compared with the conventional case in which the viewpoint conversion is not performed, it is possible to obtain a large number of dense corresponding points from the upper and lower regions.
  • the reliability calculation unit 350 determines whether or not the area is an area that can be used for the camera geometry at the subsequent stage based on the similarity of the corresponding points obtained by the corresponding point search unit 340 and the number of corresponding points. Calculate the reliability, which is an index value. If the reliability is low, it is determined that the area cannot be used in the camera geometry. The reliability is calculated for all the regions to determine whether the region can be used for camera geometry. If it is determined that the area cannot be used in a number of areas equal to or greater than a certain threshold value, it is determined that calibration (geometric calibration) cannot be performed on the image acquired this time.
  • the camera geometric calibration unit 400 performs geometric calibration of the left and right cameras 100 and 110 based on the plurality of corresponding points obtained by the corresponding point search unit 300 so that the left and right images are parallel.
  • the camera geometric calibration unit 400 includes a corresponding point inverse transformation correction unit 410, a corresponding point aggregation unit 420, a noise corresponding point deletion unit 430, a geometric calibration parameter estimation unit 440, and an availability determination unit. 450, and may function as the geometric calibration reflection unit 460.
  • the corresponding point inverse transformation correction unit 410 performs a calculation to return the coordinates of the corresponding point obtained by using the viewpoint change or the image deformation to the coordinate system on the original image.
  • an inverse transformation parameter is used to return to the coordinate system of the original image.
  • the parameters of the inverse transformation have already been obtained by the inverse transformation parameter calculation unit 230, and the coordinates of the corresponding points of the viewpoint-transformed image (left image) of the left and right images are converted to the coordinates of the original image using the inverse transformation parameters.
  • Perform inverse conversion FIG. 14 shows an inverse conversion method when the viewpoint conversion is performed on the lower area of the left image.
  • the viewpoint is changed by modifying only the lower region of the left image as shown in the interruption image among the three stages.
  • a corresponding point search is performed in both the upper region and the lower region.
  • inverse transformation coordinate transformation
  • inverse transformation is performed to return the coordinates of the corresponding point (feature point) to the coordinates before the viewpoint transformation, and the position of the corresponding point in the image before transformation is calculated.
  • a number of corresponding points found based on the image after the viewpoint conversion are inversely transformed into coordinates on the original image and then used for geometric calibration.
  • the corresponding point inverse transformation correction unit 410 inversely transforms the corresponding point coordinates of the area subjected to the viewpoint transformation (deformation) and moves the coordinate on the coordinate system of the original image. Are aggregated as corresponding points in the coordinate system of the original image.
  • a certain evaluation scale is determined from among the fundamental matrices other than the outliers obtained in this way, and the fundamental matrix having the highest evaluation value is used as an initial value.
  • the evaluation scale is, for example, a random selection of a pair of reliable corresponding points excluding the eight corresponding points not used for the generation of the basic matrix, and how much error the pair of corresponding points causes in the basic matrix. Is used as an evaluation scale.
  • the basic matrix obtained in this manner is first set as an initial value of the basic matrix indicating the correspondence between the left and right cameras 100 and 110, and further, highly accurate geometric calibration is performed using the corresponding points determined to be reliable. Perform parameter optimization.
  • the geometric calibration parameter estimating unit 440 solves the problem. This makes it possible to estimate a basic matrix (geometric calibration parameter) with higher accuracy than the 8-point method.
  • the availability determining unit 450 first determines the number of corresponding points obtained from the corresponding point aggregating unit 420 (whether the number of corresponding points exceeds a predetermined number) and the outliers other than the outliers obtained from the noise corresponding point deleting unit 430. External information such as the number of pairs of corresponding points (whether the number of pairs exceeds a predetermined number) and the magnitude of the minimized distance error obtained from the geometric calibration parameter estimating unit 440 (whether the magnitude of the distance error is less than a predetermined value) Using the information, it is determined whether or not the result of the camera geometric calibration by the geometric calibration parameter estimation unit 440 can be used.
  • the left and right camera images are parallelized using the obtained geometric calibration parameters, there is a corresponding point pair in which a vertical error does not occur in the left and right image coordinates after the parallelization among the determined corresponding point pairs.
  • the availability is determined based on whether the ratio exceeds a predetermined ratio (for example, 95%).
  • the geometric calibration reflection unit 460 when the availability is determined by the availability determination unit 450, parallelization is performed using the parameter base matrix indicating the geometry of the left and right cameras 100 and 110 estimated by the geometric calibration parameter estimation unit 440. Update the affine table of the image transformation for generating the image.
  • the parallax image generation unit 500 generates a parallax image based on the left and right images captured by the right and left cameras 100 and 110 and the latest affine table updated in real time by the camera geometry correction unit 400.
  • the parallax image generation unit 500 according to the present embodiment functions as a parallelized image generation unit 510, a stereo matching unit 520, and a distance calculation unit 530, as shown in FIG.
  • the parallelized image generation unit 510 generates a left-right parallelized image using the affine table for generating a parallelized image updated by the camera geometric calibration unit 400.
  • the stereo matching unit 520 performs stereo matching on the parallelized left and right images to generate a parallax image.
  • the distance calculation unit 530 performs a three-dimensional distance conversion from the parallax image using the base line length of the left and right cameras 100 and 110 and the internal parameters (focal length and cell size) of the camera, thereby obtaining an arbitrary value on the left and right images. Calculate the distance to the object.
  • FIG. 15 Processing flowchart of control device 10>
  • a processing flow executed by the control device 10 when the left image is vertically divided into two as described above will be described.
  • the control device 10 repeats a series of processes shown in FIG. 15 at a predetermined cycle.
  • Step S01 first, the control device 10 (viewpoint conversion unit 200) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.
  • step S02 the control device 10 (viewpoint conversion unit 200) determines to divide the left image input in step S04 into upper and lower parts. Note that the right image is not divided.
  • step S03 the control device 10 (viewpoint conversion unit 200) divides the left image into two parts vertically and sets an area including the vanishing point VP as the upper area 111.
  • the upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be imaged, and is used for the corresponding point search without performing the viewpoint conversion.
  • step S04 the control device 10 (viewpoint conversion unit 200) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as a lower region 112, and proceeds to step S05. Transition.
  • the lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.
  • step S05 the control device 10 (viewpoint conversion unit 200) generates a conversion parameter for performing the viewpoint conversion of the lower region 112, and converts the corresponding point on the lower region 112 after the viewpoint conversion by the inverse conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to the coordinates of.
  • the viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.
  • step S06 the control device 10 (viewpoint conversion unit 200) converts the viewpoint of the lower area 112 of the left image using the conversion parameters generated in step S05.
  • step S07 the control device 10 (viewpoint conversion unit 200) generates a viewpoint conversion image in which the upper region 111 divided in step S03 and the lower region 112 converted in step S06 are combined. Then, the control device 10 (corresponding point search unit 300) performs a corresponding point search on the viewpoint converted image and the right image input in step S01. That is, a process of searching for a corresponding point in the right image based on the feature points and feature amounts on the image of the upper region 111 that has not been transformed and the lower region 112 that has been transformed is executed.
  • the first correspondence that is a set of a plurality of corresponding points based on the feature points and the feature amounts is obtained from the lower area 112 of the left image after the viewpoint conversion and the lower area 112 of the right image that has not been subjected to the viewpoint conversion.
  • a point group is extracted, and from the upper region 111 of the left image before the viewpoint conversion and the upper region 111 of the right image without the viewpoint conversion, a second set of a plurality of corresponding points based on the feature points and the feature amounts is obtained.
  • a corresponding point group is extracted.
  • step S08 the control device 10 (camera geometric calibration unit 400) straddles the upper and lower regions 111 and 112 of the plurality of corresponding points (first corresponding point group) found in the lower region 112 in step S07.
  • the remaining corresponding points are excluded and the inverse transformation is performed using the inverse transformation parameter generated in step S05, and the coordinate values of the remaining corresponding points are corresponded to the original image (the left image input in step S01). Returns to the coordinate value when the point was taken.
  • step S09 the control device 10 (camera geometric calibration unit 400) inversely transforms the coordinate values of the plurality of corresponding points (second corresponding point group) on the upper region 111 found in step S07 in step S08.
  • the coordinate values of a plurality of corresponding points (first corresponding point group) on the lower area 112 are collected.
  • the coordinates of the corresponding point on the left image in the first corresponding point group are the coordinates obtained by inversely converting the coordinates before the viewpoint conversion
  • the coordinates of the corresponding point on the right image in the first corresponding point group are the coordinates of the unconverted viewpoint.
  • the coordinates of the second corresponding point group are the coordinates before the viewpoint conversion for both the left and right images.
  • step S10 the control device 10 (camera geometric calibration unit 400) performs noise removal. From the corresponding points aggregated in step S09, eight corresponding point pairs are selected at random so as to form a coordinate system scattered on the image, and a so-called eight corresponding point pair (input corresponding point) is selected based on the selected corresponding point pair (input corresponding point). Calculate the values of the fundamental matrix by the point method. Then, the input corresponding points of the basic matrix that did not become outliers based on the values of the basic matrix are distinguished from the input corresponding points that became outliers by setting a flag so that they can be used in subsequent processing.
  • step S11 the control device 10 (camera geometric calibration unit 400) estimates the parameters of the geometric calibration using the coordinates of the corresponding point not determined as noise in step S10.
  • the basic matrix obtained by the above method as an initial value, an optimization problem that minimizes the distance error between the corresponding point on the image and the estimated point calculated using the basic matrix as a cost function solve.
  • a geometric calibration parameter with higher accuracy than the 8-point method.
  • step S12 the control device 10 (camera geometric calibration unit 400) uses information such as whether the magnitude of the distance error calculated in step S11 is less than a predetermined value or whether the number of corresponding points is equal to or more than a predetermined value. It is determined whether the geometric calibration parameters calculated in step S11 can be used. If it is determined that the geometric calibration parameters can be used, the process proceeds to step S13. On the other hand, if it is determined that the affine table cannot be used, the process proceeds to step S14 without updating the affine table.
  • step S13 the control device 10 (camera geometric calibration unit 400) updates the affine table for parallelizing the left and right images used in the previous frame based on the geometric calibration parameters calculated in step S11.
  • step S14 the control device 10 (the parallax image generation unit 500) generates a parallelized image of the left and right images using the stored affine table, performs stereo matching using the generated parallelized image, and generates a parallax image. I do.
  • the left and right cameras 100 and 110 perform viewpoint conversion on a region (lower region 112) that is greatly different in appearance, thereby finding from the left and right images.
  • the use of the close correspondence points enables highly accurate geometric calibration in the processing by the camera geometric calibration unit 400.
  • the parallax image generation unit 500 can generate a high-density and high-precision parallax image.
  • the left and right images are divided into an upper region and a lower region.
  • the viewpoint conversion is performed on the left image, and then the corresponding point is searched.
  • the viewpoint conversion is performed on both the left and right images.
  • the configuration for dividing the left and right images into a plurality of regions is not essential.
  • a plurality of corresponding points are extracted from the left image after the viewpoint conversion and the right image without the viewpoint conversion, and a plurality of correspondence points are extracted from the left image before the viewpoint conversion and the right image without the viewpoint conversion.
  • the points (second corresponding point group) may be extracted and geometric calibration of the left and right cameras 100 and 110 may be performed.
  • viewpoint conversion unit 200 Another example of viewpoint conversion by the viewpoint conversion unit 200 will be described.
  • the method of dividing the camera image into two vertically has been described.
  • a method of dividing the camera image into six or a method of dividing the camera image into free areas can be used.
  • the image is divided into six regions, and a method of searching for corresponding points on the left and right images for the six regions is selected.
  • the control device 10 divides the left image into six rectangular areas, and the six rectangular areas are given a plane attribute that is predicted to appear in each rectangular area while the host vehicle is traveling. It is determined based on the parallax image of the previous frame whether or not the plane attributes given to the six rectangular areas are valid.
  • the left and right cameras 100 , 110 are determined as conversion target regions, the positions and orientations of the left and right cameras 100 and 110 with respect to the conversion target region are estimated, and the estimated positions and orientations of the left and right cameras 100 and 110 are determined.
  • the conversion target area is converted into an image from the viewpoint of the right camera based on the left image, and a left image after the viewpoint conversion is obtained by combining the remaining area excluding the conversion target area from the six areas and the conversion target area.
  • the six rectangular regions are obtained by dividing the left image into two vertically and three horizontally, and each rectangular region is arranged in two rows and three columns.
  • the image is divided into an upper stage and a lower stage, which are referred to as a first region, a second region, and a third region from the left side of the upper stage, and are referred to as a fourth region, a fifth region, and a sixth region from the left side of the lower stage.
  • the plane attribute of the lower three rectangular regions (the fourth to sixth regions) in the six rectangular regions is “road surface”, and two rectangular regions located on the left and right of the upper three rectangular regions in the six rectangular regions.
  • the plane attribute of the (first and third regions) is “wall”, and the plane attribute of the center rectangular region (second region) of the upper three rectangular regions in the six rectangular regions is “infinity”. It is.
  • This method is effective in extracting feature points, reducing the processing time of description, and narrowing down corresponding point search candidates.
  • the plane attribute predicted to appear in each area while the own vehicle is running is determined, it is easy to select the deformation amount for each of the six areas.
  • the lower three regions perform the viewpoint conversion based on the viewpoint conversion assumed to be the road surface (ground) as before. Since the middle area (second area) near the upper infinity (vanishing point) contains only a distant view or the sky, no deformation occurs.
  • the upper left and right areas depend on the scenery of the running path, but when traveling in the city, buildings, trees, and the like are "walled" against the running path. There are many things that exist on the left and right. In such a case, conversion is performed assuming walls existing on the left and right of the travel path.
  • the road surface or the wall may be converted assuming a certain fixed value, or only the lateral position and the rotation of the wall are estimated from the parallax image of the previous frame as shown in the middle diagram of FIG. A method that utilizes this may be used. Convert the disparity values in the divided area into distances and perform plane estimation, and determine the amount of outliers and how many percent of disparity points occupy within a certain distance from the final estimated plane.
  • Judgment as to whether or not it may be approximated to a plane may be performed. If it is determined that the plane can be approximated, it is also calculated whether this plane is close to the two camera viewpoints and the difference in appearance due to the change of the viewpoint is large. If the difference in appearance is small, the need for viewpoint conversion is low in the first place. If viewpoint conversion is performed when the difference in appearance is large, the search performance of the corresponding point is greatly improved, so even if there is some error in the viewpoint conversion, it is better than performing the corresponding point search on the original image before conversion. Significantly closer correspondence points can be obtained.
  • a plane attribute predicted to appear in each rectangular area while the vehicle is traveling is given to each area in advance.
  • the upper left and right regions are “walls”
  • the lower three regions are “road surfaces”
  • the upper central region is “infinity”.
  • the plane attribute “far” is attached, and it is determined whether or not the plane estimated from the parallax value of the previous frame is similar to the plane defined by these attributes. If the plane attribute is different from the pre-assigned plane attribute, it is assumed that the plane cannot be approximated well, and the viewpoint conversion is not performed. This makes it possible to avoid erroneous viewpoint conversion.
  • the plane attribute is determined in advance, it is relatively easy to remove an outlier that is an unstable element, and the parameters to be estimated are narrowed, so that the stability is enhanced.
  • the two areas (first and third areas) located on the left and right of the upper row are equivalent to walls when there are buildings and trees along the traveling path. It is possible to use viewpoint conversion assuming that there is a plane (a region whose plane attribute is “wall”). However, when there are few three-dimensional objects around the road on a rural road, it is better to assign a non-planar attribute, such as "infinity", to the two regions without assigning the attribute of "plane”. The number of corresponding points may increase. For this reason, it is possible to understand the tendency of the three-dimensional structure of the scenery appearing in each region from the parallax image of the previous frame, and then determine whether or not to perform the viewpoint conversion and perform the corresponding point search.
  • a camera image may be pasted on a three-dimensional plane and deformed. To split the camera image. Therefore, as shown in FIG. 16, a region division is performed using a diagonal line as a boundary line of each region. Based on information obtained from the parallax image of the previous frame (for example, a road surface area estimation result and a road edge area estimation result), it is determined whether or not a portion that can be approximated to a plane exists in the image. Then, a region including a portion determined to be a portion that can be approximated to a plane is estimated as a road surface plane region.
  • the distance from the area to the left and right cameras 100 and 110 is less than a threshold, and the area where the distance is less than the threshold may be set as a road surface area, that is, an area for performing viewpoint conversion. Similarly, it may be determined whether or not the same object is in a region where the left and right cameras 100 and 110 look greatly different.
  • area division based on the assumption that there is a wall along the traveling direction of the traveling path is performed by using both the image color and the three-dimensional area. Division may be performed. For each of the divided areas, whether or not it can be approximated to a three-dimensional plane is estimated from the parallax image of the previous frame, as in the case of the six areas. If the plane can be approximated, the viewpoint conversion is performed according to the plane.
  • the entire screen can be used relatively effectively.
  • the background of the camera image is complicated, it is difficult to divide the area, and it is better to use a premise that knows what plane is to some extent, such as six areas, for stable determination. High stability.
  • this method has an advantage in a case where the environment recognition apparatus includes three or more cameras and performs triangulation from a pair of multiple cameras, and it is not clear which camera mainly performs three-dimensional reconstruction. It is easy to use for three-dimensional surveying with a camera. As described above, the present invention does not need to be a three-dimensional restoration mainly performed by the right camera as shown in FIG.
  • the control device 10 of the present embodiment includes a parallax image generation unit 500A.
  • the other parts are the same as in the first embodiment, and a description thereof will be omitted.
  • the parallax image generation unit 500A illustrated in FIG. 7 includes a region-based viewpoint conversion parallelized image generation unit 550, a region-based matching unit 560, a result integration unit 570, and a distance calculation unit 580.
  • the right and left cameras 100 and 110 may differ in the appearance of a short-distance subject even in stereo matching at the time of generating a parallax image as in the case of a corresponding point search. For this reason, parallax matching may be difficult even in stereo matching.
  • viewpoint conversion is performed also during stereo matching as in the corresponding point search of the first embodiment, parallax of a plane such as a road surface can be obtained densely. Further, as with the corresponding points, the corresponding points can be obtained as appropriate for the distant scenery without deformation.
  • the corresponding point method is used for algorithms that analyze the road surface shape, analyze small irregularities, or mainly observe the road surface such as where the vehicle can travel.
  • the region-specific viewpoint-converted parallelized image generation unit 550 divides the left image into two regions (upper region and lower region) in the far and near regions as in the first embodiment.
  • the viewpoint conversion is not performed, and in the lower region, the viewpoint conversion to the right camera viewpoint is performed simultaneously with the parallelization by the affine table.
  • viewpoint conversion may be performed by dividing an image after parallelization.
  • the conversion parameters at the time of viewpoint conversion and the affine table at the time of parallelization those calculated by the viewpoint conversion unit 200 and the camera geometric calibration unit 400 are used as in the first embodiment.
  • the region-specific matching unit 560 calculates parallax values individually for the two regions generated by the image generation unit 550, and generates parallax images individually.
  • the parallax values (parallax images) belonging to the lower region among the parallax values calculated by the region-specific matching unit 560 are corrected according to the viewpoint conversion, so that the upper region and the lower region are corrected. Correction is performed to match the meaning of the parallax values, and then the parallax values (parallax images) of the upper region and the lower region are integrated.
  • the distance calculation unit 580 calculates the distance from the parallax images integrated by the result integration unit 570, using the information on the base line length of the left and right cameras 100 and 110 and the information on the internal parameters of the left and right cameras 100 and 110. As a result, a parallax image based on a greater number of densely corresponding points can be obtained in a lower region (a road surface region at a short distance from the camera), which can greatly differ in the appearance of the left and right images, so that the accuracy of the parallax image is improved. improves.
  • the left and right images that is, the pair of the right image and the left image before the viewpoint conversion (first pair)
  • the viewpoint conversion after the region division and parallelization Both the result and the result of matching the left and right images (that is, the pair of the right image and the left image after the viewpoint conversion (second pair)) with the viewpoint conversion after the region division and parallelization are generated, and two matching results are generated.
  • the parallax value and the parallax image may be generated by using the one with the higher matching score indicating how similar the left and right images are.
  • the matching result with the viewpoint conversion is returned to the inversely converted state, and that the matching score indicating how similar the left and right images are similar can be referred to from each parallax value in both cases.
  • a three-dimensional object such as a pedestrian is present in the lower region, and the stereo matching using the image after the viewpoint conversion can prevent the matching score from being reduced. Can be improved.
  • FIG. 18 Flowchart of parallax image generation processing by control device 10>
  • a processing flow executed by the control device 10 (the parallax image generation unit 500A) when the left image is vertically divided into two when generating the parallax image as described above will be described.
  • the control device 10 repeats a series of processes illustrated in FIG. 18 based on the input of the parallax image request command.
  • the process of searching for the corresponding points for calibration and estimating / updating the geometric calibration parameters performed in steps DS02 to DS04 in the figure may use any method, and the first method shown in FIG.
  • the method is not limited to the method of the embodiment, and a known method may be used.
  • an example of a method of applying the viewpoint transformation of the first embodiment to the generation of a parallax image assuming that the calibration for parallelization has already been performed on the left and right images, will be described.
  • step DS01 first, the control device 10 (the parallax image generation unit 500A) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.
  • Step DS02 the control device 10 (corresponding point search unit 300) searches for corresponding points in the left and right images.
  • step DS03 the control device 10 (camera geometric calibration unit 400) estimates geometric calibration parameters for performing parallelization of the left and right images.
  • step DS04 the control device 10 (camera geometric calibration unit 400) updates the geometric calibration parameters used when creating the parallelized image of the left and right images with the geometric calibration parameters calculated in step D02. .
  • the estimated values of the relative positions and postures of the left and right cameras 100 and 110 and the parameters of the position and posture of the road surface and the stereo camera may be updated.
  • step DS05 the control device 10 (the parallax image generation unit 500A) determines to divide the left image input in step DS01 into upper and lower parts. Note that the right image is not divided.
  • step DS06 the control device 10 (the parallax image generation unit 500A) divides the left image into upper and lower parts, and sets the area including the vanishing point VP as the upper area 111.
  • the upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be captured, and is used for stereo matching (step DS07) without performing viewpoint conversion.
  • stereo matching with the image may be performed (step DS07).
  • step DS08 the control device 10 (the parallax image generation unit 500A) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as the lower region 112.
  • the lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.
  • the control device 10 (the parallax image generation unit 500A) generates a conversion parameter for performing the viewpoint conversion of the lower region 112 and performs the inverse conversion of the corresponding point on the lower region 112 after the viewpoint conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to coordinates.
  • the viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.
  • control device 10 (the parallax image generation unit 500A) performs viewpoint conversion of the lower region 112 of the left image using the generated conversion parameters.
  • the viewpoint conversion image generated in step DS08 may be a viewpoint conversion image in which the viewpoint of the left image is converted to the viewpoint of the right camera 110. Assuming that the cameras are located at the center positions of the left and right cameras 100 and 110, image conversion may be performed as if the images of the left and right cameras were brought to the position of the center of gravity of the stereo camera.
  • step DS07 the control device 10 (the parallax image generation unit 500A) performs parallax calculation by performing stereo matching on the upper region 111 in step DS06 and the corresponding right image region to generate a parallax image.
  • step DS09 the control device 10 (disparity image generation unit 500A) calculates the disparity value by performing stereo matching on the lower region 112 converted in viewpoint in step DS08 and the corresponding right image region. Generate an image.
  • step DS10 the control device 10 (the parallax image generation unit 500A) inversely transforms the parallax value based on the viewpoint conversion image generated in step DS10 by using an inverse conversion parameter, thereby obtaining the parallax value of the parallax value. Perform conversion.
  • step DS11 the control device 10 (the parallax image generation unit 500A) determines that if two images (left and right images) on the parallelized image coordinates before the viewpoint conversion have a superimposed region, the lower region before the viewpoint conversion is performed.
  • the matching score between the corresponding portion of the right image and the matching score of the corresponding portion of the right image is compared with the matching score of the lower region 112 and the corresponding portion of the right image after the viewpoint conversion, and the disparity value with the higher matching score is used as the disparity value of the lower region. select.
  • the comparison of score matching acts to preferentially use the disparity when the viewpoint is changed on the road surface and preferentially use the disparity when the viewpoint is not changed when a three-dimensional object is present. Image accuracy is improved.
  • step DS12 the control device 10 (the parallax image generation unit 500A) combines the parallax image of the upper region 111 generated in step DS07 and the parallax image of the lower region 112 selected through the comparison in step DS11 into one sheet.
  • the control device 10 the parallax image generation unit 500A
  • the left image is vertically divided.
  • the left image and the right image are regions that are significantly different in appearance and include a portion that can be approximated to a plane, the disparity value obtained by the viewpoint conversion is used.
  • stereo matching may be performed by performing viewpoint conversion on an area different from the lower area described above, and this type of area includes an area on which viewpoint conversion is performed in the first embodiment.
  • steps S10 and S12 can be omitted.
  • the present invention is not limited to the above embodiments, and includes various modifications without departing from the gist of the present invention.
  • the present invention is not limited to those having all the configurations described in the above embodiments, but also includes those in which some of the configurations are deleted. Further, a part of the configuration according to one embodiment can be added to or replaced by the configuration according to another embodiment.
  • the configuration of the control device 10 may be a program (software) that is read and executed by an arithmetic processing device (for example, a CPU) to realize each function of the configuration of the device.
  • Information related to the program can be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), a recording medium (magnetic disk, optical disk, etc.), and the like.
  • control lines and the information lines that are understood to be necessary for the description of the embodiment are shown, but all the control lines and the information lines related to the product are not necessarily used. It is not necessarily shown. In fact, it can be considered that almost all components are interconnected.
  • viewpoint conversion unit 100 left camera, 110 right camera, 111 upper region, 112 lower region, 200 viewpoint conversion unit, 210 region setting unit, 220 conversion parameter generation unit, 221 parallax analysis unit, 222 viewpoint conversion Attribute determination unit, 223: conversion parameter calculation unit, 230: inverse conversion parameter calculation unit, 240: viewpoint conversion image generation unit, 300: corresponding point search unit, 310: feature point extraction unit, 320: feature amount description unit, 330 ...
  • Maximum error setting unit 340: Corresponding point searching unit, 350: Reliability calculating unit, 400: Camera geometric calibration unit, 410: Corresponding point inverse transformation correcting unit, 420: Corresponding point aggregating unit, 430: Noise corresponding point deleting unit, 440: geometric calibration parameter estimating unit, 450: availability determining unit, 460: geometric calibration reflecting unit, 500: parallax image generating unit, 510: parallelized image generating unit, 520: stereo Matching unit, 530 ... distance calculator

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Processing (AREA)
  • Measurement Of Optical Distance (AREA)
PCT/JP2019/026561 2018-07-18 2019-07-04 車載環境認識装置 WO2020017334A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201980046818.8A CN112424565B (zh) 2018-07-18 2019-07-04 车载环境识别装置

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-135236 2018-07-18
JP2018135236A JP7219561B2 (ja) 2018-07-18 2018-07-18 車載環境認識装置

Publications (1)

Publication Number Publication Date
WO2020017334A1 true WO2020017334A1 (ja) 2020-01-23

Family

ID=69164301

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/026561 WO2020017334A1 (ja) 2018-07-18 2019-07-04 車載環境認識装置

Country Status (3)

Country Link
JP (1) JP7219561B2 (zh)
CN (1) CN112424565B (zh)
WO (1) WO2020017334A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113810567A (zh) * 2020-05-28 2021-12-17 三星电子株式会社 用于执行对象检测的电子装置及其操作方法
WO2024004325A1 (ja) * 2022-06-27 2024-01-04 日立Astemo株式会社 外界認識装置及び外界認識方法

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7465671B2 (ja) * 2020-02-20 2024-04-11 株式会社Subaru 画像処理装置および画像処理方法
WO2024009386A1 (ja) * 2022-07-05 2024-01-11 日立Astemo株式会社 画像処理装置及び画像処理方法
WO2024157410A1 (ja) * 2023-01-26 2024-08-02 日立Astemo株式会社 画像処理装置、撮像装置及び画像処理方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001273488A (ja) * 2000-03-27 2001-10-05 Toshiba Corp 障害物検出装置
JP2003256875A (ja) * 2002-02-27 2003-09-12 Fuji Heavy Ind Ltd ステレオ画像の位置ずれ調整装置、位置ずれ調整方法、およびステレオ式監視装置
JP2005251123A (ja) * 2004-03-08 2005-09-15 National Institute Of Advanced Industrial & Technology 画像処理装置および画像処理方法
JP2009041972A (ja) * 2007-08-07 2009-02-26 Toshiba Corp 画像処理装置及びその方法
JP2009276233A (ja) * 2008-05-15 2009-11-26 Toyota Central R&D Labs Inc パラメータ計算装置、パラメータ計算システムおよびプログラム
WO2016189878A1 (ja) * 2015-05-27 2016-12-01 京セラ株式会社 演算装置、カメラ装置、車両及びキャリブレーション方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001243456A (ja) * 2000-02-29 2001-09-07 Toshiba Corp 障害物検出装置及び障害物検出方法
JP3897984B2 (ja) * 2001-02-02 2007-03-28 三菱電機株式会社 監視カメラのキャリブレーション方法及び道路監視用カメラ
JP2003209858A (ja) * 2002-01-17 2003-07-25 Canon Inc 立体画像生成方法及び記録媒体
JP4414661B2 (ja) * 2003-02-25 2010-02-10 オリンパス株式会社 ステレオアダプタ及びそれを用いた距離画像入力装置
JP2006054504A (ja) * 2004-08-09 2006-02-23 Olympus Corp 画像生成方法および装置
JP2008187566A (ja) * 2007-01-31 2008-08-14 Sanyo Electric Co Ltd カメラ校正装置及び方法並びに車両
JP5588812B2 (ja) * 2010-09-30 2014-09-10 日立オートモティブシステムズ株式会社 画像処理装置及びそれを用いた撮像装置
JP5820787B2 (ja) * 2012-08-30 2015-11-24 株式会社デンソー 画像処理装置、及びプログラム
JP2015119372A (ja) * 2013-12-19 2015-06-25 株式会社日立製作所 マルチカメラ撮影システムおよびマルチカメラ撮影画像の合成方法
GB201409625D0 (en) * 2014-05-30 2014-07-16 Isis Innovation Vehicle localisation
CN107194972B (zh) * 2017-05-16 2021-04-02 成都通甲优博科技有限责任公司 一种摄像机标定方法及系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001273488A (ja) * 2000-03-27 2001-10-05 Toshiba Corp 障害物検出装置
JP2003256875A (ja) * 2002-02-27 2003-09-12 Fuji Heavy Ind Ltd ステレオ画像の位置ずれ調整装置、位置ずれ調整方法、およびステレオ式監視装置
JP2005251123A (ja) * 2004-03-08 2005-09-15 National Institute Of Advanced Industrial & Technology 画像処理装置および画像処理方法
JP2009041972A (ja) * 2007-08-07 2009-02-26 Toshiba Corp 画像処理装置及びその方法
JP2009276233A (ja) * 2008-05-15 2009-11-26 Toyota Central R&D Labs Inc パラメータ計算装置、パラメータ計算システムおよびプログラム
WO2016189878A1 (ja) * 2015-05-27 2016-12-01 京セラ株式会社 演算装置、カメラ装置、車両及びキャリブレーション方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113810567A (zh) * 2020-05-28 2021-12-17 三星电子株式会社 用于执行对象检测的电子装置及其操作方法
WO2024004325A1 (ja) * 2022-06-27 2024-01-04 日立Astemo株式会社 外界認識装置及び外界認識方法

Also Published As

Publication number Publication date
JP7219561B2 (ja) 2023-02-08
JP2020012735A (ja) 2020-01-23
CN112424565B (zh) 2023-01-03
CN112424565A (zh) 2021-02-26

Similar Documents

Publication Publication Date Title
WO2020017334A1 (ja) 車載環境認識装置
JP5870273B2 (ja) 物体検出装置、物体検出方法及びプログラム
CN110348454B (zh) 匹配局部图像特征描述符
CN110992356A (zh) 目标对象检测方法、装置和计算机设备
US9153066B2 (en) Image processing device, imaging device, and image processing method
WO2019160032A1 (ja) ３次元計測システム及び３次元計測方法
US11184604B2 (en) Passive stereo depth sensing
JP2017526082A (ja) 動作推定方法、移動体、およびプロセッサに動作推定方法を実行させるコンピュータプログラムコードでコード化された非一時的コンピュータ可読媒体
JP5250847B2 (ja) 画像処理装置、情報処理システム、画像処理方法、およびプログラム
JP4701848B2 (ja) 画像マッチング装置、画像マッチング方法および画像マッチング用プログラム
CN111142514B (zh) 一种机器人及其避障方法和装置
US20170069072A1 (en) Image processing apparatus, image processing system, and image processing method
JP2020173584A (ja) 物体検出装置
US10949988B2 (en) Information processing apparatus, information processing method, and program
JP2009186287A (ja) 平面パラメータ推定装置、平面パラメータ推定方法及び平面パラメータ推定プログラム
CN114764823A (zh) 自校正的深度计算方法、系统、介质及深度图像处理设备
JP2019120590A (ja) 視差値算出装置、視差値算出方法及びプログラム
KR20220026423A (ko) 지면에 수직인 평면들의 3차원 재구성을 위한 방법 및 장치
JP7195785B2 (ja) ３次元形状データを生成する装置、方法、及びプログラム
JP2017045124A (ja) 視差検出装置
JP2001109879A (ja) 画像処理装置および方法、並びに媒体
US10417783B2 (en) Image processing apparatus, image processing method, and storage medium
US10453206B2 (en) Method, apparatus for shape estimation, and non-transitory computer-readable storage medium
KR20200057929A (ko) 캘리브레이트된 카메라들에 의해 캡쳐된 스테레오 영상들의 렉티피케이션 방법과 컴퓨터 프로그램
JP2017084259A (ja) 対応点探索方法、および距離測定装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19838048

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19838048

Country of ref document: EP

Kind code of ref document: A1