JP2010009236A - Plane area estimation device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate an area representing an estimation target plane with high accuracy even if there is an area having no pattern. <P>SOLUTION: A first image and a second image are imaged in different positions/attitudes by an imaging device 12, and the first image is transformed by a projective transformation matrix, by an image transformation part 32. A difference of each pixel between a transformed image and the second image is calculated by a difference calculation part 34, and a plurality of road plane candidate areas represented by a hierarchical structure are detected by use of each of a plurality of threshold values by a plane candidate area detection part 36. By a plane area decision part 38, it is sequentially decided whether or not each of the road plane candidate areas inside the road plane candidate areas decided last time that each of them does not represent only a road plane among the road plane candidate areas detected by use of the threshold values in descending order from the largest threshold value represents only the road plane. By an area integration part 40, the road plane candidate areas decided that each of them represents only the road plane are integrated, and a road plane area is estimated. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a plane area estimation apparatus and program, and more particularly, to a plane area estimation apparatus and program for estimating an area representing an estimation target plane.

  Conventionally, in order to distinguish a road surface having a plurality of lines from obstacles existing on the road surface, a projection transformation matrix is obtained from feature points on the road surface, and the value of each point when the difference from the projection transformation image is calculated. An obstacle detection device that determines whether the road surface is an obstacle or not is known (Patent Document 1).

  In addition, a planar projection transformation matrix is estimated from two cameras, an image of one camera is transformed by the estimated projection transformation matrix, and a difference from the image of the other camera is taken to detect a planar region. A flat surface detector is known (Patent Document 2).

In the techniques described in Patent Documents 1 and 2, a planar projection transformation matrix is obtained, and a planar region is detected by difference calculation using an image transformed by the projection transformation matrix.
JP 2000-293893 A JP 2006-53754 A

  However, in the techniques described in Patent Documents 1 and 2 described above, even if a shift occurs due to conversion in an area without a pattern, the difference value does not change. There is a problem that can not be.

  The present invention has been made to solve the above problems, and provides a plane area estimation apparatus and program capable of accurately estimating an area representing an estimation target plane even if there is an area without a pattern. For the purpose.

  In order to achieve the above object, a planar region estimation apparatus according to a first aspect of the present invention includes an imaging unit mounted on a moving body that outputs a plurality of images obtained by imaging a range including an estimation target plane, and the imaging unit includes An area representing the estimation target plane in the first image captured by the imaging unit when the first position and the first attitude are set is a second position different from the first position and the first attitude, and An image converting means for converting the first image by a conversion for converting into an area in which the estimation target plane is to be represented in the second image imaged by the imaging means in the second posture; and the image converting means. Based on the converted first image and the second image, difference calculation means for calculating a difference between pixel values for each pixel, and each of a plurality of thresholds having different sizes is used to calculate the calculated pixel. value Determining whether or not each of the plane candidate areas detected using the largest threshold represents only the estimation target plane, as an area detection means for detecting an area where the difference is less than the threshold as a plane candidate area; Of the plane candidate areas detected using the threshold value in descending order from the next largest threshold value, the plane candidate areas detected from the plane candidate area previously determined not to represent only the estimation target plane. A determination means for sequentially determining whether or not only the estimation target plane is represented, and a plane candidate area determined by the determination means to represent only the estimation target plane. And an area estimation means for estimating an area representing the estimation target plane.

  In the program according to the second invention, the imaging means mounted on the moving body that outputs a plurality of images obtained by imaging the range including the estimation target plane is in the first position and the first posture. A region representing the estimation target plane in the first image picked up by the image pickup means is picked up by the image pickup means at a second position and a second posture different from the first position and the first posture. In addition, the first image and the second image converted by the image converting means for converting the first image by conversion for converting into an area in which the estimation target plane is to be represented in the second image. Based on the above, a difference calculation means for calculating a difference between pixel values for each pixel, and a region where the difference between the calculated pixel values is less than the threshold using each of a plurality of thresholds having different sizes Area detection means for detecting as a plane candidate area, whether or not each of the plane candidate areas detected using the largest threshold represents only the estimation target plane, and in descending order from the next largest threshold For each of the plane candidate areas detected from within the plane candidate area that was previously determined as not representing only the estimation target plane among the plane candidate areas detected by using only the estimation target plane A determination means for sequentially determining whether or not the object is represented; and a candidate plane region determined to represent only the estimation target plane by the determination means is integrated to estimate an area representing the estimation target plane It is a program for functioning as a region estimation means.

  According to the first invention and the second invention, a plurality of images obtained by imaging a range including the estimation target plane are output by the imaging means mounted on the moving body. Then, the region representing the estimation target plane in the first image picked up by the image pickup means when the image pickup means is in the first position and the first posture by the image conversion means is set to the first position and the first position. The first image is converted by conversion for converting the estimation target plane into a region in which the estimation target plane is to be represented in the second image captured by the imaging unit at the second position and the second posture different from the first posture.

  Based on the first image and the second image converted by the image conversion unit, the difference calculation unit calculates a difference in pixel value for each pixel. By using each of a plurality of thresholds having different sizes, the region detection unit detects a region where the difference between the calculated pixel values is less than the threshold as a plane candidate region.

  Then, the determination unit determines whether each of the plane candidate areas detected using the largest threshold value represents only the estimation target plane. In addition, the plane candidates detected from the plane candidate areas previously determined as not representing only the estimation target plane among the plane candidate areas detected using the threshold values in descending order from the next largest threshold value by the determination unit. For each of the areas, a determination is sequentially made to determine whether or not only the estimation target plane is represented. The region estimation unit integrates the plane candidate regions determined by the determination unit to represent only the estimation target plane, and estimates the region representing the estimation target plane.

  As described above, the size of the threshold used for detection of the planar candidate region detected using each of the plurality of thresholds having different sizes from the difference between the pixels of the converted first image and second image. By sequentially determining whether or not only the estimation target plane is represented in descending order, and estimating the area representing the estimation target plane, the area representing the estimation target plane can be accurately detected even if there is an area without a pattern. Can be estimated well.

  According to a third aspect of the present invention, there is provided a planar area estimation apparatus including an imaging unit mounted on a moving body that outputs a plurality of images obtained by imaging a range including an estimation target plane, and the imaging unit has a first position and a first attitude. The region representing the estimation target plane in the first image picked up by the image pickup means at the second position and the second posture different from the first position and the first posture is the image pickup means. The image conversion means for converting the first image by the conversion for converting to the region in which the estimation target plane is to be represented in the second image captured by the method, and the first image converted by the image conversion means, Based on the second image, difference calculation means for calculating a difference between pixel values for each pixel, and using the largest threshold value among a plurality of threshold values having different sizes, the difference between the calculated pixel values is An area that is less than the value is detected as a plane candidate area, and it is determined whether or not the plane candidate area represents only the estimation target plane, and only the estimation target plane is not represented in descending order from the next largest threshold value. And using the threshold value, the area where the difference between the calculated pixel values is less than the threshold value is detected as a plane candidate area, and the plane candidate area is an estimation target plane. A region that represents the estimation target plane by integrating a determination unit that sequentially determines whether or not it represents only the plane candidate region that is determined by the determination unit to represent only the estimation target plane And an area estimation means for estimating.

  According to a fourth aspect of the present invention, there is provided a program according to a first aspect of the present invention, in which an imaging unit mounted on a moving body that outputs a plurality of images obtained by imaging a range including an estimation target plane has a first position and a first posture. A region representing the estimation target plane in the first image captured by the imaging unit is captured by the imaging unit at a second position and a second attitude different from the first position and the first attitude. The image conversion means for converting the first image by conversion for converting the region to be estimated to be represented in the two images, and the first image and the second image converted by the image conversion means Based on the difference calculation means for calculating the difference between the pixel values for each pixel, the difference between the calculated pixel values is less than the threshold by using the largest threshold among a plurality of thresholds having different sizes. Is determined as a plane candidate area, and it is determined whether or not the plane candidate area represents only the estimation target plane, and it is previously determined that only the estimation target plane is not represented in descending order from the next largest threshold value. Using the threshold value, the area where the difference between the calculated pixel values is less than the threshold value is detected as a plane candidate area, and the plane candidate area represents only the estimation target plane. A determination unit that sequentially determines whether or not the plane is estimated; and a plane candidate region that is determined to represent only the estimation target plane by the determination unit is integrated to estimate a region that represents the estimation target plane This is a program for functioning as region estimation means.

  According to the third invention and the fourth invention, a plurality of images obtained by imaging a range including the estimation target plane are output by the imaging means mounted on the moving body. Then, the region representing the estimation target plane in the first image picked up by the image pickup means when the image pickup means is in the first position and the first posture by the image conversion means is set to the first position and the first position. The first image is converted by conversion for converting the estimation target plane into a region in which the estimation target plane is to be represented in the second image captured by the imaging unit at the second position and the second posture different from the first posture.

  Based on the first image and the second image converted by the image conversion unit, the difference calculation unit calculates a difference in pixel value for each pixel. By using each of a plurality of thresholds having different sizes, the region detection unit detects a region where the difference between the calculated pixel values is less than the threshold as a plane candidate region.

  Then, by using the largest threshold value among a plurality of threshold values having different sizes, the determination unit detects an area where the difference between the calculated pixel values is less than the threshold value as a plane candidate area, and the plane candidate area is an estimation target. It is determined whether or not only a plane is represented. In addition, the difference between the pixel values calculated by using the threshold value is less than the threshold value from the plane candidate area previously determined that the estimation unit does not represent only the estimation target plane in descending order from the next largest threshold value. An area is detected as a plane candidate area, and determination is sequentially performed to determine whether or not the plane candidate area represents only the estimation target plane. The region estimation unit integrates the plane candidate regions determined by the determination unit to represent only the estimation target plane, and estimates the region representing the estimation target plane.

  In this way, from the difference between each pixel of the converted first image and second image, a plane candidate region is detected using threshold values in descending order of a plurality of threshold values having different sizes, and only the estimation target plane is detected. By sequentially determining whether or not the region is represented and estimating the region representing the estimation target plane, the region representing the estimation target plane can be accurately estimated even if there is a region having no pattern.

  The determination means according to the first invention determines whether each of the plane candidate areas detected using the largest threshold is an area representing only the estimation target plane or an area including the area representing the estimation target plane. Plane candidates detected from within the plane candidate region that was previously determined to be a region including the region representing the estimation target plane among the plane candidate regions detected using the threshold value in descending order from the next largest threshold value For each of the areas, it is possible to sequentially determine whether the area represents only the estimation target plane or the area including the estimation target plane.

  The determination means according to the third invention uses the largest threshold value to detect an area where the difference between the calculated pixel values is less than the threshold value as a plane candidate area, and the plane candidate area represents only the estimation target plane. It is determined whether it is an area or an area including an area representing the estimation target plane, and from the plane candidate area previously determined to be an area including the area representing the estimation target plane in descending order from the next largest threshold value. Using the threshold value, an area where the difference between the calculated pixel values is less than the threshold value is detected as a plane candidate area, and the plane candidate area includes an area representing only the estimation target plane and an area representing the estimation target plane. It is possible to sequentially determine which region is to be determined.

  The planar region estimation apparatus according to the first and third inventions is a feature point from each of the first image and the second image imaged by the imaging means, and the first image and the second image Based on the search means for searching for the corresponding points between them and the corresponding points searched by the search means, the relative relationship between the first position and the first posture and the second position and the second posture is calculated. The three-dimensional position of the feature point on the estimation target plane in the first image is calculated based on the position / orientation calculation means that performs the search, the corresponding point searched by the search means and the relative relationship calculated by the position / orientation calculation means. A position calculation unit; and a plane parameter calculation unit that calculates a parameter representing the estimation target plane based on the three-dimensional position of the feature point on the estimation target plane calculated by the position calculation unit. A conversion matrix for conversion is calculated based on the relative relationship calculated by the posture calculation means and a parameter representing the estimation target plane calculated by the plane parameter calculation means, and the first image is converted based on the conversion matrix. be able to.

  The above planar area estimation device searches for a feature point and a corresponding point between the first image and the second image from each of the first image and the second image captured by the imaging means. And a position and orientation calculation means for calculating a relative relationship between the first position and the first orientation and the second position and the second orientation based on the corresponding points searched by the search means, and the search means And a position calculation means for calculating a three-dimensional position of the corresponding point searched by the search means based on the corresponding point searched by the position and the relative relationship calculated by the position and orientation calculation means. Based on the three-dimensional position of the corresponding point in the plane candidate area, it can be determined whether or not the plane candidate area represents only the estimation target plane. Thereby, it can be accurately determined whether or not the plane candidate region represents the estimation target plane.

  The difference calculation means can calculate a difference between pixel values of a predetermined area including the pixel for each pixel. Thereby, the plane candidate region can be detected based on the difference between the pixel values considering the shift of the conversion of the first image.

  As described above, according to the planar region estimation apparatus and program of the present invention, there is an effect that a region representing an estimation target plane can be accurately estimated even if there is a region without a pattern.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  As shown in FIG. 1, a planar area estimation device 10 according to the present embodiment is mounted on a vehicle (not shown) and captures an image of the front of the host vehicle including a road plane and outputs an image. And a computer 14 for estimating an area representing a road plane based on an image obtained from the imaging device 12 and displaying the estimation result on the display device 16.

  The image pickup device 12 picks up an image of the front of the host vehicle and generates an image signal of the image, and includes an image pickup unit (not shown) configured to perform A / D conversion on the image signal generated by the image pickup unit. A D conversion unit (not shown) and an image memory (not shown) for temporarily storing the A / D converted image signal are provided.

  The computer 14 includes a CPU, a RAM, and a ROM that stores a program for executing a planar area estimation processing routine described later, and is functionally configured as follows. The computer 14 acquires an image input unit 18 that acquires a plurality of images captured at different positions and orientations by the imaging device 12, and a feature point that extracts a plurality of feature points that are easy to track on each of the acquired images. An extraction unit 20, a corresponding point search unit 22 that searches for corresponding points between the two images from the feature points in each of the two images obtained by the feature point extraction unit 20, and a corresponding point search unit 22 The other image from which the corresponding point has been searched with reference to the position and orientation of the imaging device 12 when the image of one image from which the corresponding point has been searched is captured using the image coordinates of each image at each corresponding point thus obtained as an input. The position and orientation in which the change to the position and orientation of the imaging device 12 when the image is captured (relative relationship between the position and orientation) is calculated as the amount of movement of the motion of the imaging device 12 in the XYZ-axis direction and the rotation amount based on the XYZ axis Based on the image coordinates of each image at each corresponding point obtained by the output unit 24 and the corresponding point search unit 22 and the movement amount and the rotation amount calculated by the position and orientation calculation unit 24, the three-dimensional coordinates of each corresponding point And a three-dimensional position calculation unit 26 for calculating.

  The image input unit 18 acquires two images captured at different positions and orientations from the imaging device 12. In the following description, of the two images captured at different positions and orientations, the image captured at the first position and the first orientation is referred to as the first image, and the second position and the second orientation are used. The captured image will be described as a second image.

  The feature point extraction unit 20 extracts feature points from the first image acquired from the imaging device 12 and extracts feature points from the second image acquired from the imaging device 12. A feature point refers to a point that can be distinguished from surrounding points and that makes it easy to obtain a correspondence between different images. The feature points are automatically extracted by using a method (for example, a Harris operator) that detects pixels in which the gradient value of the change in shading increases two-dimensionally. In the method using the Harris operator, feature points are extracted as described below. First, the matrix M is calculated by the following equation (1), where the luminance of the point (u, v) of the image is I (u, v).

However, I _u and I _v represent horizontal and vertical differentiation, respectively, and G _σ represents smoothing by a Gaussian distribution with a standard deviation σ.

Then, using the eigenvalues λ ₁ and λ ₂ of the matrix M calculated by the above equation (1), the corner strength is calculated by the following equation (2).

  However, k is a preset constant, and a value of 0.04 to 0.06 is generally used. In the method using the Harris operator, a point at which the corner intensity is equal to or greater than a threshold value is selected, and the selected point is extracted as a feature point.

  The corresponding point search unit 22 searches the feature points extracted in the second image for points corresponding to the feature points of the first image extracted by the feature point extraction unit 20, and associates the feature points with each other. Do.

  In associating feature points between two images, a pair of points having similar brightness distributions in a small area set around the feature point is selected, and the selected point pair is regarded as a corresponding point. . A value such as SSD (Sum of Squared Differences) may be used to determine whether the two feature points are similar.

For example, regarding the feature point p = (x, y) of the first image I ₁ as a pair with the combination of the feature point p ′ = (x ′, y ′) of the second image I ₂ , the following (3) Each formula is calculated.

ただし、ｒは、小領域の大きさを決定する変数であり、予め定められている。

Here, r is a variable for determining the size of the small area and is determined in advance.

  Then, the feature point of the second image when the value calculated by the above equation (3) is minimized is searched for as a point corresponding to the feature point of the first image.

  The above calculation is performed for each feature point of the first image, and a point corresponding to each of the feature points of the first image is searched from the feature points of the second image. In addition, the corresponding point search unit 22 searches for at least eight sets of corresponding points between the two images with respect to the feature points extracted from each of the two images by the feature point extraction unit 20.

The position / orientation calculation unit 24 changes the position and orientation of the imaging device 12 when each of the two images is captured from the image coordinates of at least eight sets of corresponding points in the two images obtained from the corresponding point search unit 22. (Amount of movement in the XYZ axis direction and a rotation amount based on the XYZ axis) are calculated. As shown in FIG. 2, the change in position and orientation is a rotation matrix R from the first image to the second image (rotation amount based on the X axis, rotation amount based on the Y axis, and Z axis). This is a movement composed of six elements: a reference rotation amount) and a translation vector t (a movement amount t _{x in} the X-axis direction, a movement amount t _{y in} the Y-axis direction, and a movement amount t _{z in the} Z-axis direction). . Note that the elements of the rotation matrix R and the translation vector t are physical quantities representing conversion of image coordinates between two images.

Here, a calculation method of the rotation matrix R and the translation vector t from the first image to the second image will be described. For the image coordinates I _i of the corresponding points of the n points in the first image and the image coordinates I _i ′ of the corresponding points of the n points in the second image (n ≧ 8), if the corresponding points are correct and there is no error, There is a 3 × 3 matrix F that satisfies the following equation (4).

However, I _i = (u _i , v _i , 1) ^T , I _i ′ = (u _i ′, v _i ′, 1) ^T , and the image coordinates (u _i , v _i ) in the first image The image coordinates of the point in the second image corresponding to the point are (u _i ′, v _i ′).

Here, the matrix F satisfying the above equation (4) has a constant multiple indefiniteness. That is, when F _s satisfies the above equation (4), αF _s also satisfies the above equation (4) (where a is a real number). Therefore, the matrix F can be expressed as the following equation (5).

  Further, from the above equations (4) and (5), the following equation (6) is established.

Here, if there are eight or more pairs of corresponding points I _i and I _i ′, at least eight of the above formula (6) can be obtained, so that eight variables f _{11 to} f ₃₂ can be obtained. It should be noted that the eight formulas obtained need to be independent from each other, and even when errors are included, in order to perform stable calculation, other corresponding points are moving as differently as possible. It is preferable to search for a set of points as corresponding points.

  When the matrix F can be calculated as described above and the calibration matrix K of the imaging device 12 is known, the rotation matrix R and the translation vector t are obtained from the following equations (7) and (8). Can be calculated.

  The calibration matrix K of the imaging device 12 is expressed by the following equation (9).

However, _{f x} is the focal length in the X direction of the imaging device 12, _{f y} is the focal length in the Y direction of the image pickup device _{12, (c u, c v} ) is the image center. The image coordinates on the image are represented by a U-direction coordinate u corresponding to the X direction and a V-direction coordinate v corresponding to the Y direction.

  The three-dimensional position calculation unit 26 sets the image coordinates Ii and Ii ′ of n corresponding points between the two images obtained from the corresponding point search unit 22 and the two images obtained from the position / orientation calculation unit 24. The three-dimensional coordinates of the n corresponding points are calculated using the rotation matrix and the translation vector. The three-dimensional coordinates of the corresponding points can be calculated by the following method.

  First, the image coordinates of the corresponding points of the two images are (u, v), (u ′, v ′), the rotation matrix between the images is R, the translation vector is t, and the imaging device 12 is calibrated. When the action matrix is K, matrices P and P ′ such as the following expressions (10) and (11) are defined.

Then, if p ⁱ and p ′ ⁱ are the vectors in the i-th row of the matrices P and P ′, respectively, the three-dimensional coordinates X = (x, y, z, 1) ^T of the corresponding points are expressed by the following equation (12). It can be obtained as a solution.

  The computer 14 also includes a plane estimation unit 30 that estimates a parameter representing a road plane based on the three-dimensional coordinates of each corresponding point on the road plane, and a road plane area of one image as a road plane area of the other image. The image conversion unit 32 that converts one image based on a projective conversion matrix that indicates conversion for conversion so as to be superimposed on each other, and the difference value of each pixel between the converted image obtained by converting one image and the other image A difference calculation unit 34 to calculate, a plane candidate region detection unit 36 to detect a road plane candidate region based on the difference value of each pixel, and whether or not the detected road plane candidate region is a region representing a road plane And a region integration unit 40 that integrates regions representing road planes and displays them on the display device 16 as estimation results.

  The plane estimation unit 30 represents the road plane using the three-dimensional position of the corresponding point in the region representing the road plane of the first image among the three-dimensional positions of the corresponding point obtained by the three-dimensional position calculation unit 26. Estimate the parameters. The road plane to be estimated can be expressed by the following equation (14) in a three-dimensional space.

上記（１４）式において、（ａ,ｂ,ｃ）が、推定する道路平面のパラメータである。

In the above equation (14), (a, b, c) are parameters of the road plane to be estimated.

  The plane estimation unit 30 extracts corresponding points from the area representing the road plane of the first image, and obtains the parameters (a, b, c) from the three-dimensional coordinates of the extracted corresponding points. When three or more three-dimensional coordinates are obtained, the above parameters can be calculated. However, when the three-dimensional coordinates are obtained from more than three points, the least square method or the LmedS (Least Media Squares) method is used. Thus, the parameters (a, b, c) are estimated. The area representing the road plane of the first image may be obtained in advance based on the position and orientation of the imaging device 12 with respect to the road plane. Or you may obtain | require the area | region showing the road plane of a 1st image using the road plane area estimated by the process 1 step before. The estimated road plane parameter represents the relative position and posture of the road plane with respect to the first position and the first posture of the imaging device 12.

  The image conversion unit 32 is based on the change in the position and orientation between the first image and the second image calculated by the position / orientation calculation unit 24 and the road plane parameters obtained by the plane estimation unit 30. Calculating a projective transformation matrix indicating a transformation for transforming the region representing the road plane in the first image so as to overlap the region in which the road plane is to be represented in the second image, and calculating the calculated projective transformation The first image is converted by the matrix.

  The change in position and orientation between the first image and the second image is represented by the rotation matrix R and the translation vector t, and the road plane represented by the road plane parameters (a, b, c) is ax + by + cz. When = 1, the projective transformation matrix H is calculated by the following equation (15).

  However, the matrix K is a calibration matrix of the imaging device 12.

Using the projective transformation matrix H calculated by the above equation (15), a transformed image obtained by projectively transforming the first image is generated. If the pixel of the first image is Ii, the corresponding pixel Ji of the converted image is calculated by the following equation (16).
Ji = H · Ii (16)
For example, the first image as shown in FIG. 3A is projectively transformed to generate a converted image as shown in FIG.

  The difference calculation unit 34 calculates the difference value of each pixel for the converted image generated by the image conversion unit 32 and converted from the first image by the projective conversion matrix and the second image. For example, the difference value of each pixel between the converted image as shown in FIG. 3C and the second image as shown in FIG. 3B is calculated.

  The plane candidate area detection unit 36 detects a plurality of plane candidate areas from the difference value for each pixel obtained by the difference calculation unit 34.

  Here, the principle of the present embodiment will be described. Conventionally, when extracting a plane candidate region using two images, the relative relationship between the position and orientation between images and the relative relationship between the position and orientation of the estimation target plane are obtained, and the two images are By calculating the projective transformation matrix for the planar region at, taking the difference between the transformed image obtained by transforming one image with the projective transformation matrix and the other image, and binarizing the difference value of each pixel with a threshold value Seeking a planar area. When the difference between the converted image obtained by projective transformation and the other image is taken, the converted image and the second image represent the same position if the pixel is on the plane area, and the difference value becomes small. A planar area can be extracted. However, in a region without a texture pattern, the difference does not increase even if a shift occurs between the converted image and the second image, and the difference value decreases even if it does not belong to the plane region. End up.

  On the other hand, each of a plurality of threshold values is used to binarize the difference value of each pixel, and a region where the difference value is smaller than the threshold value is detected as a divided region represented by a hierarchical structure. For each divided area, use the 3D position of the feature points present inside the image to determine whether or not it is a plane area, thereby erroneously extracting the area without texture as a plane area Can be prevented.

  Therefore, in the present embodiment, the plane candidate area detection unit 36 detects a plurality of plane candidate areas represented by a hierarchical structure as described below, using a plurality of thresholds having different sizes.

  First, in the detection of a plane candidate area, a predetermined area centered on the target pixel is set for each target pixel, and a difference value between the pixels of the predetermined area is set as SAD (Sum of Absolute Differences) or SSD (Sum (Sum)). of Squared Differences), and if the calculated value is less than the threshold, it is determined that the pixel of interest is a pixel on the plane candidate region. If the calculated value is equal to or greater than the threshold, the pixel of interest is on the plane candidate region. It is determined that the pixel is not.

  The threshold value is changed, and using each threshold value, it is determined whether or not each target pixel is a pixel on the plane candidate area, and a plurality of plane candidate areas are detected. For example, when a road plane candidate region is detected using three threshold values that have a relationship of Th1> Th2> Th3, as shown in FIG. 4A, if the road plane candidate region is detected using the largest threshold value Th1, Wide road plane candidate areas including areas not on the plane are detected. Further, as shown in FIGS. 4B and 4C, the road plane candidate area is divided into many areas and detected as the threshold values used for detection are reduced to Th2 and Th3.

  As shown in FIG. 4C, when the threshold value is reduced to Th3, a part of the road plane is not detected as a candidate area. This is based on changes in the estimated position and orientation, and the plane parameters. This is because the difference value becomes larger than the threshold value Th3 even in the road plane due to the influence of the included error even in a portion where the luminance change is large. Even when the threshold value is reduced to Th3, a region other than a road plane having no texture is detected as a candidate region.

  When detecting a road plane candidate area with each threshold, if detection is performed based only on the SAD or SSD calculation value of each pixel, an isolated plane candidate area is detected or detected due to error or noise. For example, a missing area may be generated in the plane candidate area, so that the contraction / expansion process may be performed on the detected road plane candidate area.

  As described above, when a plurality of road plane candidate areas are detected using each of a plurality of threshold values, the relationship between the candidate areas is obtained from the inclusion relation of the road plane candidate areas, and the road is represented by a hierarchical structure. A plane candidate area is acquired.

  When the threshold value used for detection is changed from a large threshold value to a small threshold value, the candidate area detected with the small threshold value becomes a partial area within the candidate area detected with the large threshold value. Therefore, a plurality of road plane candidate areas represented by a hierarchical structure as shown in FIG. 5 is obtained from the inclusion relationship between the candidate areas detected with a large threshold value and the candidate areas detected with a small threshold value. The candidate area A1 detected using the largest threshold is set as the highest hierarchy, and the candidate area A1 is divided into candidate areas B1 to B5 detected with the next largest threshold. For example, the candidate area B2 is Is divided into candidate areas C2 and C3 detected with a large threshold. A plurality of road plane candidate areas represented by a hierarchical structure are obtained from the inclusion relation of these candidate areas.

  The plane area determination unit 38 calculates a plurality of road plane candidate areas represented by the hierarchical structure obtained by the plane candidate area detection unit 36, the positions of the corresponding points searched by the corresponding point search unit 22, and the three-dimensional position calculation. Based on the three-dimensional position of the corresponding point calculated by the unit 26, it is determined whether each of the road plane candidate areas is an area representing only the road plane, as will be described below.

  Using the obtained hierarchical structure of the candidate plane area, it is determined whether or not only the road plane is represented in order from the candidate road plane area at the upper level of the hierarchical structure. First, it is determined whether the road plane candidate area A1 corresponding to the highest hierarchy represents only the road plane, does not include the road plane area, or neither. This determination is performed based on whether or not the feature point in the determination target road plane candidate region is a point on the road plane. As shown in FIG. It is determined whether the point is on the plane. For all feature points in the road plane candidate area, if the distance between the three-dimensional position of the feature point and the road plane is less than or equal to the threshold value, all the feature points are composed only of points on the road plane, and the road plane candidate area Is determined to represent only the road plane. Further, for all the feature points in the road plane candidate area, if the distance between the three-dimensional position of the feature point and the road plane is larger than the threshold, all the feature points are not on the road plane, and the road plane candidate area Is determined not to include the road plane area.

  For example, since the road plane candidate area A1 corresponding to the highest hierarchy includes both points on the road plane and points not on the road plane, the road plane candidate area includes an area representing only the road plane, and a road It is not any area including a planar area. In such a case, it is determined that it is unknown, and the above determination process is similarly performed for each of the road plane candidate areas corresponding to the hierarchy below the road plane candidate area A1 in the hierarchical structure.

  For example, since the road plane candidate area B1 corresponding to the hierarchy one level lower than the highest level includes only feature points that are not on the road plane, it is determined not to include the road plane area, and further corresponds to the lower hierarchy. The above determination process is not performed for the road plane candidate region C1. Further, the other road plane candidate area B2 corresponding to the hierarchy one level lower than the highest level includes both feature points on the road plane and feature points not on the road plane, so it is determined that it is unknown. In addition, the above-described determination process is performed for each of the road plane candidate areas C2 and C3 corresponding to the next lower hierarchy.

  Thus, if it is determined that the road plane candidate area is unknown in the above determination process, the road plane candidate area corresponding to the next lower hierarchy is determined, and the road plane candidate area is only the road plane. Or the road plane candidate area corresponding to a lower hierarchy is not subjected to the determination process. As a result, the determination result for each road plane candidate region as shown in FIG. 7 is obtained.

  The area integration unit 40 integrates the road plane candidate areas determined by the plane area determination unit 38 as representing only the road plane as shown in FIG. 8A, as shown in FIG. 8B. The integrated candidate area is estimated as a road plane area.

  The candidate area integration processing is performed as described below. When n candidate regions R1 to Rn determined to represent only a planar region are obtained, it is first determined whether or not two different candidate region pairs Ri and Rj are adjacent to each other.

  Whether or not they are adjacent to each other is determined based on whether or not a straight line connecting the center of Ri and the center of Rj passes through candidate regions other than Ri and Rj. If the straight line connecting the centers does not pass through other candidate areas, it is determined that they are adjacent to each other, and if they pass through other candidate areas, it is determined that they are not adjacent. If Ri and Rj are not adjacent to each other, another candidate region pair is selected to determine whether or not they are adjacent to each other. If the two candidate areas Ri and Rj are adjacent to each other, the two candidate areas are integrated into one area.

  The region integration unit 40 repeats the above integration processing until there is no candidate region that can be integrated, estimates the integrated candidate region as a road plane region, and outputs it to the display device 16.

  The integration of the two candidate areas is performed as described below. When integrating two candidate region pairs Ri and Rj, first, the closest point among the points on the boundary of one candidate region is obtained from the points on the boundary of one candidate region, and the two points Are connected by straight lines. If the obtained straight line does not pass through the two candidate areas, the obtained straight line is added to a set of straight lines that are boundary candidates of the integrated area. For example, as shown in FIG. 9A, a straight line connecting the point Pm on the boundary of the candidate area Ri and the closest point on the boundary of the candidate area Rj does not pass through the inside of the candidate areas Ri and Rj. It is added to the set of straight lines that are the region boundary candidates. Further, as shown in FIG. 9B, since a straight line connecting the point Pn on the boundary of the candidate area Ri and the closest point on the boundary of the candidate area Rj passes through the inside of the candidate area Ri, do not do.

  The process as described above is performed for all points on the boundary between the candidate areas Ri and Rj, and a set of straight lines as boundary candidates for the integrated area is obtained.

  When a set of straight lines as boundary candidates for the integrated area is obtained, two straight lines are selected from the set of straight lines as boundary candidates, and the area surrounded by the selected two straight lines is set as the integrated area In addition, two straight lines that maximize the area of the integrated region are specified. For example, when a set of straight lines of boundary candidates as shown in FIG. 10A is obtained, if the straight lines L3 and L5 are selected, an integrated region as shown in FIG. When L1 and L7 are selected, an integrated region as shown in FIG. 10C is obtained. In this case, since the area of the integrated region as shown in FIG. 10C is maximized, the straight lines L1 and L7 are specified as two straight lines that maximize the area of the integrated region.

  Finally, an area overlapping with the road plane candidate area determined not to be a road plane area is deleted from the obtained integrated area, and the two candidate area pairs Ri and Rj are integrated.

  Next, the operation of the planar area estimation apparatus 10 according to the present embodiment will be described. In addition, the case where a road plane area is estimated while driving | running | working the vehicle carrying the plane area estimation apparatus 10 is demonstrated to an example.

  First, when the imaging device 12 starts continuous imaging in front of the host vehicle, the computer 14 executes a planar area estimation processing routine shown in FIG.

  First, in step 100, the first image captured in the first position and the first orientation and the second image captured in the second position and the second orientation are sequentially input from the imaging device 12. get.

  In step 102, a plurality of feature points are extracted from each of the first image and the second image acquired in step 100. In step 104, a plurality of feature points are extracted from the plurality of feature points extracted in step 102. At least eight sets of corresponding points corresponding to the first image and the second image are searched. In step 106, a second image based on the position and orientation of the imaging device 12 when the first image is captured based on the image coordinates of at least eight sets of corresponding points searched in step 104 is obtained. A translation vector and a rotation matrix representing a change in the position and orientation of the imaging device 12 when the image is taken are calculated.

  In step 108, the translation vector and rotation matrix between the first image and the second image calculated in step 106, and the first image and the second image searched in step 104, The three-dimensional coordinates of the feature point indicated by the corresponding point are calculated based on the image coordinates of the corresponding point. In the next step 110, a parameter representing the road plane is estimated based on the three-dimensional coordinates of the feature point indicated by the corresponding point in the region representing the road plane of the first image calculated in step 108 above.

  In step 112, based on the translation vector and rotation matrix between the first image and the second image calculated in step 106, and the parameter representing the road plane estimated in step 108, A projective transformation matrix is calculated. In step 114, the first image is converted according to the projective transformation matrix calculated in step 112.

  In the next step 116, a difference value of each pixel between the converted image obtained by converting the first image in step 114 and the second image acquired in step 100 is calculated. In step 118, a plurality of road plane candidate areas represented by a hierarchical structure are detected from the difference value of each pixel calculated in step 116, using each of a plurality of thresholds having different sizes.

  In step 120, the variable n indicating the hierarchy of the hierarchical structure is set to 1 which is an initial value, and at least one road plane candidate area detected using the largest threshold is set as a determination target area. Next, in step 122, among the three-dimensional coordinates of the corresponding points calculated in step 108, the n-th coordinate is calculated based on the three-dimensional coordinates of the corresponding points in the road plane candidate region that is the determination target region of the nth hierarchy. It is determined whether the road plane candidate area that is the determination target area of the hierarchy represents only the road plane, does not include the road plane area, or is unknown (is not any).

  In step 124, it is determined whether or not there is a road plane candidate region determined to be unknown in the determination on the determination target region in the nth layer in step 122. If there is a road plane candidate area determined to be unknown, in step 126, the variable n indicating the hierarchy of the hierarchical structure is incremented, and in step 128, it was previously determined that it was unknown in step 122 above. The nth layer road plane candidate region existing in the road plane candidate region is set as the determination target region of the nth layer, and the process returns to step 122.

  In step 124, it is determined that there is no road plane candidate area determined to be unknown, and that all the determination target areas in the nth layer represent only the road plane or do not include the road plane area. If YES, go to step 130. Further, even if there is a road plane candidate area determined to be unknown in step 124, the determination in step 122 is performed for the road plane candidate area detected using the smallest threshold. If yes, the process proceeds to step 130.

  In step 130, the road plane candidate areas determined to represent only the road plane in step 122 are integrated and estimated as a road plane area. In step 132, the estimated road plane area is displayed on the display device 16, and the plane area estimation processing routine is terminated.

  The step 130 is realized by a plane candidate integration processing routine shown in FIG. First, in step 150, among the road plane candidate areas determined to represent only the road plane in step 122, two road plane candidate area pairs Ri, Rj (1 ≦ i, j ≦ n, i ≠). j) is selected. In step 152, it is determined whether or not the pair Ri and Rj of the two road plane candidate areas selected in step 150 are adjacent to each other. When it is determined that the pair Ri and Rj of the two road plane candidate areas are not adjacent to each other, the process returns to step 150 to select another pair of road plane candidate areas.

  On the other hand, if it is determined in step 152 that the pair Ri, Rj of the two road plane candidate areas are adjacent to each other, in step 154, the pair Ri, Rj of the two road plane candidate areas is integrated.

  In step 156, it is determined whether or not the determination processing in step 152 has been performed for all pairs of road plane candidate areas among the road plane candidate areas determined to represent only the road plane in step 122. If there is a pair of road plane candidate areas that have not been subjected to the determination process of step 152, the process returns to step 150 to select another pair of road plane candidate areas. On the other hand, in step 156, if the determination process of step 152 is performed for all road plane candidate area pairs, the plane candidate integration process routine is terminated.

  The processing in step 154 is realized by the processing routine shown in FIG. First, in step 158, a straight line from a point on the boundary of one road plane candidate region to the closest point on the other road plane candidate region is obtained. In step 160, it is determined whether the straight line obtained in step 158 does not pass through any road plane candidate area. If neither road plane candidate area passes through, the boundary candidate of the integrated area is determined. The straight line obtained in step 158 is added to the set of straight lines.

  In step 162, it is determined whether or not the processing in steps 158 and 160 has been executed for all points on the boundary of one road plane candidate region. If not executed for all the points on the boundary, the process returns to step 158, and the processes of steps 158 and 160 are executed for the other points on the boundary of one road plane candidate region.

  If it is determined in step 162 that the processing has been performed for all points on the boundary, in step 164, when a region surrounded by two straight lines is defined as an integrated region from a set of straight lines as boundary candidates. Two straight lines that maximize the area of the integrated region are specified, and the two road plane candidate regions are integrated using the specified two straight lines as boundaries of the integrated region.

  In step 166, the overlapping area with the road plane candidate area determined not to include the road plane area in step 122 is deleted from the integrated area obtained in step 164, and the processing routine is terminated.

  As described above, according to the planar region estimation device according to the first embodiment, a plurality of different sizes are obtained from the difference between the pixels of the converted image obtained by converting the first image and the second image. For a plurality of candidate plane areas represented by a hierarchical structure detected using each of the threshold values, sequentially determine whether or not they represent only road planes in descending order of the threshold values used for detection, An area representing a road plane can be detected as an area as large as possible. Further, even if there is an area without a pattern, it can be prevented that it is erroneously detected as an area representing a road plane. In addition, it is possible to accurately estimate the area representing the road plane by integrating the areas representing only the road plane.

  Further, based on the three-dimensional position of the corresponding point in the road plane candidate area, it can be accurately determined whether or not the road plane candidate area represents only the road plane.

  Next, a planar area estimation apparatus according to the second embodiment will be described. In addition, since the planar area estimation apparatus which concerns on 2nd Embodiment has the structure similar to 1st Embodiment, it attaches | subjects the same code | symbol and abbreviate | omits description.

  The second embodiment is different from the first embodiment in that the first image is converted using an optical flow approximation.

  In the second embodiment, the conversion for converting the area representing the road plane in the first image so as to overlap the area where the road plane should be represented in the second image is performed by the image conversion unit 32. An approximate expression of optical flow is calculated, and the first image is converted by the calculated approximate expression.

Changes in the position and orientation between the first image and the second image calculated by the position / orientation calculation unit 24 are represented by a rotation matrix w = (w _x , w _y , w _z ) ^T and a translation vector t = ( t _x , t _y , t _z ) When expressed by ^T , an approximate expression (u, v) of the optical flow between the images of the point p = (x, y) ^T projected from the point on the road plane AX + BY + CZ = 1 ) ^T is calculated by the following equations (17) and (18).

From the above equations (17) and (18), the point on the second image corresponding to the point p = (x, y) ^T in the first image is p ′ = (x + u, y + v) ^T. For each pixel of the first image, the corresponding pixel in the converted image is obtained by the above equations (17) and (18), and the converted image is generated by converting the first image.

  In addition, since the other structure and effect | action of the planar area estimation apparatus which concern on 2nd Embodiment are the same as that of 1st Embodiment, description is abbreviate | omitted.

  Next, a planar region estimation apparatus according to the third embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the third embodiment, a road plane candidate region is detected by using a threshold value from within a road plane candidate region that was previously determined to represent only a road plane or not include a road plane region. This is different from the first embodiment.

  As illustrated in FIG. 14, the computer 314 of the planar region estimation apparatus 310 according to the third embodiment includes an image input unit 18, a feature point extraction unit 20, a corresponding point search unit 22, and a position / orientation calculation unit 24. And the three-dimensional position calculation unit 26, the plane estimation unit 30, the image conversion unit 32, the difference calculation unit 34, and the plane area determination unit 38, which will be described later, based on the difference value of each pixel. A plane candidate area detecting unit 336 for detecting a road plane candidate area from the determined road plane candidate areas, and whether or not the road plane candidate area detected by the plane candidate area detecting unit 336 represents only a road plane. A plane region determination unit 338 for determination and a region integration unit 40 are provided.

  The plane candidate area detection unit 336 sequentially detects a plurality of road plane candidate areas for a plurality of thresholds having different sizes in descending order from the larger threshold, as described below.

  First, based on the difference value of each pixel, using the largest threshold value, an area where the difference value is less than the threshold value is detected as a road plane candidate area.

  Next, among the road plane candidate areas obtained by the previous detection, the difference value is less than the threshold value using the next largest threshold value from within the road plane candidate area determined to be unknown by the plane area determination unit 338. Is detected as a road plane candidate region.

  Then, until there is no road plane candidate area determined to be unknown by the plane area determination unit 338, among the road plane candidate areas obtained by the previous detection, the plane area determination unit 338 determines that it is unknown. From the road plane candidate area, using the next large threshold value, the detection process of detecting the area where the difference value is less than the threshold value as the road plane candidate area is repeated.

  The plane area determination unit 338 determines the position on the image of the corresponding point searched by the corresponding point search unit 22 and the three-dimensional position calculation unit 26 every time the road plane candidate area is detected by the plane candidate area detection unit 336. Based on the calculated three-dimensional position of the corresponding point, whether the detected road plane candidate area represents only the road plane, does not include the road plane area, or is unknown (is not any) judge.

  If it is determined that the road plane candidate area is unknown, the road plane candidate area is detected again from the road plane candidate area determined to be unknown by the plane candidate area detection unit 336 using a threshold value. Is done. On the other hand, when it is determined that the road plane candidate area represents only the road plane or does not include the road plane area, the road plane candidate area is determined by the plane candidate area detection unit 336 from the road plane candidate area. Not detected.

  Next, a planar area estimation processing routine according to the third embodiment will be described with reference to FIG. In addition, about the process similar to 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  First, in step 100, a first image and a second image are acquired in order from the imaging device 12, and in step 102, a plurality of feature points are extracted from each of the acquired first image and second image. In step 104, at least eight sets of corresponding points corresponding to the first image and the second image are searched from the plurality of extracted feature points. In step 106, a translation vector and a rotation matrix representing a change in the position and orientation of the imaging device 12 when the second image is taken with reference to the position and orientation of the imaging device 12 when the first image is taken. calculate.

  In step 108, the three-dimensional coordinates of the feature point indicated by the corresponding point are calculated. In the next step 110, a parameter representing the road plane is estimated. In step 112, a projective transformation matrix is calculated, and in step 114, the first image is converted according to the calculated projective transformation matrix.

  In the next step 116, the difference value of each pixel between the converted image obtained by converting the first image and the second image is calculated. In step 350, a variable n for identifying a threshold is set to 1 which is an initial value, and in step 352, each of the values calculated in step 116 is calculated using the largest threshold (nth largest threshold). A road plane candidate region is detected from the pixel difference value.

  In the next step 354, among the three-dimensional coordinates of the corresponding point calculated in step 108, the corresponding point in the road plane candidate region detected using the nth largest threshold value in step 352 or step 356 described later. Whether each of the road plane candidate areas detected using the nth largest threshold value represents only the road plane, does not include the road plane area, or is unknown (whichever But not).

  Then, in step 124, it is determined whether or not there is a road plane candidate area determined to be unknown in the determination in step 354. If there is a road plane candidate area determined to be unknown, in step 356, the variable n for identifying the threshold value is incremented. In step 358, each of the road plane candidate areas is detected using the nth largest threshold value from the difference value of each pixel in each road plane candidate area previously determined to be unknown in step 354. Return to step 354.

  In step 124, there is no road plane candidate area determined to be unknown, and all road plane candidate areas detected using the nth largest threshold value represent only road planes, or roads When it is determined that the plane area is not included, the process proceeds to step 130. Further, even when there is a road plane candidate area determined to be unknown in step 124, the determination in step 354 is performed for the road plane candidate area detected using the smallest threshold. If yes, the process proceeds to step 130.

  In step 130, the road plane candidate areas determined to represent only the road plane in step 122 are integrated and estimated as a road plane area. In step 132, the estimated road plane area is displayed on the display device 16, and the plane area estimation processing routine is terminated.

  As described above, according to the planar region estimation device according to the third embodiment, a plurality of different sizes are obtained from the difference between each pixel of the converted image obtained by converting the first image and the second image. A road plane candidate area is detected using threshold values in descending order of threshold values, and it is sequentially determined whether or not only a road plane is represented, and an area representing a road plane is detected as an area as large as possible. it can. Further, even if there is an area without a pattern, it can be prevented that it is erroneously detected as an area representing a road plane. In addition, it is possible to accurately estimate the area representing the road plane by integrating the areas representing only the road plane.

  In the first to third embodiments, the case where a corresponding point is searched between two images from a plurality of feature points extracted from the two images has been described as an example. However, a feature point may be extracted from one of the two images, and a corresponding point corresponding to the extracted feature point may be searched from the other image.

  Further, in the above embodiment, the case where the translation vector and the rotation matrix are calculated as an example representing the change in the position and orientation of the imaging device has been described. However, the present invention is not limited to this, and the position of the imaging device Another index may be calculated as representing a change in posture.

  Also, in the plane area estimation processing routine, if it is determined that the road plane candidate area detected using the smallest threshold is also unknown, the road plane candidate area is detected and an area representing the road plane is displayed. The case where the repetition of the determination of the end is described as an example. However, the present invention is not limited to this, and a new lower threshold is prepared to detect a road plane candidate area, and an area representing the road plane is used. You may make it determine whether there exists.

It is a block diagram which shows the planar area estimation apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the translation vector and rotation matrix showing the change of a position and orientation. (A) An image diagram showing a first image, (B) An image diagram showing a second image, and (C) An image diagram showing a converted image obtained by converting the first image by projective transformation. (A) Image diagram showing a road plane candidate region detected using the largest threshold Th1, (B) Image diagram showing a road plane candidate region detected using the next largest threshold Th2, and (C) the smallest threshold It is an image figure which shows the road plane candidate area | region detected using Th3. It is an image figure which shows the hierarchical structure of the detected road plane candidate area | region. It is an image figure which shows the feature point which exists on a road plane, and the feature point which is not on a road plane. It is an image figure which shows the determination result with respect to each road plane candidate area | region. (A) The image figure which shows the road plane candidate area | region determined to represent only a road plane, (B) The image figure which shows the area | region showing the road plane estimated by unifying a road plane candidate area | region. (A) A straight line connecting a point on the boundary of one candidate region to the closest point on the boundary of the other candidate region, and showing a straight line that does not pass through any candidate region, and (B) It is a figure which shows the straight line which connects the nearest point on the boundary of the other candidate area | region from the point on the boundary of one candidate area | region, and passes through the inside of one candidate area | region. (A) The figure which shows the set of the straight line used as the boundary candidate of an integrated area, (B) The figure which shows the integrated area enclosed by two straight lines selected from the set of the straight line used as a boundary candidate, (C) The boundary candidate It is a figure which shows the integrated area | region where the area surrounded by two straight lines specified from the set of straight lines becomes the maximum. It is a flowchart which shows the plane area estimation process routine in the computer of the plane area estimation apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the plane candidate integration process routine in the computer of the plane area estimation apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the process routine of the process which integrates two candidate area | regions. It is a block diagram which shows the plane area estimation apparatus which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows the plane area estimation process routine in the computer of the plane area estimation apparatus which concerns on the 3rd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10,310 Planar area estimation apparatus 12 Imaging apparatus 14, 314 Computer 20 Feature point extraction part 22 Corresponding point search part 24 Position and orientation calculation part 26 Three-dimensional position calculation part 30 Plane estimation part 32 Image conversion part 34 Difference calculation part 36, 336 Plane candidate area detection unit 38, 338 Plane area determination unit 40 Area integration unit

Claims (9)

An imaging unit mounted on a moving body that outputs a plurality of images obtained by imaging a range including the estimation target plane;
An area representing the estimation target plane in the first image captured by the imaging unit when the imaging unit is in the first position and the first orientation is different from the first position and the first orientation. Image conversion means for converting the first image by conversion for converting into an area in which the estimation target plane is to be represented in the second image captured by the imaging means at the second position and the second orientation;
Difference calculating means for calculating a difference in pixel value for each pixel based on the first image and the second image converted by the image converting means;
Using each of a plurality of thresholds having different sizes, an area detection unit that detects an area where the difference between the calculated pixel values is less than the threshold as a plane candidate area;
It is determined whether each of the plane candidate areas detected using the largest threshold represents only the estimation target plane, and the plane candidate areas detected using the threshold in descending order from the next largest threshold Determining whether or not each of the plane candidate areas detected from within the plane candidate area previously determined as not representing only the estimation target plane represents only the estimation target plane. Sequential determination means;
Area estimation means for integrating the plane candidate areas determined to represent only the estimation target plane by the determination means and estimating the area representing the estimation target plane;
A plane region estimation apparatus including: An imaging unit mounted on a moving body that outputs a plurality of images obtained by imaging a range including the estimation target plane;
The region representing the estimation target plane in the first image captured by the imaging unit when the imaging unit is in the first position and the first orientation is different from the first position and the first orientation. Image conversion means for converting the first image by conversion for converting to an area in which the estimation target plane is to be represented in the second image imaged by the imaging means at the second position and the second orientation;
Difference calculating means for calculating a difference in pixel value for each pixel based on the first image and the second image converted by the image converting means;
Using a largest threshold value among a plurality of threshold values having different sizes, an area where the difference between the calculated pixel values is less than the threshold value is detected as a plane candidate area, and the plane candidate area includes only an estimation target plane. The calculated pixel value is determined using the threshold value from within the plane candidate region that was previously determined not to represent only the estimation target plane in descending order from the next largest threshold value. A determination unit that sequentially detects a region where the difference is less than the threshold as a plane candidate region and determines whether or not the plane candidate region represents only the estimation target plane;
Area estimation means for integrating the plane candidate areas determined to represent only the estimation target plane by the determination means and estimating the area representing the estimation target plane;
A plane region estimation apparatus including:   The determination means determines whether each of the plane candidate areas detected using the largest threshold is an area that represents only the estimation target plane or an area that includes an area that represents the estimation target plane. In the descending order from the next largest threshold value, the plane candidate areas detected using the threshold value are detected from the plane candidate areas previously determined to be areas including the area representing the estimation target plane. The planar region estimation apparatus according to claim 1, wherein for each of the planar candidate regions, determination is sequentially performed to determine which of the region representing only the estimation target plane and the region including the region representing the estimation target plane.   The determination unit detects an area where the difference between the calculated pixel values is less than the threshold as a plane candidate area using the largest threshold, and the plane candidate area represents only the estimation target plane. And the plane candidate area previously determined to be an area including the area representing the estimation target plane in descending order from the next largest threshold value. The region where the calculated pixel value difference is less than the threshold is detected as a plane candidate region using the threshold value, and the plane candidate region represents only the estimation target plane, and The planar area estimation apparatus according to claim 2, wherein determination is sequentially performed to determine which area includes an area representing the estimation target plane. Search means for searching for a point that is a feature point and corresponds between the first image and the second image from each of the first image and the second image imaged by the imaging means;
Position and orientation calculation means for calculating a relative relationship between the first position and the first attitude and the second position and the second attitude based on the corresponding points searched by the search means;
Position calculation means for calculating a three-dimensional position of a feature point on the estimation target plane in the first image based on the corresponding point searched by the search means and the relative relationship calculated by the position and orientation calculation means. When,
Plane parameter calculation means for calculating a parameter representing the estimation target plane based on the three-dimensional position of the feature point on the estimation target plane calculated by the position calculation means;
The image conversion unit calculates a conversion matrix of the conversion based on the relative relationship calculated by the position and orientation calculation unit and a parameter representing the estimation target plane calculated by the plane parameter calculation unit, and the conversion The planar area estimation apparatus according to claim 1, wherein the first image is converted based on a matrix. Search means for searching for a point that is a feature point and corresponds between the first image and the second image from each of the first image and the second image imaged by the imaging means;
Position and orientation calculation means for calculating a relative relationship between the first position and the first attitude and the second position and the second attitude based on the corresponding points searched by the search means;
Position calculating means for calculating a three-dimensional position of the corresponding point searched by the searching means based on the corresponding point searched by the searching means and the relative relation calculated by the position and orientation calculating means; Further including
6. The determination unit according to claim 1, wherein the determination unit determines whether the plane candidate area represents only the estimation target plane based on a three-dimensional position of the corresponding point in the plane candidate area. The planar area estimation apparatus according to any one of the preceding claims.   The planar area estimation apparatus according to any one of claims 1 to 6, wherein the difference calculation unit calculates a difference between pixel values of a predetermined area including the pixel for each pixel. Computer
In the first image imaged by the imaging means when the imaging means mounted on the moving body that outputs a plurality of images obtained by imaging the range including the estimation target plane is in the first position and the first posture. An area representing the estimation target plane is an area in which the estimation target plane is to be represented in the second image captured by the imaging unit at a second position and a second attitude different from the first position and the first attitude. Image converting means for converting the first image by conversion for converting to
A difference calculating means for calculating a difference between pixel values for each pixel based on the first image and the second image converted by the image converting means;
An area detecting means for detecting, as a plane candidate area, an area where the difference between the calculated pixel values is less than the threshold using each of a plurality of thresholds having different sizes.
It is determined whether each of the plane candidate areas detected using the largest threshold represents only the estimation target plane, and the plane candidate areas detected using the threshold in descending order from the next largest threshold Determining whether or not each of the plane candidate areas detected from within the plane candidate area previously determined as not representing only the estimation target plane represents only the estimation target plane. Sequential determination means, and a program for functioning as a region estimation means for estimating a region representing the estimation target plane by integrating the plane candidate areas determined to represent only the estimation target plane by the determination means . Computer
In the first image imaged by the imaging means when the imaging means mounted on the moving body that outputs a plurality of images obtained by imaging the range including the estimation target plane is in the first position and the first posture. An area representing the estimation target plane is an area in which the estimation target plane is to be represented in the second image captured by the imaging unit at a second position and a second attitude different from the first position and the first attitude. Image converting means for converting the first image by conversion for converting to
A difference calculating means for calculating a difference between pixel values for each pixel based on the first image and the second image converted by the image converting means;
Using a largest threshold value among a plurality of threshold values having different sizes, an area where the difference between the calculated pixel values is less than the threshold value is detected as a plane candidate area, and the plane candidate area includes only an estimation target plane. The calculated pixel value is determined using the threshold value from within the plane candidate region that was previously determined not to represent only the estimation target plane in descending order from the next largest threshold value. An area where the difference between the two is less than the threshold is detected as a plane candidate area, and a determination unit that sequentially determines whether the plane candidate area represents only the estimation target plane, and the estimation by the determination unit A plan for functioning as an area estimation means for integrating the plane candidate areas determined to represent only the target plane and estimating the area representing the estimation target plane. Grams.

Images