JP2008097444A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of keeping a plurality of images within a predetermined range, even if error occurs to the plurality of images, by reducing an inclination angle of a common plane to a relative horizontal plane. <P>SOLUTION: The image processor 1 converts an image acquired by a camera 2 into a bird's eye view image and stores it as bird's eye view image It-1 in a temporary image storage part 13. The image processor 1 also acquires an image, including a common area common to the bird's eye view image and converts it into a bird's eye view image It. A common plane is derived from the bird's eye view image It-1 and the bird's eye view image It and both images are compared. In deriving the common plane, two or more pairs of corresponding points, corresponding between both images, are extracted to select four that correspond to points, arranged such that the inclination angle defined by the relative horizontal plane and a straight line passing through a pair of corresponding points is an allowable inclination angle or smaller. The common plane is derived based on these four corresponding points. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus capable of deriving a common plane, for example, the ground, from captured images.

  For example, it is required to recognize surrounding obstacles and side grooves by detecting whether the height around the vehicle is low or high based on an image obtained by imaging the surroundings of the vehicle with an imaging means. Yes. In order to perform such image processing, it is required to obtain a common plane such as the ground as a reference when obtaining the height around the vehicle. As a technique for obtaining such a common plane, there is a plane estimation method using a stereo image disclosed in Japanese Patent Application Laid-Open No. 2004-94707 (Patent Document 1).

In this plane estimation method, a plurality of feature points are extracted from a standard image in a stereo image captured by an imaging device, and a matching process for searching for a corresponding point in a reference image is performed for each feature point to obtain a parallax. Each feature point is compared with other feature points based on the parallax and the coordinate points on the image to determine whether the feature point is a point indicating a road (plane) or an object on the road. Then, the three-dimensional position of the road is calculated from the remaining feature points excluding the feature points determined to indicate an object, and a common plane is derived.
JP 2004-94707 A

  By the way, when deriving such a common plane, for example, the common plane is required to fall within a predetermined angle or less with respect to a relative horizontal plane. However, in the invention disclosed in Patent Document 1, a feature point is extracted from a stereo image captured by an imaging device, and the three-dimensional road is extracted from the remaining feature points excluding the feature point determined to indicate an object. Plane estimation is performed to derive the common plane by calculating the position. For this reason, when an error occurs due to the resolution or the like of an imaging apparatus that acquires a stereo image (a plurality of images), the accuracy in deriving the common plane is lowered due to this error, and as a result, the common plane becomes relative. There was a problem that it might be greatly inclined with respect to the horizontal plane.

  Accordingly, an object of the present invention is to provide an image processing apparatus capable of reducing the inclination angle of the common plane with respect to the relative horizontal plane and keeping it within a predetermined range even when an error occurs in the captured image. There is.

  An image processing apparatus according to the present invention that has solved the above problems includes an image acquisition unit that acquires a plurality of images obtained by imaging a common area at least partially shared from a plurality of different positions, and a plurality of images acquired by the image acquisition unit. Corresponding point extraction means for extracting a plurality of corresponding points between images, and when a detection error is expected in the corresponding points, an inclination angle formed by a straight line passing through a pair of corresponding points in the image and a relative horizontal plane is a common plane. An inclination angle determination means for determining whether or not the inclination angle is equal to or less than a preset allowable inclination angle with respect to all of a plurality of straight lines passing through a plurality of sets of corresponding points obtained from the plurality of corresponding points; And a common plane deriving unit that derives a common plane based on a plurality of corresponding points when it is determined that the tilt angle is equal to or smaller than the allowable tilt angle.

  In the image processing apparatus according to the present invention, when a detection error at a corresponding point is expected, an inclination angle formed by a straight line passing through a set of corresponding points in the image and a relative horizontal plane is a preset allowable inclination with respect to the common plane. It is determined whether the angle is equal to or smaller than the angle. Then, when it is determined that the inclination angle with the relative horizontal plane is less than or equal to the allowable inclination angle for all of the plurality of straight lines passing through the plurality of sets of corresponding points obtained from the plurality of corresponding points, based on the plurality of corresponding points. Deriving a common plane. In this way, when the inclination angle formed by the straight line connecting the pair of corresponding points and the relative horizontal plane is equal to or smaller than the allowable inclination angle, the common plane is derived, and thus an error occurs in the corresponding points in the captured image. Even so, the inclination angle of the common plane with respect to the relative horizontal plane can be reduced to fall within a predetermined range.

  Here, a distance for calculating a distance threshold between corresponding points for one set of corresponding points based on a determination criterion in the inclination angle determining means and a corresponding point distance calculating means for obtaining a distance between a set of corresponding points. Threshold value calculating means, and the tilt angle determining means can compare the distance between a set of corresponding points with a distance threshold value between the corresponding points.

  Thus, by calculating the distance threshold between corresponding points for a set of corresponding points, and comparing the distance between the corresponding points with the distance threshold between the corresponding points, the inclination angle Can be determined by a simple calculation.

  Furthermore, the inclination angle determination means can determine the inclination angle when the detection error expected at the corresponding point is the maximum detection error.

  As described above, when the detection error is the maximum detection error, the inclination angle of the common plane with respect to the relative horizontal plane can be reliably within the predetermined range.

  According to the image processing apparatus of the present invention, even when an error occurs in the captured image, the inclination angle of the common plane with respect to the relative horizontal plane can be reduced and fall within a predetermined range.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

  As shown in FIG. 1, the image processing apparatus 1 according to the present embodiment includes a sensor information capturing unit 11, an overhead view creation unit 12, a temporary image storage unit 13, a corresponding point candidate extraction unit 14, a corresponding point selection unit 15, A plane deriving unit 16, an image viewpoint converting unit 17, and an object height determining unit 18 are provided. Furthermore, a camera 2 (2F, 2R, 2L, 2B) and an information output unit 3 are connected to the image processing apparatus 1, and the camera 2F is based on images output from the cameras 2F, 2R, 2L, 2B. , 2R, 2L, and 2B are detected.

  As shown in FIG. 2, the front camera 2 </ b> F is attached to a front grill portion at the front center of the vehicle S, and images the front of the vehicle S. The right camera 2R is attached to the right door position of the vehicle S, the left camera 2L is attached to the left door of the vehicle S, and the rear camera 2B is attached to the rear trunk cover of the vehicle S. , And the rear are respectively imaged. The cameras 2F, 2R, 2L, and 2B are each composed of a wide-field camera such as a so-called fisheye camera, and can capture images of the imaging areas EF, ER, EL, and EB with the respective cameras 2F, 2R, 2L, and 2B. . The cameras 2F, 2R, 2L, and 2B output images obtained by capturing images to the sensor information capturing unit 11 in the image processing apparatus 1, respectively.

  The sensor information capturing unit 11 in the image processing apparatus 1 captures an image output from the camera 2 and outputs it to the overhead view creation unit 12. The overhead view creation unit 12 detects the brightness of each pixel of the image output from these cameras 2 and maps it to a blank overhead view to create an overhead view.

  The overhead view creation unit 12 outputs the created overhead view to the image temporary storage unit 13 and the corresponding point candidate extraction unit 14. The image temporary storage unit 13 temporarily stores and stores the bird's-eye view (overhead image) output from the bird's-eye view creation unit 12, and uses the previously stored bird's-eye view image as a read request of the corresponding point candidate extraction unit 14. Output in response. The corresponding point candidate extraction unit 14 reads the overhead image stored in the temporary image storage unit 13 when the overhead image stored in the temporary image storage unit 13 is output from the overhead map creation unit 12. The bird's-eye view image read out here is a temporary storage image created immediately before the bird's-eye view image output from the bird's-eye view creation unit 12 (hereinafter referred to as “front bird's-eye view image”). The corresponding point candidate extraction unit 14 compares the overhead image output from the overhead view creation unit 12 with the previous overhead image read from the image temporary storage unit 13 and extracts a corresponding point candidate. The corresponding point candidate extraction unit 14 outputs the extracted corresponding point candidates to the corresponding point selection unit 15.

  The corresponding point selection unit 15 selects a corresponding point that can be easily used for image comparison from the corresponding point candidates output from the corresponding point candidate extraction unit 14. The corresponding point selection unit 15 outputs the overhead view image and the previous overhead view image including the selected corresponding point to the common plane deriving unit 16. The common plane deriving unit 16 derives a common plane based on the corresponding point candidates, the bird's-eye view image, and the previous bird's-eye view image output from the corresponding point selection unit 15.

  Here, when four selection points at which the inclination angle between the straight line passing through the selected corresponding point and the relative horizontal plane is equal to or less than the allowable inclination angle are selected, the common plane including the selected corresponding point, The overhead view image and the previous overhead view image are output to the image viewpoint conversion unit 17. When the inclination angle formed by the straight line passing through the selected corresponding point and the relative horizontal plane exceeds the allowable inclination angle, the corresponding point selection unit 15 is made to select the selected point again. The image viewpoint conversion unit 17 generates a viewpoint conversion image by performing viewpoint conversion on the image of the previous overhead view output from the common plane deriving unit 16 so as to have the same viewpoint as the viewpoint of the overhead view. The image viewpoint conversion unit 17 outputs the overhead image, the viewpoint conversion image, and each corresponding point to the object height determination unit 18.

  The object height determination unit 18 determines the height of the object around the vehicle based on the overhead view image, the viewpoint conversion image, and the corresponding points output from the image viewpoint conversion unit 17. When the object height determination unit 18 determines the height of the object, the object height determination unit 18 generates a bird's-eye view image that is colored according to the height of the object, and outputs it to the information output unit 3. The information output unit 3 includes a display that displays an image, for example, and displays the overhead view image output from the object height determination unit 18.

  Next, a processing procedure of the image processing apparatus according to the present embodiment will be described. FIG. 3 is a flowchart showing a processing procedure of the image processing apparatus according to the present embodiment, and FIG. 4 is a flowchart showing a processing procedure following FIG.

  As shown in FIG. 3, in the image processing apparatus 1 according to the present embodiment, the sensor information capturing unit 11 acquires an image output from the camera 2 (S1). Next, the overhead view creation unit 12 converts the viewpoint of the image captured by the camera 2 to generate an overhead view image (S2). When generating the bird's-eye view image, as shown in FIG. 5, the viewpoint of the fisheye image output from the camera 2 is converted to the viewpoint from above which is viewed from above, and the overhead image It is created in the following procedure. .

First, a distance Z _v from a virtual viewpoint O _{c of the} camera (virtual camera) 2α shown in FIG. 5 to a virtual screen (corresponding to a horizontal plane serving as a reference plane of the present invention) K is obtained by the following equation (0).

Z _v = W / {2 tan (Ω / 2)} (0)
Here, W: Width of the overhead view to be created (Pixel)
Ω: projection viewing angle (deg)

ただし、 Ｌ：魚眼画像の半径
ｆ（）：魚眼カメラ特性係数 Next, the image point of the virtual screen _{(X p, Y p, Z} p) fisheye image coordinates from _{(u p, v} p) to calculate the (S2). Therefore, first by the following (1) to (3), each coordinate _{(X se, Y se, Z} se) the determined from these coordinates, the fish-eye image coordinate value by the following equation (4) _{(u p} , V _p )

Where L: radius of fisheye image f (): fisheye camera characteristic coefficient

ただし、 λ：カメラ向き横角度（ｄｅｇ）
ξ：カメラ向き縦角度（ｄｅｇ）
ω：カメラ向き回転角度（ｄｅｇ）

Then, these coordinates are affine-transformed in the direction of viewing the horizontal plane by the following equations (5) and (6).

Where λ: Camera-oriented lateral angle (deg)
ξ: Camera vertical angle (deg)
ω: Camera rotation angle (deg)

  These calculations are performed for each pixel in the image output from each camera to generate an overhead image.

  When the overhead view creation unit 12 generates the overhead view image, the overhead view creation unit 12 outputs the generated overhead view image to the image temporary storage unit 13 and the corresponding point candidate extraction unit 14. The temporary image storage unit 13 temporarily records and stores the overhead image It output from the overhead view creation unit 12 (S3). Further, the corresponding point candidate extraction unit 14 reads the previous overhead image It-1 temporarily stored in the image temporary storage unit 13 when the overhead image It is output from the overhead view creation unit 12 (S4). .

  Then, the corresponding point candidate extraction unit 14 compares the overhead image It output from the overhead map creation unit 12 with the previous overhead image It-1 read from the image temporary storage unit 13, and sets a search range (S5). As a search range, an area in which a relative horizontal plane always exists, for example, a searchable range of several meters in the vicinity of the vehicle is set in advance, and a common common among searchable ranges in the overhead view image It-1 and the previous overhead view image It-1. Set the region as the search range. In addition, by performing imaging by the camera 2 every 20 ms, for example, when the vehicle S travels at a normal speed, images obtained by imaging the range where the overhead image It and the previous overhead image It-1 overlap from different positions are obtained. Can be acquired. The common area that is the overlapping range is set as the search range.

  Subsequently, corresponding point candidates are extracted within the search range (S6). In extracting corresponding point candidates, for example, feature points are extracted using a feature extraction process such as a Harris operator or a Laplacian of Gaussian filter. By these extraction methods, corresponding point candidates are extracted between the overhead image It and the previous overhead image It-1 for the feature points extracted in the search range set in step S5. When extracting corresponding point candidates, a correlation value between small regions (for example, 3 × 3 pixels, 5 × 5 pixels, and 7 × 7 pixels) around the feature point is obtained, and the correlation value that is the highest value is obtained. Only a pair of points is extracted, and the extracted two points are set as corresponding point candidates. For the derivation of the correlation value, for example, a pattern recognition technique such as template matching based on normalized correlation or difference summation can be used. Alternatively, it is assumed that the interval Δt between the time t when the image that is the source of the bird's-eye view image is captured and the time t-1 when the image that is the source of the previous bird's-eye view is captured is small, and the optical flow is used. Point candidates can also be extracted.

  When the corresponding point candidates are extracted, the corresponding point selection unit 15 randomly extracts a large number of four or more corresponding point candidates from the overhead image It (S6). The corresponding point selection unit 15 selects four corresponding points P1, P2, P3, and P4 from the extracted corresponding points (S7), and the selected corresponding points P1, P2, P3, and P4 are used as the common plane deriving unit 16. Output. Note that these corresponding point candidates may be extracted from the front overhead image It-1 instead of the overhead image It.

  The common plane deriving unit 16 derives a common plane from the output four corresponding points, but a certain amount of detection error is expected in the camera 2 that images the corresponding points. When this detection error increases, it is assumed that the common plane is greatly inclined with respect to the relative horizontal plane, and it is difficult to accurately detect surrounding objects. Therefore, here, whether or not four corresponding point candidates are formally adopted is examined. Therefore, first, with respect to the four corresponding points P1, P2, P3, and P4 that are output, relative distance threshold values Th12, Th13, Th14, Th23, Th24 with respect to six sets of corresponding points with two corresponding points as one set. Th34 is derived from the position of the corresponding point and the resolution of the camera 2 (S8).

The relative distance threshold is derived as follows. Here, the derivation of the relative distance threshold Th12 for the corresponding points P1 and P2 will be described with reference to FIG. Each symbol in the description here has the following meaning.
L ₁ : Horizontal distance from the camera position at time t to point P 1 L ₂ : Horizontal distance from the camera position at time t to point P ₂ r 1: Distance resolution at point P 1 r 2: Distance resolution at point P 2 h _c : Camera installation Height H1: First inclined plane H2: Second inclined plane RL: Relative horizontal plane α ₁ : Inclination of first inclined plane H1 with respect to relative horizontal plane RL when there is a maximum error between point P1 and point P2 α ₂ : Point The inclination of the second inclined plane H2 with respect to the relative horizontal plane RL when P1 and the point P2 have the maximum error B: Distance the camera has moved at time t and time t−1 d ₁₂ : between the points P1 and P2 Relative distance between

As shown in FIG. 6, the distance resolution r1 at the point P1 can be expressed by the following equation (7). Here, the distance traveled by the camera 2 can be represented by vehicle speed × Δt. This resolution is obtained based on the stereo vision principle of the camera at time t and time t-1.
r1 = (B ² + L ₁ ² ) tan δθ / B−L ₁ tan δθ (7)
Where δθ: camera field of view / vertical angle of view

Similarly, the distance resolution r2 at the point p2 can be expressed by the following equation (8).
r2 = (B ² + L ₂ ² ) tan δθ / B−L ₂ tan δθ (8)

Further, the resolutions G ₁ and G ₂ in the height direction at the points P1 and P2 can be obtained by the following equations (9) and (10), respectively.
G ₁ = r ₁ sin (β ₁ ) (9)
G ₂ = r ₂ sin (β ₂ ) (10)
However, β ₁ = tan ⁻¹ (h _c / L ₁ ), β ₂ = tan ⁻¹ (h _c / L ₂ )

At this time, planes that can be calculated when the point P1 and the point P2 have the maximum error include a first inclined plane H1 and a second inclined plane H2 shown in FIG. Among them, towards the maximum inclination alpha ₂ with respect to the relative horizontal RL of the second inclined plane H2, the distance to the camera 2 corresponds to the error that has in a direction away from the point P1 near the relative horizontal RL of the first inclined plane H1 It is larger than the maximum inclination alpha ₁ for.

Accordingly, when deriving the maximum inclination alpha ₂ with respect to the relative horizontal RL of the second inclined plane H2, the maximum inclination alpha ₂ with respect to the relative horizontal RL of the second inclined plane H2 may be calculated according to the following equation (11).
tan α ₂ = (G ₁ + G ₂ ) / (d ₁₂ −r ₂ cos (β ₂ ) −r ₁ cos (β ₁ )) (11)

  Here, an inclination threshold value α that is an allowable inclination angle is preset for the inclination plane. This inclination threshold value α is appropriately set depending on the height of the detection target. A procedure for setting the inclination threshold value α in the present embodiment will be described with reference to FIG. Here, the curb ES is set as the object at the lowest height position to be detected, and the object at a position higher than the curb ES is set to be detectable at the position of the maximum distance La from the vehicle S.

By setting the height of the curb ES to Hmin and not more than the height of the curb ES, an object located higher than the curb ES can be detected at the position of the maximum distance La from the vehicle S. Here, if the slope threshold value α is set with a margin, with the height Hmin / 2 that is half of the height Hmin of the curb ES being set as a reference, the slope threshold value α is obtained by the equation (12). be able to.
α = tan ⁻¹ (Hmin / 2La) (12)

  The inclination threshold value α can be set by the above equation (12). As a specific example, assuming that the maximum distance La from the vehicle S is 2 (m) and the height of the curb ES is Hmin = 10 (cm), the inclination threshold value α≈1.4 (deg).

Using this gradient threshold alpha, a maximum inclination alpha ₂ is whether smaller than the inclination threshold alpha determine the relative horizontal RL of the second inclined plane H2, point by the following equation (13) P1, P2 A relative distance threshold Th12 corresponding to is obtained. Further, by the following equation (14), determining the relative distance _{d 12} between the points P1 and P2.
Th12 = r ₁ cos (β ₁ ) + r ₂ cos (β ₂ ) + (G1 + G2) / tan α (13)
d ₁₂ = L ₂ −L ₁ (14)

Thus, when asked the relative distance threshold Th12 and relative distance _{d 12} for points P1 and P2, the relative distance _{d 12} determines whether greater than the relative distance threshold Th12 (S9).

Thereafter, the relative distance threshold values Th13, Th14, Th23, Th24 are also applied to the points P1 and P3, the points P1 and P4, the points P2 and P3, the points P2 and P4, and the points P3 and P4 by the same procedure. , Th34, and obtains the relative distances _{d 13, d 14, d 23} , d 24, d 34. Then, it is determined whether or not the corresponding relative distance di is larger than the relative distance threshold Thi (S9).

  As a result, if it is determined that the relative distance di is greater than the relative distance threshold value Thi for all the corresponding two points, the selected four points P1, P2, P3, and P4 are used as formal corresponding points. Adopt (S10). On the other hand, if it is determined that the relative distance di is equal to or less than the relative distance threshold Thi, the process returns to step S7, and the corresponding point selection unit 15 reselects four corresponding points from the corresponding point candidates. Steps S7 to S9 are repeated until this condition is satisfied.

On the other hand, the inclination threshold value can be directly used without obtaining the relative distance threshold value Th12. In this case, whether or not the conditional expression of α ≧ α ₂ , that is, tan α ≧ tan α ₂ (α <π / 2) is satisfied so that the maximum inclination α _{2 of} the second inclined plane H2 with respect to the relative horizontal plane RL becomes small. Judging. When it is determined that all the corresponding two points satisfy the condition of tan α ≧ tan α ₂ (α <π / 2), the selected four points P1, P2, P3, and P4 are officially corresponded. Adopt as a point

ここで、ホモグラフィ行列Ｈは３×３の行列であることから、平面Ｈ上に存在する点のそれぞれの画像への投影点の画像座標の組はすべて上記（１６）式で表すことができる。言い換えると、平面上にある点は、その平面のホモグラフィ行列がわかっていれば、ステレオ画像の一方の画像点からもう一方の画像上の対応点の位置が決定できることになる。したがって、ホモグラフィ行列Ｈは、三次元空間内で同一平面上にある４点の画像上での対応がわかれば、直接に求めることができる。 When the four points P1, P2, P3, and P4 are formally adopted in this way, a common plane homofluffy matrix is derived (S11). The common plane homography is derived in the following procedure. First, as shown in FIG. 8, when a point P that is one of the points P1 to P4 on an arbitrary plane H is viewed from the viewpoint C1 of the first camera and the viewpoint C2 of the second camera, the respective cameras Assume that the coordinates are p1 (u ₁ , v ₁ ), p2 (u ₂ , v ₂ ). At this time, the relationship between the two coordinates can be expressed by the following equation (15). Further, the homography matrix of the common plane is shown in Equation (16).

Here, since the homography matrix H is a 3 × 3 matrix, all sets of image coordinates of the projection points onto the respective images of the points existing on the plane H can be expressed by the above equation (16). . In other words, for a point on a plane, if the homography matrix of the plane is known, the position of the corresponding point on the other image can be determined from one image point of the stereo image. Therefore, the homography matrix H can be obtained directly if the correspondence on the four-point image on the same plane in the three-dimensional space is known.

Assume that correspondence between two images is given to n points existing on a certain plane in the three-dimensional space, as shown in FIG. Here, the coordinates of the first point P11 and the second point P12 in the first image when viewing the first point P1 and the second point P2 with the first camera are p11 (u ₁₁ , v ₁₁ ) and p12, respectively. (U ₁₂ , v _1i2 ). Further, the coordinates of the first point P11 and the second point P12 in the second image when the first point P1 and the second point P2 are viewed with the second camera are p11 (u ₁₁ , v ₁₁ ), p12 ( u ₁₂ , v ₁₁₂ ).

By generalizing this, the coordinates of the i-th point in the first image captured by the first camera are (u _1i , v _1i ), and the coordinates of the i-th point in the second image captured by the second camera are (u _2i , _v2i ). Further, if the constant α in the above equation (15) is α ₁ for this point, the above equation (15) can be expressed as the following equation (17).

When this equation (17) is modified, the following equation (18) is obtained.

Further, when n correspondences are established between the first image and the second image, the above equation (18) can be expressed as the following equation (19).
BJ = 0 (19)

Here, assuming that one arbitrary element in the matrix J in the above equation (19) is 1, the above equation (19) is solved by the least square method of a linear equation. At this time, in order to avoid solutions not in the sense of ^{J = 0, | J | 2} = 1 and in terms of and, min under the | BJ ^| by capturing a ^second minimization problem, the solution matrix ^{B T} It is given as an eigenvector corresponding to the smallest eigenvalue of B. In this way, a common plane homography matrix can be derived.

  When the common plane homography matrix is derived, the image viewpoint conversion unit 17 converts the front overhead image It-1 using the derived common plane homography matrix (S12) to create a converted overhead image It '. The converted overhead image It ′ created here is an image obtained by converting the image at time t−1 and estimating the image at time t. Now, FIG. 10A is a schematic diagram of the front overhead image It-1, FIG. 10B is a schematic diagram of the overhead image It, and FIG. 10C is a schematic diagram of the converted overhead image It ′. As can be seen from FIG. 10, when the overhead image It and the converted overhead image It ′ are compared, the positions of the relative horizontal planes RL coincide. On the other hand, positions other than the relative horizontal plane RL do not coincide with each other, and a shift occurs. For example, the position of the object M is shifted between the overhead image It and the converted overhead image It ′.

  Further, when the front overhead image is converted using an inappropriate homography matrix without using appropriate corresponding points, the front overhead image It-1 shown in FIG. 11A and FIG. If a converted overhead image is created from the previous overhead image It-1 with respect to the overhead image It shown in FIG. 11C, the image is blurred in the converted overhead image It ′ as shown in FIG. When such an image is obtained as a converted overhead image, for example, it is possible to re-select corresponding points by assuming that the conversion is based on an inappropriate homography matrix.

  Returning to the relationship between the overhead image It and the converted overhead image It ′ shown in FIG. 10, how the local regions of both overhead images change in order to detect a shift between the overhead image It and the converted overhead image It ′. Find out what you are doing. For this purpose, the object height determination unit 18 derives a local movement vector between the overhead view image It and the converted overhead view image It ′ (S13).

  In order to derive the local movement vector, as shown in FIG. 12A, a superimposed image is generated by superimposing the overhead image It and the converted overhead image It ′. In the superimposed image shown in FIG. 12A, the image of the overhead image It is indicated by a broken line, and the image of the converted overhead image It ′ is indicated by an alternate long and short dash line. In the superimposed image shown in FIG. 12A, first, local regions Xt-1A and Xt-1B in the converted overhead image It ′ are selected. These local areas can be selected as appropriate. For example, an area having a brightness greatly different from that of the surrounding area can be selected.

  When the local regions Xt-1A and Xt-1B in the converted overhead image It ′ are selected, corresponding local regions XtA and XtB corresponding to these local regions Xt-1A and Xt-1B in the overhead image It are derived. When deriving the corresponding local regions XtA and XtB, for example, a pattern recognition technique such as template matching by normalized correlation or sum of differences can be used. Further, it is possible to derive the corresponding local regions XtA and XtB using the optical flow, assuming that the interval Δt between the time t and the time t1− is very small.

  When the corresponding local regions XtA and XtB corresponding to the local regions Xt-1A and Xt-1B are thus extracted, the movement vectors ViA and ViB for the corresponding local regions XtA and XtB are used as local movement vectors from the local regions Xt-1A and Xt-1B. Ask for each.

  After obtaining the local movement vector, the object height determination unit 18 determines the object height (S14). The object height determination unit 18 is a relative horizontal plane RL, a three-dimensional object higher than the relative horizontal plane RL, or a groove or descending step lower than the relative horizontal plane RL based on the amount of movement of the local movement vector. Determine whether. Specifically, it is determined whether or not the absolute value of the movement amount of the local movement vector exceeds a predetermined threshold value. When it is determined that the absolute value of the local movement vector does not exceed the predetermined threshold value, it is in the relative horizontal plane RL. judge.

  Further, when it is determined that the amount of movement of the local movement vector exceeds a predetermined threshold value, the level relative to the horizontal plane is determined, and it is determined that the local region Xt-1A and the corresponding local region XtA are at a position lower than the horizontal plane. In this case, it is determined that it is a “side groove or descending step”. On the other hand, if it is determined that the local region Xt-1A and the corresponding local region XtA are at a position lower than the horizontal plane, it is determined to be a “three-dimensional object”.

  Thus, if the height with respect to the horizontal surface of a local position is determined, the determined height information will be recorded on an overhead view (S15). Based on the height information, the height information of the overhead view is completed. The overhead view completed here is as shown in FIG. 13, for example. In this overhead view, the colored display is divided into the areas of “relative horizontal plane”, “side groove / down step”, and “three-dimensional object”. By performing such colored display, it is possible to highlight the “relative horizontal plane”, “side groove / down step”, and “three-dimensional object”.

  When the bird's-eye view is completed in this way, the completed bird's-eye view is output to the information output unit 3 (S16) and displayed on a display or the like, or information is provided to other devices.

  As described above, in the image processing apparatus according to the present embodiment, when deriving a common plane, a straight line passing through a set of corresponding points and a relative horizontal plane RL are obtained for an image obtained from images captured by a plurality of cameras. It is determined whether or not the inclination angle formed is equal to or smaller than a preset allowable inclination angle with respect to the common plane. When it is determined that the inclination angle with respect to the relative horizontal plane RL is equal to or less than the allowable inclination angle for all the straight lines connecting a pair of corresponding points, a common plane is derived based on the plurality of corresponding points. As described above, the common plane is derived when the inclination angle formed by the straight line connecting the pair of corresponding points and the relative horizontal plane RL is equal to or smaller than the allowable inclination angle. However, the inclination angle of the common plane with respect to the relative horizontal plane RL can be reduced to fall within a predetermined range. In addition, since the distance between the two selected points and the distance threshold value are compared to determine whether the inclination angle with the relative horizontal plane RL is equal to or less than the allowable inclination angle, the calculation is simple. It can be.

  The preferred embodiment of the present invention has been described above, but the present invention is not limited to the above embodiment. For example, in the above embodiment, a camera is mounted on a vehicle and a plurality of images are acquired as the vehicle moves. However, a moving unit that moves the camera, a photographer that captures an image, and the like move the camera to move the plurality of images. It can be set as the aspect which acquires.

  In the above embodiment, the relative distance threshold is obtained from the straight line connecting the two points, and the distance between the two points is compared with the relative distance threshold and the distance between the two points. The angle of the straight line relative to the relative horizontal plane is compared with the inclination threshold α, and when the angle of the straight line connecting the two points to the relative horizontal plane exceeds the inclination threshold α, the corresponding point is selected again. You can also.

1 is a block configuration diagram of an image processing apparatus according to an embodiment of the present invention. It is a schematic diagram which planarly views the installation position of the camera in a vehicle and the imaging range from a camera. It is a flowchart which shows the process sequence of the image processing apparatus which concerns on this embodiment. It is a flowchart which shows the process sequence following FIG. It is a schematic diagram explaining the setting position of a virtual camera. It is explanatory drawing explaining the positional relationship used when calculating a relative distance threshold value. It is explanatory drawing explaining the positional relationship used when setting an inclination threshold value. It is a schematic diagram explaining the state at the time of seeing the same point from two camera positions. It is a schematic diagram explaining the state at the time of seeing two identical points from two camera positions. (A) is the figure which shows the front bird's-eye view image It-1 of time t-1, (b) is the figure which shows the bird's-eye view image It of time t, (c) is the conversion bird's-eye view which converted front bird's-eye view image It-1. It is a figure which shows image It '. (A) is the figure which shows the front bird's-eye view image It-1 of the time t-1, (b) is the figure which shows the bird's-eye view image It of the time t, (c) is the conversion which failed to convert the front bird's-eye view image It-1. It is a figure which shows the bird's-eye view image It '. (A) is a figure which shows the superimposed image which superimposed the overhead image It and the conversion overhead image It ', (b) is a figure explaining the local movement vector in the superimposed image. It is a figure of the screen which shows the height relationship of the area | region in an image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2 (2R, 2B, 2L, 2F) ... Camera, 3 ... Information output part, 11 ... Sensor information taking-in part, 12 ... Overhead view creation part, 13 ... Temporary image storage part, 14 ... Correspondence Point candidate extraction unit, 15 ... corresponding point selection unit, 16 ... common plane derivation unit, 17 ... image viewpoint conversion unit, 18 ... object height determination unit.

Claims (3)

Image acquisition means for acquiring a plurality of images captured from a plurality of different positions in a common area at least partially in common;
Corresponding point extracting means for extracting a plurality of corresponding points between a plurality of images acquired by the image acquiring means;
When a detection error is expected at the corresponding point, it is determined whether or not the inclination angle formed by the straight line passing through the pair of corresponding points in the image and the relative horizontal plane is equal to or smaller than a preset allowable inclination angle with respect to the common plane. An inclination angle determination means;
Commonly based on a plurality of corresponding points when it is determined that the inclination angle with the relative horizontal plane is less than or equal to the allowable inclination angle for all of the plurality of straight lines passing through a set of corresponding points obtained from the plurality of corresponding points. A common plane deriving means for deriving a plane;
An image processing apparatus comprising: A corresponding point distance calculating means for obtaining a distance between a pair of corresponding points;
Distance threshold value calculating means for calculating a distance threshold value between corresponding points for a set of corresponding points based on a determination criterion in the inclination angle determining means,
The image processing apparatus according to claim 1, wherein the inclination angle determination unit compares a distance between the pair of corresponding points with a distance threshold value between the corresponding points.   The image processing apparatus according to claim 1, wherein the inclination angle determination unit determines an inclination angle when a detection error expected at the corresponding point is a maximum detection error.

Images