WO2006123438A1

WO2006123438A1 - Method of detecting planar road region and obstruction using stereoscopic image

Info

Publication number: WO2006123438A1
Application number: PCT/JP2005/009701
Authority: WO
Inventors: Masatoshi Okutomi; Akihito Seki
Original assignee: The Circle For The Promotion Of Science And Engineering
Priority date: 2005-05-20
Filing date: 2005-05-20
Publication date: 2006-11-23

Abstract

A method of detecting a planar road region and an obstruction, the method detecting a travelable planar region and an obstruction by using a stereoscopic image from a vehicle mounted camera. The method has a step of dynamically estimating a projective transformation matrix relative to a road surface; a step of extracting a travelable planar region by using the projective transformation matrix; a step of calculating the inclination of a road surface by decomposing the projective transformation matrix; a step of creating a VPP image based on the inclination of the road surface; a step of presenting the travelable planar surface on the VPP image, and the position and direction of user’s vehicle; a step of calculating a distance for each direction based on the travelable planar region on the VPP image, the distance being the distance from the user’s vehicle to the obstruction; a step of calculating relative speed for each direction based on the travelable planar region on the VPP image, the relative speed for each direction being relative speed from the user’s vehicle to the obstruction; and a step of superposing displaying the VPP image superposed with the travelable planar region and displaying the distance for each direction and the relative speed for each direction.

Description

Specification

Road plane area and obstacle detection method using stereo images

In the present invention, in order to realize driver assistance for safe driving of automobiles and automatic driving of autonomous mobile vehicles, on-vehicle stereo cameras, road plane areas, leading vehicles, oncoming vehicles, parking vehicles, pedestrians, etc. The present invention relates to a road plane area and an obstacle detection method using stereo images for detecting all obstacles present on the road. Background art

Conventionally, laser radar, ultrasonic waves, and millimeter wave radars have been used for visually recognizing the driving environment in the guidance of autonomous mobile vehicles, particularly for detecting areas that can be driven, and for detecting obstacles in the driving environment. It can be roughly divided into the method of using images and the method of using images.

In the detection method using laser radar or millimeter wave radar, the device is generally expensive, and sufficient spatial resolution cannot be obtained. The detection method using an ultrasonic sensor has problems that it is difficult to measure far away and that the spatial resolution is low.

In addition, the method using an image can be divided into a method using a single eye and a method using a compound eye. Conventionally, most of the methods using images are monocular, that is, using images obtained from one viewpoint, and are mainly used in developed environments such as expressways, and white lines on the road surface (for example, The traveling area is detected by detecting separation lines and center lines. However, In general roads and parking lots where the presence of white lines, etc. is not guaranteed, and the colors and patterns of the road surface are varied, the density pattern projected on a monocular image is obtained by the monocular imaging method. Therefore, there is a problem that it is difficult to stably distinguish the travelable area and the obstacle.

On the other hand, in the imaging method using compound eyes, that is, the detection method using a stereo image, the three-dimensional structure of the environment can be used in principle, so the driving environment may be recognized more stably. In particular, since the area that can be traveled can be considered to be almost a plane in the space, the image is two-dimensionally projected to detect whether it is a planar area or an obstacle from the overlapping state of each image. For example, Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3 and Non-Patent Document 4 have already been proposed.

However, the conventional detection methods shown in Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3, and Non-Patent Document 4 assume that the positional relationship between the camera and the road plane is always constant. However, by carrying out calibration in advance, the fixed two-dimensional projective transformation is performed to detect the planar area of the road and obstacles. However, since the running surface (road surface) may tilt or the vehicle body may tilt with movement, the assumption that the positional relationship between the in-vehicle camera and the road plane is always constant does not actually hold. Often there is. Therefore, with the conventional detection method using the same fixed two-dimensional projective transformation, it is extremely difficult to cope with the inclination of the running surface (road surface) or the inclination of the vehicle body as the vehicle moves. There was a problem that there was.

As described above, road plane areas and obstacle detection methods can be broadly classified into those that use laser radar, ultrasound and millimeter wave radar, and those that use images. Use laser radar, ultrasonic wave and millimeter wave radar However, this detection method has a problem that the apparatus is expensive, the measurement accuracy is low, and the spatial resolution is low. In addition, detection methods that use images are limited to high-speed roads and other well-established usage environments, and cannot deal with vibrations during driving and slopes of roads. In the environment, there was a problem that the measurement accuracy deteriorated remarkably.

The present invention has been made for the above-described circumstances, and an object of the present invention is to use only image information obtained from an in-vehicle stereo camera, and to vibrate the camera due to the inclination of the road surface or the traveling of the car. It is now possible to dynamically determine the area in which the vehicle can travel in the real space, and to calculate and present the distance and relative speed to the obstacle in each direction as seen from the vehicle. It is intended to provide a road plane area and obstacle detection method using stereo images. Disclosure of the invention

The present invention relates to a road plane region and an obstacle detection method using stereo images, and the object of the present invention is a stereo moving image composed of a reference image and a reference image captured by an imaging means mounted on an automobile. A road plane area and an obstacle detection method using a stereo image that can detect a road plane area that can be traveled and all obstacles existing on the road surface by using only an image. A first step of dynamically estimating a projective transformation matrix with respect to, a second step of extracting a plane region that can be traveled using the projection transformation matrix obtained in the first step, and a first step of The third step of calculating the slope of the road surface by decomposing the obtained projection transformation matrix and the road surface is raised based on the slope of the road surface calculated in the third step. A fourth step of generating a virtual projection plane image viewed from And a fifth step of calculating the travelable plane area and the position and direction of the vehicle on the virtual projection plane image generated in the fourth step. The

Further, the present invention provides a sixth step of calculating a direction-specific distance from the vehicle to the obstacle from the plane area where the vehicle can travel on the virtual projection plane image, and the top of the virtual projection plane image. And a seventh step of calculating a relative speed for each direction from the vehicle to the obstacle from the plane area where the vehicle can travel, or alternatively, the virtual projection plane image includes the traveling The method further comprises an eighth step of displaying the possible plane regions in a superimposed manner and displaying the direction-specific distance and the direction-specific relative velocity information, or in the first step, LOG filter processing and histogram flattening processing are performed on the reference image and the reference image, and then the projection transformation matrix corresponding to the road surface between the stereo images is dynamically estimated by a region-based method. In fact, In the second step, the reference image is projectively transformed by the projective transformation matrix, a difference image between the reference image and the projective transformed reference image is obtained, and a smoothing filter is applied to the difference image. A binarized image is obtained by binarization using a threshold value, and a plane region that can be traveled is obtained by using the estimation result of the previous time plane region and taking a textureless region into the binary image. In the third step, the road plane attitude parameter indicating the inclination of the road surface is determined by the distance from the reference camera optical center to the road plane and the normal vector of the road plane. In the fourth step, by using the road plane attitude parameter, the reference image is projectively transformed to generate a virtual projection plane image parallel to the road plane. Ri is effectively achieved. Brief Description of Drawings

FIG. 1 is a flowchart showing the overall flow of the present invention.

FIG. 2 is a schematic diagram for explaining plane projection and two-dimensional projective transformation.

FIG. 3 is a diagram showing an example of a stereo original image.

FIG. 4 is a diagram showing a result image obtained by subjecting the stereo original image shown in FIG. 3 to LOG filter processing and histogram flattening processing.

FIG. 5 is a schematic diagram for explaining a projection transformation matrix and time domain estimation procedure using time series information.

FIG. 6 is a diagram showing a planar area extraction result.

FIG. 7 is a schematic diagram for explaining a region expected from the previous time. FIG. 8 is a diagram for explaining planar region extraction of a stereo image in which a large textureless region exists.

FIG. 9 is a schematic diagram for explaining textureless area processing. FIG. 10 is a diagram showing a planar area extraction result obtained by performing textureless area processing on a stereo image having a large textureless area.

FIG. 11 is a diagram showing a planar area extraction result obtained by performing textureless area processing on a stereo image in which a textureless area is also present in the planar area.

FIG. 12 is an overall flowchart of plane extraction in the present invention.

Fig. 13 is a schematic diagram for explaining the method of considering the textureless area.

FIG. 14 is a diagram for explaining the virtual projection plane image (VPP image) of the present invention. It is a schematic diagram of

Figure 15 shows an example of a reference image o

Figure 16 shows an example of a reference image o

Figure 17 shows the result of planar area extraction.

FIG. 18 is a diagram showing the result of overlaying the planar area of FIG. 17 on the original image of FIG.

FIG. 19 is a view showing a VPP image of the reference image of FIG.

FIG. 20 is a diagram showing a VPP image of the planar region of FIG.

FIG. 2 1 is a diagram for explaining the characteristics of the VPP image.

Fig. 22 is a schematic diagram for explaining the calculation of the vehicle ft force in the plane area on the VPP image and the distance by direction to these obstacles.

FIG. 23 is a diagram for explaining a relative speedometer in a direction-specific plane area. FIG. 24 is an image showing a road plane area and an obstacle detection result using the present invention.

FIG. 25 is a diagram showing an example of a road plane area and an obstacle detection result using the present invention.

FIG. 26 is a diagram showing another example of the road plane area and the obstacle detection result using the present invention. BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

In this embodiment, two left and right stereo cameras are mounted on a car, and the car equipped with the stereo camera (hereinafter referred to as the vehicle equipped with the stereo camera). All obstacles present on the road surface, such as pedestrians, parking vehicles, oncoming vehicles, and preceding vehicles, under conditions where there are vibrations and changes in the slope of the road surface due to the movement of the vehicle (referred to as a vehicle or vehicle) It is assumed that a situation is detected.

In the present embodiment, the right stereo camera is used as the reference camera, and the left stereo camera is used as the reference camera. Then, the stereo image taken with the reference camera is taken as the reference image, and the stereo image taken with the reference camera is taken as the reference image. Note that in the present invention, only a stereo image (that is, a base image and a reference image) is used to detect a plane area and an obstacle that can travel, so a camera that captures a stereo image (that is, a camera) In some cases, the left stereo camera is used as the reference camera, and the right stereo camera is used as the reference force camera. You can also do this.

By the way, as a stereo camera that takes a stereo image that goes to step S 1 0 0 and step S 1 1 0 below, the reference force meter is mounted on the left side of the vehicle, and the reference force meter is the vehicle. It is mounted on the right side. The standard camera is mounted on the right side of the car, and the reference power camera is mounted on the left side of the car.

First, as a point of interest of the present invention, using two in-vehicle stereo force cameras, an area corresponding to the plane of the space from the stereo image captured by the stereo camera and the orientation of the plane Thus, the travelable area in the coordinate system corresponding to the real space is dynamically determined, and the distance to the obstacle and the relative speed in each direction of the vehicle are calculated and presented. It is where it is.

That is, the road plane area and the obstacle using the stereo image according to the present invention. The object detection method is a method for dynamically determining a flat area where a vehicle can travel (hereinafter referred to as a travelable area) using image information obtained from two on-vehicle stereo cameras. By mapping the travelable area to the coordinate system corresponding to the real space, the distance and relative speed to the obstacle in each direction of the vehicle are calculated * and presented, warning of vehicle collision risk and collision avoidance This is a method to enable driver assistance and automatic driving of vehicles.

In the present invention, it is desirable to use a C CD camera as a stereo camera mounted on an automobile, but the present invention is not limited to this. In the following embodiments, it is assumed that a CD camera is used as a stereo force camera.

FIG. 1 is a flowchart showing the overall flow of the present invention. As shown in FIG. 1, in the road plane region and obstacle detection method using the stereo image according to the present invention, first, a projective transformation matrix for the road surface is dynamically estimated (step S 1 0). 0). Next, by using the projective transformation matrix obtained in step S 1 0 0 0, the road surface plane area (hereinafter, the road plane plane area is referred to as the road plane area, and may also be referred to as the travelable plane area). (Yes) is extracted (step S 1 1 0). Furthermore, the slope of the road surface is calculated by decomposing the projective transformation matrix obtained in step S 1 0 0 (step S

1 2 0). Based on the slope of the road surface calculated in step S 1 2 0, an image of the road surface viewed from above (in the present invention, this image is referred to as a virtual projection plane (VPP) image) is generated ( Step S

1 3 0). Then, on the virtual projection plane image (V P P image) generated in step S 1 3 0, the travelable plane area and the position and direction of the vehicle are presented.

(Step S 1 4 0). Next, the distance in each direction to the farthest part from the vehicle, that is, the obstacle, in the road plane area on the virtual projection plane image is calculated. Step SI 5 0). Similarly, the relative speed in each direction to the farthest portion from the vehicle in the road plane area on the virtual projection plane image, that is, to the obstacle is also calculated (step S 1 60). Finally, the plane area that can be traveled is displayed superimposed on the virtual projection plane image (VPP image), and the direction-specific distance and direction-specific relative velocity information calculated in steps S 1 5 0 and S 1 6 0 are displayed. Is displayed (step S 1 70).

Next, the present invention will be described in detail step by step. First, the principle of plane projection and two-dimensional projective transformation used in the present invention will be described.

As shown in Fig. 2, when a point M on a plane in the space is projected onto two stereo images, the stereo coordinates of the standard image I _b in the stereo image are m _b , the reference image _Assuming that the homogeneous coordinates on I _r are m _r , each is well known to be related by two-dimensional projective transformation, as shown in Equation 1 below.

[Equation 1]

m _r -Hm _b

Here, H is a 3 X 3 projective transformation matrix, and is assumed to be equal, allowing constant indefiniteness. In Fig. 2, O is the optical center of the reference camera, o 'is the optical center of the reference camera, R is the rotation matrix from the reference camera coordinate system to the reference camera coordinate system, and t is the reference. In the camera coordinate system, the translation vector of the reference force camera and the reference camera, d is the distance between the reference camera and the plane, and n is the normal vector of the plane.

Therefore, if the area that can be traveled, such as a road, can be considered to be almost flat in the space, by applying an appropriate projective transformation to one stereo image, Images that match. In other words, in the present invention in which obstacle detection is performed using plane extraction, 0

When there is an obstacle, the area is not recognized as a plane, so it is possible to detect that the area is not a travelable area. In addition, obstacles are included in areas other than the plane area, and as a result, obstacles can be detected by measuring the spread of the plane.

<Step S 1 0 0> Dynamically estimating the projective transformation matrix for the road surface First, for stereo images captured and input by two stereo cameras mounted on the car, In order to eliminate differences in brightness, the LOG (Laplacian Of Gaussian) finalizer is used, and the histogram is flattened. FIG. 3 shows an example of the input stereo original image. Fig. 3 (A) shows the reference image, and Fig. 3 (B) shows the reference image. Fig. 4 shows the resulting image obtained by applying LOG filter processing and histogram flattening processing to the stereo original image shown in Fig. 3. Fig. 4 (A

) Shows the image after processing of the reference image in FIG. 3 (A), and FIG. 4 (B) shows the image after processing of the reference image in FIG. 3 (B).

Next, a projective transformation matrix corresponding to a plane (in this embodiment, a road surface) is obtained by the region-based method disclosed in Non-Patent Document 5. In other words, in a certain region R ¹ , ^ that minimizes the following evaluation function is obtained by iterative optimization.

[Equation 2] e () = ∑ {I _r (Hm _b ) -I _b (m _b )} ²

m _b eR ′ where I _b (m) and I _r (m) represent the density values of the reference image and the reference image at the image position m, respectively.

For the above estimation, the projective transformation matrix 7 ^ 1

And region R ¹ (in the present invention, hereinafter referred to as “computation region”) is required. Therefore, in the present invention, the time series information is used, and the projective transformation matrix and the calculation area are obtained by the procedure described below.

First, at each time, a projection transformation matrix between stereo images, a projection transformation matrix H _m between two consecutive images of the reference image, and a planar region R for the reference image are obtained. When estimating a certain time t, the above estimation results up to the previous time t 1 are used. Hereinafter, the procedure will be described in more detail with reference to FIG.

(1) First, a projective transformation matrix between consecutive reference images I _b (t — 1) and I _b (t) is obtained. At that time, the initial value of the projective transformation matrix is the projection transformation matrix — estimated at the previous time, and the calculation domain is obtained at the previous time for I _b (t — 1). The planar area R (t − 1) can be used.

(2) Next, by using the projective transformation matrix („) obtained in (1), the plane area R (t-1) at the previous time is transformed, and the predicted value of the plane area at the current time is calculated. R ^ t).

(3) Furthermore, the projection transformation matrix between the stereo images I _b (t) and I _r (t) is estimated at the previous time — 1) as the initial value, and t calculated in (2). ) As the calculation area.

In the present invention, according to the above steps (1), (2), (3), the projection transformation matrix and the calculation area as the initial value sufficiently close to the true value are obtained in the continuous estimation using the time series image. Therefore, in the present invention, the projection transformation matrix and the calculation region estimation procedure described above require that the projection transformation matrix be obtained from the stereo image each time. Therefore, it is not necessary to know the camera's buttock parameters, the arrangement of the two cameras, the positional relationship between the camera and the road surface, etc. Even if there is a change due to the car body tilting or the camera vibrating due to the unevenness of the surface.

<Step S 1 1 0> Using the obtained projective transformation matrix, extract the plane area of the road surface (the plane area that can be driven)

Using the projective transformation matrix obtained in step S 1 0 0, an area corresponding to a plane is extracted from the reference image. Figure 6 shows the results of a series of processing using the stereo original image of Fig. 3. Fig. 6 (A) is a reference image (same as Fig. 4 (A)) that has been subjected to LOG filtering and histogram flattening. FIG. 6 (B) shows the result of projective transformation of the reference image of FIG. 4 (B) using the projective transformation matrix estimated by the method described in step S100. Figure 6 (C) shows the difference (absolute value) between the image in Fig. 6 (A) and the image in Fig. 6 (B). Looking at the image in Fig. 6 (C), the positions of the images in Fig. 6 (A) and Fig. 6 (B) coincide with the plane (road surface). Because it is black and it is shifted in other parts, it looks whitish. Next, the difference image in Fig. 6 (C) is binarized after applying an averaging filter in Fig. 6 (D). Fig. 6 (D) uses the image of Fig. 6 (A) and Fig. 6 (B) [Sum of Absolute Difference)! / Equivalent to performing a tricky matching.

Here, when the planar area extraction result of the previous time can be used, the result is used as follows. As described in step S 1 0 0, the plane area expected at the current time is obtained from the projection transformation matrix between the plane area R (t — 1) at the previous time and the time series of the reference image. Flat between time series images If the surface area does not change so rapidly, as shown in Fig. 7, the boundary between the planar area and the non-planar area at the current time falls within a certain width with respect to the boundary expected from the previous time. it is conceivable that. Therefore, the other parts can be regarded as planar or non-planar areas based on the results of the previous time, and this is reflected in the matching results in Fig. 6 (D). In addition, since the plane area that can be traveled is considered to have a certain size, the results of applying a process that excludes small areas and elongated areas by shrinking and expanding processes are shown in Fig. 6. (E). Figure 6 (F) shows the original image with reduced brightness other than the extracted plane area so that it can be seen how the extracted result corresponds to the original image. . From Fig. 6 (F), it can be seen that the extracted area is in good agreement with the actual road surface. On the other hand, in the method described above, the reference image and the reference image after the projective transformation are used to create a flat surface. It is assumed that the position on the image is shifted with respect to other than this, resulting in a difference in image density. However, problems arise when, for example, the walls of buildings without textures occupy a large area on the image. In such a region, even if the position is shifted, there is no difference in density. Fig. 8 (A) and Fig. 8 (B) show examples of such images. The matching result in this case (equivalent to Fig. 6 (D)) is shown in Fig. 8 (D). It will be shown. As shown in Fig. 8 (D), the matching result clearly includes the wall of the building in front.

Therefore, the following textureless region processing is applied to deal with such problems. The specific processing procedure will be described below with reference to FIG.

First, the area where the output result of the LOG filter is close to 0 is the textureless area. Four

Extracted as regions and labeled for each region (A and B in Fig. 9 (a)). In other words, these areas are difficult to determine planar areas by matching. In fact, if such an image is judged by matching, a region like the shaded area in Fig. 9 (c) is obtained, and as a result, the portion C is misjudged.

Next, the following determination is performed for textureless areas A and B (see Fig. 9 (d)). If the entire textureless area is included in the matching results in Fig. 9 (c), the textureless area is a planar area (in other words, area B). It should be noted that there are some boundaries between the actual planar area and the non-planar area, such as edges and shading differences on the image. In the present invention, the boundary portion is not a textureless region and belongs to the matching region. Therefore, even if the textureless area exists at the edge of the planar area, the textureless area appears inside the matching area by the boundary in the processing result. If the entire textureless area is not included in the matching results in Fig. 9 (c), the entire textureless area is defined as a non-planar area (that is, area A).

Finally, from the matching results in Fig. 9 (c), an accurate planar region can be obtained by removing the textureless region determined as a non-planar region (see Fig. 9 (e)).

The textureless region processing described above can be interpreted as follows. First, in the textureless region, it is not possible to determine whether or not there is a displacement even if local matching is performed. Therefore, the textureless areas are put together, and each area is regarded as a lump, and the deviation is judged. At that time, in the textureless area included in the planar area, However, since there is no displacement, the whole will match, but in the case of a textureless area in a non-planar area, the position will be displaced, so that a certain part of the area will always overlap with another area around it. The part is determined to be a non-planar part by matching. Therefore, the entire textureless area is determined as a non-planar area.

Figure 10 shows the results obtained by performing the above textureless region processing. FIG. 10 (B) shows the result of extracting the textureless region from the reference image of FIG. 10 (A). Figure 10 (C) shows the result of extracting the planar area by applying the textureless area processing described above. FIG. 10 (D) is a superimposed display with the original image.

Figure 11 shows another result obtained by performing textureless region processing. In this example, there are textureless areas in the sky (non-planar area) and the road surface (planar area). Only the textureless region corresponding to (region) is excluded from the matching results, and it can be clearly seen that good results are obtained as shown in Fig. 11 (E) and Fig. 11 (F).

In summary, the overall flow of plane extraction in the present invention is as shown in FIG. 12, and the processing of each part will be described below.

<1> L O G filter

As a disadvantage of the region-based method used in the present invention (see Non-Patent Document 5), if the brightness of the two images is different, processing cannot be performed successfully. Therefore, in order to remove the brightness difference between the images, a LOG filter is applied to the input stereo original image. The LOG filter is expressed as shown in Equation 3 below. In this embodiment, σ = 1 and window size = 7. 6

[Equation 3]

Further, since the contrast of the image to which the LOG filter has been applied is very low, the image is increased by increasing the contrast by histogram flattening.

<2> Projective transformation matrix estimation

By the method described in step S 1 0 0, the projection transformation matrix of the plane between stereo images is dynamically estimated.

<3> Projective transformation image

The reference image is projectively transformed using the projective transformation matrix representing the plane obtained by 2>. The plane represented by the projective transformation matrix matches as if it were taken from the reference camera, and anything that is not on the plane represented by the projective transformation matrix is distorted and projected. In the present invention, this property is used to obtain a planar area. <4> Difference image

Since the points existing on the plane are accurately projected onto the reference image, the difference in luminance value is small. Conversely, the difference in luminance values of points that are not on the plane increases. An area with a small difference is set as a plane area by performing threshold processing. At this time, the difference between each pixel is greatly affected by noise. Specifically, this is achieved by applying a smoothing filter to the difference image. Then, binarize using the threshold.

<5> Use of previous time plane area

In stereo moving images, images taken at the previous time and the current time have a very high correlation. Similarly, by using the high correlation in the processing results, it is possible to perform stable processing by using the processing results of the previous time. 7

Yes. Here, it is considered that the boundary between the planar region and the non-planar region extracted at the current time falls within a certain width with respect to the boundary expected from the previous time. Therefore, the other parts can be regarded as a planar area or non-planar area based on the result of the previous time, and this is reflected in the binary image (matching result).

<6> Textureless area

If only the difference image of 4> is used, there will be no change in density value in the texture-free area, that is, in the textureless area, even if different places overlap. It may be recognized as a road area. Also, if the textureless area is completely removed from the road area as it is, there is a risk that it will become a non-road area accidentally even though it is originally a road area. This is because there are textureless areas in the road area. Therefore, when an LOG filter is applied, the region where the spatial frequency is close to 0 is extracted as a textureless region and labeled for each region (A and B in Fig. 13 (a)).

<7> Consider textureless area

For each textureless region labeled in <6>, the region that is completely within the matching result of the reference image and the reference image (Fig. 13 (c)) is the road region, and the region where the deviation occurs ( C) in Figure 13 (c) is a non-road area.

<8> Open i ng processing

The road area we want to find is the road area where cars can travel. Therefore, the area where the car cannot travel is removed from the extracted results. First, a small area where the car cannot travel is removed from the road area extraction result. The area is obtained from the road area extraction result and the area below the threshold is excluded. Leaving. The entire road area is contracted and expanded.

<Step S 1 2 0> Calculate the slope of the road surface by decomposing the projective transformation matrix

Here, a method for obtaining the distance from the reference camera to the road plane and the normal vector n of the road plane, which are parameters representing the posture of the road plane, from the projective transformation matrix obtained in the plane area extraction will be described.

First, the projective transformation matrix H of the road plane is expressed by the following equation 4 in the case of a camera arrangement as shown in Fig. 2.

[Equation 4]

H = A ₂ (R + ¹

a

Where R is the rotation matrix from the reference camera coordinate system to the reference camera coordinate system, t is the translation vector of the reference camera and reference camera in the reference camera coordinate system, and d is the reference camera and road the distance of the plane, n represents at normal base data collected by Le road plane, a have a ₂ is an internal parameter of each the reference camera reference camera.

Here, adding the constant term k ≠ 0 indicates that the projective transformation matrix obtained from the image has a constant multiple of degrees of freedom. When the internal parameter A have A ₂ of camera are known, the number 5 below holds.

[Equation 5]

Here, the singular value decomposition of the projective transformation matrix H ′ is performed by the method of Faugeras et al. [Equation 6]

[Equation 7]

In the above equation 6, if the corresponding rows and columns in U and V are interchanged, they can be rearranged as> σ ₂ > ₃ > 0. After that, it is assumed that they are arranged in descending order. Compare number 5 and number 6 above.

[Equation 8]

UtRV = R, U * t = t 'V = n ¹

Then, the following number 9 is obtained.

[Equation 9]

t

The original R and d n can be obtained from the following relational expression

[Equation 1 0]

R = UR V

-= U- d d

n = Vn t, t

To obtain R,-, n from the above number 1 0, R,-, n can be obtained a a

I understand that.

Here we introduce the basis of the standard camera coordinate system (, ε β, n = + « ₂ e ₂ + ₃ e _3, and the three equations obtained from Equation 9 and n are Since V is a unit vector and n is a unit vector than an orthonormal matrix, and R 'is a rotation matrix, there is no norm change. Is done.

[Equation 1 1]

(The square of the norm on the left side) = (the square of the norm on the right side)

(k-n _t f + (k · Ujf = (σ. ^.) ² + ( ^b ) ² · ≠ zo)

( ² -bi>; ² + ( ² -bi »0 (i ≠ j) If this is set as ( ² —bi) = A, · = ι ~ 3) and solved as a simultaneous linear equation of, 1 get 2

[Equation 1 2]

= 0

(k ^{2 and} fX ^2- and ₂ ² ) ( ^2- and ₃ ² ) = 0 Therefore, if the singular value of H 'does not have (I) multiple solutions as shown below, (II) has multiple solutions Cases (III) Consider cases with triple solutions.

(I) σ _χ ≠ σ ₂ ≠ σ ₃

(II) <Tj = σ ≠ σ ₃ or σ ₁ ≠ σ ₂ = σ ₃

(III) σ ₁ = σ ₂ = σ ₃

Here, k = ± _{CT l} or k = ± a ₃ does not hold. This is indicated by a contradiction. Therefore, it can be seen that the constant multiple term k applied to H is obtained as σ ₂ when H 'is decomposed into singular values. Then when k> 0 This corresponds to the case where two cameras are arranged on the same side of the plane, and the camera takes a picture of the same plane, and is the camera arrangement handled in the present invention. In the case of the triple solution of (III), the optical centers of the two cameras are coincident, that is, they are only rotated. Therefore, from these relationships, general n is expressed by the following Equation 13 You can write

[Equation 1 3]

n (= ± 1)

(I) and _t ≠ and ₂ ≠ and ₃

? ¾: get 116 ₂ = 6 ₂ after 0. That is, R is the rotation matrix are _two laps e. The following formulas 14 and 15 and 16 are obtained by calculating each with the following conditions.

[Equation 1 4]

Number 1 5]

[Equation 1 6]

7%

― ^Σ ι 一 3

1 σ '2 0

(II) σ] = σ ₂ ≠ σ ₃ or ≠ σ ₂ = ₃ and (ill) σ i = σ ₂ (σ ₂ = σ ₃ )

From the above equation 1 = 3, = = 0, = ± 1 (= ± 1, = = 0). R 'becomes an identity matrix. At this time,

[Equation 1 7]

R = I

t σ, -σ, <

― = ― ^L n

You can write d and 2.

Here, due to the difference in the sign of,, R, t /, and n have indefiniteness of four solutions. Therefore, we give the condition that the plane is visible from two cameras. In other words, the line of sight that exits the optical center passes through the lens and reaches the plane. Of the n candidates obtained, select the n that makes the inner product with m positive. This reduces the number from 4 to 2. Furthermore, in order to make the solution unique, this embodiment solves this problem by adding a condition that the positional relationship between the two cameras does not change during shooting. Also to find d t

The absolute value of obtained and t can be obtained from the following equation

a

d In other words, d is determined by giving the baseline | t | of two cameras.

Step S 1 3 0> Generate an image (virtual projection plane image) of the road plane viewed from above

Using the road plane attitude parameters obtained in step S 1 2 0, as shown in Fig. 14, a virtual projection plane (VPP: Virtual Projection Plane) parallel to the road plane based on the reference image is used. ) Generate an image. The symbols in Figure 14 are defined as follows.

n _p is the normal vector in the coordinate system of the road plane and the reference camera, f is the focal length of the reference camera, d is the distance between the reference camera optical center and the road plane, and e. _z is the optical axis of the reference camera. '

Here, the normal vector obtained by decomposing the projective transformation matrix in step S 1 2 0! Let it be ^.

First, the optical axis of the reference _camera e _oz ! _E , where the outer product of ^ is the axis of rotation. _{If the} transformation matrix that rotates the angle between _z and n _p by 0 is R ",

[Equation 1 8] c _x a + cosO ο _λ ο ₂ α― c ₃ sin6 c _x c ₇ + c ₂ sin6

c _x c ₂ + c ₃ sin0 c ₂ a + cos0 c ₂ c ₃ a-c _x sin6

c ^ a one c ₂ sin6 c ₂ c, a + c _x sin6 c ₃ ² + cosO

V [Equation 1 9] '

a = (l-cos6)

[Equation 2 0]

^{C = n} _P ^{X e} . z ⁼ ( ^C l, ^C 2, ^C 3) ^r

[Equation 2 1]

It can be expressed. The projective transformation matrix that converts the reference camera into a VPP image rotated by this R "is A ^., The internal parameter matrix of the reference camera, that is, the interior of the virtual camera (that is, the V pp camera). If the parameter matrix is A _vpp ,

[Equation 2 2]

H "= A, R〃A ¹ can be expressed.

Next, the direction of the projection of the vertical axis of the VPP image and the optical axis of the reference camera on the road plane is matched. The unit vector e _0z = (0,0,1 / in the optical axis direction of the reference camera is considered to be a vector u viewed from the VPP camera coordinate system, and is a vector obtained by orthogonally projecting it to the VPP image coordinate system. Consider Toll V.

First,

[Equation 2 3]

Next, _let us consider a vector u such that u = (u _x , u _y , 0) ^T, which is an orthogonal projection of u onto the X — y plane, where u is the _x component and u _y is the u component.

When this U is projected onto the VPP image coordinates, there is nothing on the homogeneous VPP image coordinates. Limited point = (,, ο) Converted to ^Γ .

[Equation 2 4]

= (However, II 11 = 1)

And define 'and call this the camera direction. The rotation matrix R to be obtained is a rotational transformation that matches the camera direction to “V Ρ の一 one V direction of image coordinates” = “V Ρ の一 one y direction of camera coordinate system”.

[Equation 2 5]

(0, -l, 0f = Rv

Find R that satisfies By combining with H ”in the above equation 2 2, the projective transformation matrix H from the reference image to the V P P image is

[Equation 2 6]

H = RH ''

It becomes.

The reference image is subjected to projective transformation using the H in Equation 26 to generate a VPP image. Reference image coordinates << ¾ = O _0i , v ₀ f obtained by projecting a point that can be expressed as m in the three-dimensional space onto the reference image, Dot,.

In between

[Equation 2 7]

= Hm _Qi

Therefore, the luminance of the point m on the VPP image can be obtained by substituting the luminance value of the point m _oi on the corresponding original image using the inverse matrix H- ^{1 of} H. [Equation 2 8]

u [Equation 2 9]

v Two H i + H;

. '· W ¾

However, ^ is a representative of a row i and column j elements of the matrix T ^1.

Using the above projective transformation matrix and calculating the V P P image with the reference image in Fig. 16 as input, Fig. 19 is generated. Note that FIG. 15 is a reference image, and FIG. 18 is the result of superimposing the planar area shown in FIG. 17 on the original image.

<Step S 1 4 0> Calculate the position and direction of the plane area that can be traveled on the V P P image and the vehicle

First, Fig. 20 shows an image obtained by converting only the planar region (white region) into a VPP image as shown in Fig. 17 by the method described in step S 1 30 (in this image). The white area is the flat area).

Next, features of the VPP image will be described. The points on the estimated plane are converted into VPP images with the correct coordinates by the transformation of Equation 27 above, but points floating from the plane (for example, cars and walls) are converted into VPP images. When converted, it is not converted to the correct position. In other words, by converting to a VPP image, an image of the estimated road plane region viewed from above in parallel was generated, and the coordinates obtained from the image are a coordinate system corresponding to real space. .

Next, in order to obtain the optical center dropped on the VPP image, R〃 in the above equation 18 and the internal parameter A of the virtual camera, and the normal vector / letter _{p of the} road plane If v is the homogeneous coordinate of the optical center on the homogeneous VPP image coordinates, and o _vpp = (o o ₂ , o ₃ ) = RA _vpp ITn _p , v PP image The image coordinates of is obtained as 0 „, 0 as (o ^ Ov) ^^^", "^).

° 3 ° 3

O is the point where the optical center of the basic camera is projected onto the road plane. Hereinafter, this point will be the reference camera position origin. Therefore, from the positional relationship between the camera and the vehicle

V Ρ わかる You can see the position of the vehicle on the image. As shown in Fig. 2 1,

V Ρ Ρ The vertical axis Ρ in the image coincides with the optical axis. In other words, if the mounting angle between the optical axis and the vehicle is known, the direction of the vehicle can be calculated in the VPP image.

<Step S 1 5 0> Calculate the distance in each direction to the farthest part from the vehicle in the road plane area on the V P P image, that is, the obstacle.

To determine the area expansion, prepare an image obtained by converting a planar area as shown in Fig. 20 into a VPP image, and tilt it Θ from the reference camera position origin on the image with respect to the optical axis (Y axis). Stretch the straight line and calculate the edge of the planar area (see Figure 22). By obtaining the coordinates of the edge of the plane area from the reference camera position origin O on the image, the spread length of the plane on the image at a position tilted 0 from the optical axis can be obtained. The length on the image can be converted to the actual distance using the internal parameters of the virtual camera and the distance d between the optical center of the reference camera and the road plane. In other words, the distance for each direction of the planar area was calculated by this process.

In this example, due to restrictions such as the resolution of the captured image, the upper limit was calculated as the area expansion from the reference camera position origin to 32 m in the optical axis direction. The measurement range was 25 degrees from the optical axis, with increments of 0.5 degrees.

<Step S 1 6 0> The highest point from the vehicle in the road plane area on the VPP image. Calculate the relative speed in each direction to the far part, that is, the obstacle

Next, the relative speed by direction to the farthest part from the vehicle in the road plane area on the VPP image, that is, to the obstacle is calculated. For each time, calculate the distance for each direction of the road plane area by using the method described in step S 1 50 for each direction.

Here, for example, in order to find the relative velocity of the plane tilted Θ from the optical axis (Y axis), the distance for each direction tilted Θ from the optical axis (Y axis) is used in the past 5 frames. Since the image was taken at 30 fps, 1 frame is 1/30 s, so using the least squares method, the gradient is obtained from the time-series data of distance by direction for each 1/30 s. Thus, the relative speed in that direction could be calculated (see Fig. 23). Therefore, by performing the same process for each direction, the relative velocity of the planar area is calculated for almost the entire planar area.

<Step S 1 7 0> V P P Overlays the plane area that can be driven on the image, and displays the calculated distance by direction and relative speed information by direction.

By the method described in step S 1 5 0 and step S 1 60, the distance and relative speed of the road plane area in each direction can be calculated.

'In the present invention, it has been devised so that the distance by direction and the relative speed by direction of the obtained road plane can be displayed easily. The result is the image shown in FIG. First, the image on the right side of the image shown in Fig. 24 is obtained by superimposing a reference image converted into a VPP image and a flat region converted into a VPP image. In the image on the right side of the image shown in Fig. 24, the light blue area is the combined plane area, and each point at the end of the plane area is the end point of the area ( Each point is a distance according to the direction of the plane, and represents the point indicated by arrow A in the figure. The following rules apply to the coloration of the end points.

First, if the expansion speed of the plane is negative (the plane contracts, that is, the direction toward the camera), there is a danger, so it is displayed as a warm color point. On the other hand, if the plane expansion speed is positive (the plane expands, that is, the direction away from the camera), the risk is low, so it is displayed with a cold color point, and the plane expansion Those without are displayed as green dots. The gradient below the left side of the image shown in Figure 24 shows the transition of the color of the point due to the spread speed of the plane, and each number corresponds to the speed per hour (km / h). . In addition, it is drawn every 5 m concentric arcs (indicated by arrow B 'in the figure) so as to be a constant distance from the reference camera position origin, and the width of the car equipped with the camera is shown in red (Indicated by arrow C in the figure).

Next, the image on the left side of the image shown in Fig. 24 is a composite of the plane area with the reference image, and the plane area is shown in light blue. Each endpoint of the planar area is

By converting the coordinates of the image calculated based on the VPP image using H- ¹ in Equation 27, the coordinates corresponding to the end points of the area in the reference image are calculated. Is displayed (indicated by arrow D in the figure). The method of determining the color used here uses the same rule as the end point of the image on the right side of the image shown in Fig. 24. In addition, the distance from the origin of the reference camera position is displayed in units of meters on a point at a certain angle from the optical axis (every 5 degrees in this example) (indicated by arrow E in the figure). ing) .

FIGS. 25 and 26 show examples of road plane areas and obstacle detection results using the present invention by the above-described method of displaying distance by direction and relative speed information by direction. Figures 25 and 26 show the camera mounted on the car. The result of applying the present invention to the captured stereo moving image is shown. Incidentally, the result is the road plane area and the obstacle detection result in the urban area, and 3 frames (1/10 seconds) from the processing result of the stereo moving image. ) The results for each are extracted and displayed.

As described above, in the road plane area and the obstacle detection method using the stereo image according to the present invention having steps S1100 to S1700, the two stereo cameras mounted on the automobile are used. By using time-series information obtained from the captured stereo video, it is possible to stably estimate the projection transformation matrix and road plane area continuously, and the road surface itself can be tilted or run. The projection transformation matrix is not constant, even if the car body is tilted due to curves or unevenness of the road surface, or the camera mounted on the car vibrates, and the projection transformation matrix is not always constant. Therefore, we were able to solve these problems.

Further, in the present invention, a virtual projection plane image (VPP image) is generated based on the slope of the road surface calculated by decomposing the projective transformation matrix, and the vehicle in the plane area on the VPP image is further generated. The distance from each to the obstacle and the relative speed by direction are calculated. Finally, the plane area that can be traveled is superimposed on the VPP image, and the calculated distance by direction and relative speed information by direction are calculated. The display of the area that can be traveled to the vehicle and the obstacle is a visual display, which is a highly practical method. It is preferable to use a CCD camera as the stereo camera to be used. However, the present invention is not limited to this, and any other means can be used as long as it is a photographing means capable of photographing a stereo moving image. That. In addition, the present invention is applied to a traveling object other than an automobile. It is also possible to apply. Industrial applicability

As described above, according to the road plane region arrangement and the obstacle detection method using the stereo image according to the present invention, since the CCD camera is used as the image sensor which is a stereo image capturing device, the apparatus is inexpensive. It is versatile and has an excellent effect of achieving cost reduction.

According to the road plane area and the obstacle detection method using the stereo image according to the present invention, it is possible to dynamically obtain the area where the vehicle can travel and the distance from the vehicle to the obstacle in each direction stably relative speed. So, based on the required travelable area and the distance from the vehicle to the obstacle in each direction, based on the relative speed, it is possible to provide driver assistance such as warning of collision risk, avoidance of collision, automatic driving of the vehicle, etc. Excellent effect.

Furthermore, in the road plane area and obstacle detection method using stereo images according to the present invention, the camera is very easy to install and calibrate. There is a mistake in something. Reference List>

Non-patent document 1:

Noreon 'Q.—T, Ueno' Weber, J, Coller 'D, Malik' Malik, J, 'Integrity 'Stereo' based 'Approach' Auto-automatic 'Beek Nore' guidance (An Integrated stereo-based approach to automat i G vehicle guidance) ", 1 9 5 5 years, Proc. I." Chi "'C' buoy (Proc. ICCV), D. 5 2-5 7 Non-patent literature 2:

'Onoguchi, K' Takeda, N and Watanabe, M, “Planner. Projection” Stereopsis Method For Low Do Extrusion (Planar

Projection Stereopsis Method for Road Extraction) ”, 1 9 5 5 Procedia. I'R'o'S (Proc. IROS), p

2 4 9-2 5 6 Non-Patent Document 3

“Visual” by Stor Johann, K, Zielke, T, Mallot, H, Vonseelen, W · Obstacle Detection · Fore-automatically 'Guided Vehicles (Visual Obstacle Detection for Automatically Guided Vehicles; 1990), Proc. 'Arnole' A (Proc. ICRA), p. 7 1 6-7 6 6 Non-Patent Document 4:

Xie, M, “Matching 'Free' Stereo 'Vision • Fore' Detecting 'Obstacle' On 'Grand' Plane, Machine 'Vision &End' Application ( Matching free stereo vision for detecting obstacles on a ground plane , Machine Vision and Application), 1 9 96, Volume 9. Number 1 (Vol. 9, No. l), p. 9-13 Non-patent document 5:

Heung-Yeung Shum, Richard Szeliski, "" Non-Ramic "Image" Mosaik Technical "Reho" Panoramic Image Mosaics ", Technical report, 1 997, M. S. Arnolety Arnole 9 7 — 2 3, Microsoft Soft Research (MSR-TR-97-23, Microsoft Research; Patent Document 6:

Co-authored by O. D. Faugeras and F. Lustman, ““ Motion ”and“ Structure ”, From • Motion, Peacewise, Planner, Environment,, ("Motion and Structure from Motion In A Piecewise Planar Environment"), 1 9 98, Procedure. 'Artificial' Intelligence, Volume 1.2, Number 3 (Proc. International Journal of Pattern Recognition and Artificial Intelligence, Vol.2, No.3), p 4 5 8 — 5 0 8

Claims

The scope of the claims

1. All the obstacles that exist on the road surface area and the road surface that can be traveled using only the stereo moving image composed of the standard image and the reference image, which is taken by the imaging means mounted on the car. A road plane region and obstacle detection method using a stereo image that can detect a road surface, which is obtained in a first step and a first step for dynamically estimating a projective transformation matrix for the road surface. Using the projective transformation matrix, a second step of extracting a plane area that can travel;

A third step of calculating the slope of the road surface by decomposing the projective transformation matrix obtained in the first step;

A fourth step of generating a virtual projection plane image in which the road surface is viewed from above based on the inclination of the road surface calculated in the third step;

On the virtual projection plane image generated in the fourth step, a fifth step of calculating the travelable plane area and the position and direction of the vehicle,

A road plane area and an obstacle detection method using a stereo image characterized by comprising:

2. a sixth step of calculating a direction-specific distance from the own vehicle to the obstacle from the plane area where the vehicle can travel on the virtual projection plane image;

A seventh step of calculating a relative speed for each direction from the vehicle to the obstacle from the plane area where the vehicle can travel on the virtual projection plane image;

The road surface area and obstacle detection method using the stereo image according to claim 1, further comprising:

3. The virtual plane image is displayed by superimposing the travelable plane area. And an eighth step of displaying the direction-specific distance and the direction-specific relative velocity information;

The road surface area and obstacle detection method using the stereo image according to claim 2, further comprising:

4. In the first step, the base image and the reference image are subjected to a LOG filter process and a histogram flattening process, and then a region-based method is used to deal with the road surface between the stereo images. The projection transformation matrix to be estimated dynamically,

In the second step, the reference image is projectively transformed by the projective transformation matrix, a difference image between the reference image and the projective transformed reference image is obtained, a smoothing filter is applied to the difference image, A binary image is obtained by binarization using a threshold, and a plane region that can be traveled is extracted by using the estimation result of the previous time plane region and taking a textureless region into the binary image. In fact,

In the third step, the road plane attitude parameter indicating the inclination of the road surface is a distance from the reference camera optical center to the road plane and a normal vector of the road plane,

4. The stereo image according to claim 3, wherein in the fourth step, the reference image is projectively transformed using the road plane attitude parameter to generate a virtual projection plane image parallel to the road plane. Road plane area and obstacle detection method using