JP3279610B2 - Environment recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an environment recognizing device, and more particularly to an environment recognizing device suitable for recognizing an obstacle ahead of a traveling road in an autonomous vehicle or the like.

[0002]

2. Description of the Related Art In traveling control of an autonomous traveling vehicle, it is important to accurately recognize a situation ahead of a traveling road. In particular, an obstacle existing ahead of the traveling path must be recognized as soon as possible, and appropriate traveling control must be performed to avoid a collision. In order to perform such environment recognition, various methods have been conventionally proposed. For example, JP
Japanese Patent Application Laid-Open No. 4-26913 discloses a control device for an autonomous vehicle that arranges two cameras on the left and right sides, captures a stereo image of an obstacle in front of a traveling road, and performs appropriate control based on the stereo image. I have.

[0003]

However, the conventional environment recognition apparatus based on stereo images has a problem that the amount of calculation for image data is enormous and it takes time to recognize an obstacle. In other words, while the actual obstacle is a three-dimensional object, the stereo image used as the material for analysis is a two-dimensional projection image. Processing must be performed. Such processing requires an enormous amount of calculation, and it is difficult to recognize an obstacle or the like in a short time.

Accordingly, an object of the present invention is to provide an environment recognizing device capable of performing environment recognition in a short time by using simpler operations.

[0005]

[Means for Solving the Problems]

(1) The first invention of the present application, in an environment recognition device, a stereo image input unit having a right imaging device and a left imaging device to obtain a two-dimensional stereo image including a right image and a left image of a recognition target. A conversion information storage means for storing conversion information between the left and right images determined based on the relationship between the orientation and the position of the left and right imaging devices, and a predetermined reference plane defined; a formula indicating this reference plane; Image conversion means for performing at least one of right image conversion for creating a virtual left image corresponding to the right image, and left image conversion for creating a virtual right image corresponding to the left image, based on the conversion information between For at least one of the combination of the right image and the virtual right image and the combination of the left image and the virtual left image, an image difference unit for creating a difference image and a plurality of different reference planes are defined. The plurality of difference images obtained by the definition are compared with each other, a reference plane from which a difference image closest to a preset reference pattern is obtained is selected, and at the height position of the selected reference plane, And an obstacle recognition device that recognizes an obstacle corresponding to the difference image.

(2) A second invention of the present application provides a stereoscopic image having a right image pickup device and a left image pickup device in an environment recognition device in order to obtain a two-dimensional stereo image composed of a right image and a left image of a recognition target. Input means, conversion information storage means for storing conversion information between the left and right images determined based on the relationship between the orientation and position of the left and right imaging devices, a predetermined reference plane is defined, and an expression indicating this reference plane An image that performs at least one of right image conversion for creating a virtual left image corresponding to a right image and left image conversion for creating a virtual right image corresponding to a left image based on conversion information between left and right images. A plurality of conversion means, a combination of a right image and a virtual right image, and a combination of at least one of a combination of a left image and a virtual left image; Recognize the mutual change of a plurality of difference images obtained by defining the reference plane, and if the mutual change is based on a change pattern set in advance, the obtained difference image is regarded as an obstacle. And an obstacle recognition device for recognition.

(3) The third invention of the present application is the environment recognition device according to the first or second invention, wherein different reference planes are respectively defined, and a plurality of images each of which can perform independent image conversion processing. A conversion device, constituting an image conversion means, for each virtual image converted by the plurality of image conversion devices, a plurality of image difference devices to create a difference image independently, constituting an image difference means, Each difference image for a plurality of different reference planes is
Each of them can be calculated and calculated at the same time. (4) The fourth invention of the present application is the environment recognition device according to the above-described first to third inventions, wherein the stereo image input means has a function of continuously capturing the right image and the left image as time-series data. Image conversion means obtains a time-series difference image indicating a difference between images given before and after in time, and a predetermined target area including a temporal change based on the time-series difference image. The image conversion process is performed only for this target area. (5) The fifth invention of the present application is the environment recognition device, wherein a stereo image input unit having a right imaging device and a left imaging device is provided to obtain a two-dimensional stereo image including a right image and a left image of the recognition target. a conversion information storage means for storing conversion information between the left and right images determined based on the left and right image pickup apparatus mutual orientation and position of the relationship, the predetermined reference
An image performing at least one of right image conversion for creating a virtual left image corresponding to a right image and left image conversion for creating a virtual right image corresponding to a left image based on an expression indicating a plane and conversion information. Conversion means, image difference means for creating a difference image for at least one of a combination of a right image and a virtual right image, and a combination of a left image and a virtual left image, and recognition of a partial feature on a traveling road surface Then right
Defining three points P1, P2, P3 in the position indicating characteristics of the real image formed of any hand in the image and the left image,
Conversion information for these three points without defining a reference plane
By performing image conversion processing based on the distribution, the real image
3 points P1, P2, P3 on the virtual image on the left and right opposite sides
Find the epipolar lines Q1, Q2, Q3 corresponding to
By utilizing this feature, epipolar lines Q1, Q2, Q
3, the corresponding points P for the three points P1, P2, and P3
1 ', P2', P3 'are determined, and a plane determined based on the three points P1, P2, P3 and their corresponding points P1', P2 ', P3' is defined as a reference plane corresponding to the traveling road surface.
Reference plane defining means for performing processing of giving a reference plane to image conversion means
Step and image conversion means indicating a reference plane corresponding to the road surface
The difference image obtained when performing image conversion using the equation
Provided, the obstacle recognition device for recognizing the obstacle on the basis
That's what I did. (6) The sixth invention of the present application is the environment recognition device according to the fifth invention described above, wherein the reference plane defining means comprises three points P1, P
2, P3 and a plane determined based on the corresponding points P1 ', P2', P3 'are used as temporary reference planes, and in an environment where no obstacle exists, image conversion using the temporary reference plane is performed.
The virtual image and the real image obtained by the means are compared, and a correction is made to the temporary reference plane until they match, and the corrected reference plane is defined as a reference plane corresponding to the traveling road surface . The processing is performed. (7) The seventh invention of the present application is directed to an environment recognition device, wherein a stereo image input unit having a right imaging device and a left imaging device is provided to obtain a two-dimensional stereo image including a right image and a left image of a recognition target. a conversion information storage means for storing conversion information between the left and right images determined based on the left and right image pickup apparatus mutual orientation and position of the relationship, the predetermined reference
An image performing at least one of right image conversion for creating a virtual left image corresponding to a right image and left image conversion for creating a virtual right image corresponding to a left image based on an expression indicating a plane and conversion information. A conversion unit, an image difference unit that creates a difference image for at least one of a combination of the right image and the virtual right image, and a combination of the left image and the virtual left image, and the irradiation point on the road surface has a high luminance. Means for irradiating the traveling road surface with a light beam so that it can be recognized as a point;
Image conversion or left drawing so that the irradiation point on the image matches
Make sure that the irradiation point on the image matches the irradiation point on the virtual left image
On a reference plane that allows for
Is defined as a reference plane corresponding to the road surface, and this reference plane is
A reference plane defining means for performing processing to be given to the image converting means, and an image
The conversion means uses the formula showing the reference plane corresponding to the traveling road surface
Based on the difference image obtained when performing image conversion,
And the obstacle recognition device for recognizing the harm was so provided
It is a thing. (8) The eighth invention of the present application is the environment recognition device, wherein a stereo image input means having a right imaging device and a left imaging device is provided to obtain a two-dimensional stereo image including a right image and a left image of the recognition target. a conversion information storage means for storing conversion information between the left and right images determined based on the left and right image pickup apparatus mutual orientation and position of the relationship, the predetermined reference
An image performing at least one of right image conversion for creating a virtual left image corresponding to a right image and left image conversion for creating a virtual right image corresponding to a left image based on an expression indicating a plane and conversion information. A conversion unit, an image difference unit that creates a difference image for at least one of a combination of the right image and the virtual right image, and a combination of the left image and the virtual left image, and the irradiation point on the road surface has a high luminance. Means for irradiating the traveling road surface with a light beam at a predetermined irradiation angle so that it can be recognized as a point; and means for determining the distance between the vehicle and the irradiation point by receiving the light beam reflected from the irradiation point. Based on the position of the irradiation point on the right image or the left image , the irradiation angle, and the distance, a reference plane corresponding to the traveling road surface is defined, and this reference plane is used as image conversion means.
A reference plane defining means for performing processing to provide, image conversion means run
Image conversion is performed using the expression indicating the reference plane corresponding to the road surface.
Obstacle recognition based on the difference image obtained when
And an obstacle recognition device for performing the operation.
You.

[0008]

[Operation] When a stereo image composed of a right image and a left image is obtained, a relational expression between the right image and the left image is derived from a mutual positional relationship between the two cameras used to capture the stereo image. be able to. Therefore, based on the right image, the corresponding left image can be predicted,
Conversely, based on the left image, the corresponding right image can be predicted. In other words, the right image can be converted to the virtual left image and the left image can be converted to the virtual right image based on the derived relational expression. However, it is necessary to define some reference plane as a condition for performing such conversion. That is, this conversion can be performed only on the assumption that each point on the two-dimensional image is a point on a predetermined reference plane in the three-dimensional space.

The present invention makes use of this property, and its basic principle is as follows. That is, first, it is assumed that there are no obstacles in front of the traveling road,
It is assumed that the environment in front is only a flat road surface. In this case, all points (pixels) on the stereo images captured by the two cameras can be considered to be points on a flat road surface. That is, a flat road surface can be defined as a reference surface. Then, by giving the condition of this reference plane to a relational expression derived from the mutual positional relationship between the two cameras, the right image can be converted to a virtual left image, and the left image can be converted to a virtual right image. it can. Then, theoretically, the right image should match the virtual right image, and the left image should match the virtual left image. If a mismatching portion (that is, an image appearing on the difference image) occurs between these, the point on the stereoscopic image is all points on a flat road surface. That is, the assumption was incorrect. In other words, the mismatched portion is not a point on the road surface serving as the reference surface. After all, the image indicating the mismatched portion is a set of points that are not on the road surface serving as the reference surface, and indicates an image of the obstacle. This is the basic principle of the present invention.
To perform recognition processing based on this basic principle, it is only necessary to perform a coordinate conversion operation based on a conversion formula and a comparison operation for comparing two images. Therefore, environment recognition can be performed in a short time using a simple operation. Will be able to do it. The basic principle is disclosed in Japanese Patent Application No. Hei 3-91168.

A feature of the first invention of the present application resides in that recognition of an obstacle height is added to recognition of an obstacle based on the basic principle. As mentioned above, in the basic principle of the present invention,
In order to calculate a virtual image, it is necessary to define a predetermined reference plane. The inventor of the present application pays attention to the fact that the shape of the obtained mismatched portion changes depending on the relationship between the position of the reference plane and the position of the actual obstacle, and defines a plurality of reference planes to define the obstacle. The height was recognized. That is, when a reference plane is defined at the actual height position of the obstacle, what kind of shape mismatching portion is obtained is prepared in advance as a reference pattern. Then, a plurality of reference planes having different heights are defined, and the unmatched portions are respectively calculated and obtained. If a reference plane whose shape of the mismatched part is closest to the reference pattern is selected, the selected reference plane is determined. The position of the obstacle can be recognized from the position.

The second invention of the present application makes it possible to more reliably detect an obstacle by applying the first invention. As described above, the shape of the obtained mismatched portion changes depending on the position of the reference plane to be defined. Therefore,
If the shape change of the mismatched portion is known in advance when a true obstacle is present, the recognized obstacle is determined based on the shape change of the mismatched portion actually obtained. It is possible to determine whether the thing is true or due to misrecognition. Application
In the invention of the third aspect, simultaneous operations by a plurality of image conversion devices are performed.
Will be In the fourth invention of the present application, a portion including a temporal change
The image conversion processing is performed only on the image data. In addition, the fifth of the present application
In the invention of the present invention, the road surface determination using the epipolar line is performed.
According to the sixth aspect of the present invention, more accurate road surface determination is possible.
become. Further, in the seventh and eighth inventions of the present application, on the road surface
The road surface is determined by beam irradiation.

[0012]

[Example] §1. Principle of Stereo Image Conversion Hereinafter, the present invention will be described in detail with reference to an illustrated embodiment. First, the principle of mutual conversion between left and right images will be briefly described.
Now, as shown in FIG. 1, a case where a predetermined object 10 is imaged by a right camera 11 and a left camera 12 to obtain a stereo image will be considered. In this case, as shown in FIG. 2, the right camera 11 captures the object 10 in a three-dimensional right coordinate system RR,
The left camera 12 captures the same object 10 in the three-dimensional left coordinate system LL. The three-dimensional right coordinate system RR is a coordinate axis XR
, YL and ZR, and the three-dimensional left coordinate system LL is another three-dimensional coordinate system represented by coordinate axes XL, YL and ZL. And between the two coordinate systems,
It is known that the following conversion formula holds.

[0013]

(Equation 1) Here, a matrix of nine coefficients R with subscripts is called a rotation matrix, and the right camera 1
The matrix is determined based on the correlation between the direction of the camera 1 and the direction of the left camera 12, and a matrix of three coefficients T with subscripts is called a translation matrix, and the positional deviation between the right camera 11 and the left camera 12 Are determined based on the correlation of

When capturing a stereo image, the right camera 1
1 captures the object 10 on the coordinate system RR to generate a right image PR, and the left camera 12 captures the object 10 on the coordinate system LL to generate a left image PL. As shown in FIG. 3, the right image PR is a two-dimensional image defined on a two-dimensional right coordinate system R expressed by two coordinate axes xR and yR, and the left image PL
Is a two-dimensional image defined on a two-dimensional left coordinate system L represented by two coordinate axes xL and yL. The projection image 10R obtained on the coordinate system R is an image obtained by projecting the object 10 on the coordinate system RR onto a predetermined plane, and the projection image 10L obtained on the coordinate system L has the coordinates It is an image obtained by projecting the object 10 on the system LL onto a predetermined plane.

Here, the relationship between the three-dimensional coordinate system and the two-dimensional coordinate system will be considered. For example, as shown in FIG. 4, a three-dimensional coordinate system XYZ is located on the right half of a point O indicating the lens position of the camera.
Is defined. The axis directions of each coordinate system are as shown in the figure.
The Y axis is a direction perpendicular to the paper surface. Then, a plane perpendicular to the Z axis is defined at a position at a distance f to the left of the point O,
A two-dimensional coordinate system xy is defined on this plane. The y-axis is also in the direction perpendicular to the paper. In this model, a projection image q for one point Q in the coordinate system XYZ is converted to a coordinate system xy
Think about getting on. That is, based on the coordinate value Q (XQ, YQ, ZQ) represented by the coordinate system XYZ, the coordinate value q (xq, yq) represented by the coordinate system xy is obtained. In this case, it can be understood from FIG. 4 that a conversion equation of xQ = f (XQ / ZQ) is obtained. Similarly, a conversion formula of yQ = f (YQ / ZQ) is obtained. Here, f is a constant determined by the optical system of the camera. From this, the following conversion formula is obtained between the coordinate systems RR and LL shown in FIG. 2 and the coordinate systems R and L shown in FIG.

XR = fR (XR / ZR) yR = fR (YR / ZR) xL = fL (XL / ZL) yL = fL (YL / ZL) The relationship between the three-dimensional coordinate system RR and LL is described above. Is obtained by a conversion formula using a coefficient matrix,
The relationship between the three-dimensional coordinate system RR and the two-dimensional coordinate system R and the relationship between the three-dimensional coordinate system RR and the two-dimensional coordinate system R are obtained by the above-described conversion formulas. Therefore, if these conversion formulas are used, a conversion formula indicating the relationship between the two-dimensional coordinate system R and the two-dimensional coordinate system L should be obtained. However, this conversion equation cannot be solved only by the above assumptions. In addition, it is necessary to define another predetermined reference plane. The reason can be easily understood by considering a model as shown in FIG. That is, when the projection point P is obtained on the xy plane in FIG. 5, where the actual point corresponding to the projection point P is located in the three-dimensional space can be determined from this information. Can not. However, assuming that a predetermined reference plane is determined in the three-dimensional space and that the actual point is a point on the reference plane, the position of the actual point in the three-dimensional space is determined. For example, in the example of FIG.
Once the reference plane S1 is determined, the actual position of the point P1 is determined,
Once the reference plane S2 is determined, the actual position of the point P2 is determined.

Therefore, a predetermined reference plane is given in the form of a plane general formula aX + bY + cZ = 1. Here, a, b, and c are constants for determining the reference plane. The equation of such a reference plane is defined and the point (XR, YR, ZR) in the three-dimensional right coordinate system is defined.
If the condition that the points (XL, YL, ZL) in the three-dimensional left coordinate are all points on this reference plane is added, the relationship between the two-dimensional coordinate system R and the two-dimensional coordinate system L is shown. The conversion equation can be solved as follows.

XL = (AxR + ByR + C) / (DxR + EyR + 1) yL = (FxR + GyR + H) / (DxR + EyR + 1) Here, A to H are predetermined constants. Using this conversion formula, for any point (xR, yR) in the right image,
The corresponding point (xL, yL) in the left image can be obtained by calculation, and vice versa.

§2. The relationship between the reference plane and the difference image Now, as shown in FIG. 6, consider a case where an obstacle N exists in front of the traveling road when the vehicle M is traveling on the road surface S0. When the right camera 11 and the left camera 12 are mounted on the vehicle M and an obstacle N in front is captured, the left image and the right image are as shown in FIGS. 7A and 7B. Therefore, the road surface S0
It is assumed that image conversion for obtaining a virtual right image from a left image has been performed with reference to. If the obstacle N does not exist, the virtual right image should match the actual right image.
That is, if the obstacle N does not exist, all points on the left image and all points on the right image are points existing on the road surface S0. Should theoretically match the true image. However, if the obstacle N is present, the point on the obstacle N is not on the road surface S0. Therefore, in the image conversion using the road surface S0 as the reference plane, a mismatch occurs. Specifically, a virtual right image obtained by image conversion based on the left image as shown in FIG. 7A is distorted as shown in FIG. Does not completely match the true right image shown in FIG. Therefore, in order to extract a mismatched part, a difference between the right image shown in FIG. 7B and the virtual right image shown in FIG. 7C is obtained, and a difference image shown in FIG. 7D is created. The image remaining on the difference image corresponds to the mismatched portion, and corresponds to a set of points that are not on the road surface S0, that is, the obstacle N.

As described above, the virtual image obtained based on one of the stereo images, the other image of the stereo image,
The basic principle of obstacle recognition in the present invention is to take the difference between the two and to recognize the obtained image on the difference image as an obstacle. Generally, when an obstacle N having a certain height is present on the road surface S0, an image having an inverted triangular shape as shown in FIG. 7D is obtained on the difference image.

Now, consider a case where the obstacle N is not on the road surface S0. For example, let us consider a case where an obstacle N exists at a height H from the road surface S0 as shown in FIG. Now, five reference planes S0, S1, S2, S3,
S4 is defined at the position shown in FIG. The reference plane S0 is a road surface, and the reference planes S1 to S4 each have a height h1 from the road surface.
To h4, and height h2 = H. When a difference image is obtained for each of the five reference planes, images as shown in FIGS. 9A to 9E are obtained. Here, in the difference image of FIG.
It can be seen that an image having an inverted triangular shape is formed similarly to the difference image shown in FIG. That is, when the reference plane is defined at the same level as the height position of the obstacle N, an image having an inverted triangular shape is formed on the difference image. Regardless of whether the level of the reference plane is shifted upward or downward from this position, the image changes to a shape in which two triangles are overlapped. The present invention focuses on this property and attempts to recognize the height position of the obstacle N. That is, difference images are obtained for several different reference planes, and an image in this difference image that is closest to an inverted triangle as shown in FIG. 9C is selected, and the height h2 of the reference plane at this time is determined. This is recognized as the height H of the obstacle N.

When the reference plane coincides with the height position of the obstacle N, the shape of an image obtained as a difference image is not always an inverted triangle. The shape of the image on the difference image varies depending on the shape of the obstacle N. Therefore,
Strictly speaking, a predetermined reference pattern is set in advance based on the predicted shape of the obstacle, and a reference plane on which an image closest to the reference pattern is obtained as a difference image is set at the height position of the obstacle. Will be selected as the reference plane indicating. However, in the case of a general obstacle having a certain height in the vertical direction, the reference pattern is almost an inverted triangle, so in practice, a difference image in which the obtained image is closest to the inverted triangle is selected. No problem to do so.

§3. Configuration of environment recognition device according to the present invention
And operation Subsequently, the configuration and operation of the environment recognizing apparatus according to the present invention will be described with reference to a block diagram shown in FIG. 10. The stereo image input means 100 includes a right camera 11 and a left camera 12 for performing stereo shooting as shown in FIG. 1, and obtains a right image PR and a left image PL. In this embodiment, since this environment recognition device is mounted on an autonomous vehicle and used, the right camera 11 and the left camera 12 are attached to the left and right positions at the front of the vehicle body. Conversion information storage means 1
Reference numeral 10 stores information necessary for performing the mutual conversion between the two-dimensional coordinate system R and the two-dimensional coordinate system L described above.
As described above, this information is uniquely determined based on the relationship between the orientation and the position of the right camera 11 and the left camera 12. The image conversion means 200 is composed of a plurality of image conversion devices 210 to 214, and the conversion information storage means 1
Based on the conversion information in 10 and the formula of the predetermined reference plane,
Virtual right images IR0 to IR4 corresponding to the left image PL are created. As described above, in order to create such a virtual image, it is necessary to define a predetermined reference plane. Different reference planes are set in the image converters 210 to 214, respectively, and the created virtual right images IR0 to IR4 are images obtained based on the different reference planes.

On the other hand, the image difference means 300 is composed of a plurality of image difference devices 310 to 314, and the virtual right images IR0 to IR0 created by the image conversion devices 210 to 214.
The difference images DR0 to DR4 between IR4 and the right image PR obtained from the right camera 11 are output. These difference images DR0-
DR4 is input to the obstacle recognition device 400.

The operation of this device is as follows. Now, for example, it is assumed that reference planes S0 to S4 shown in FIG. 8 are set in the image conversion apparatuses 210 to 214, respectively. In this case, the difference images DR0 to DR4 output from the image difference devices 310 to 314 are as shown in FIGS. 9 (a) to 9 (e). The obstacle recognition device 400 selects, from these difference images DR0 to DR4, a predetermined reference pattern, that is, an image including an image most similar to an inverted triangle pattern. In this example, the difference image DR2 as shown in FIG. 9C is selected, and the corresponding reference plane S2 is selected as the reference plane indicating the height of the obstacle. Thus, the obstacle recognizing device 400 recognizes the image included in the difference image DR2 as an obstacle, and places the obstacle at the position of the height h2 (the height from the road surface S0 to the reference surface S2) from the road surface. It can be recognized that it exists.

As described above, the first function of this device is that the height position of the obstacle can be recognized together with the recognition of the obstacle. In the recognition of the height position by such a method, an image conversion process performed in order to obtain a virtual image in the image conversion means 200 has the largest computational burden. In the apparatus of the present embodiment, the image conversion means 20
0 is constituted by a plurality of image conversion devices 210 to 214, and each of them can independently execute an image conversion process based on a unique reference plane, so that a plurality of image conversion processes can be performed simultaneously as a parallel process. . For this reason,
The calculation time can be reduced.

The second function of this device is to improve the accuracy of recognition of obstacles. Theoretically, it should be possible to reliably recognize an obstacle by the above-described process. However, in practice, erroneous recognition may be performed due to various factors. That is, an object that is not a true obstacle (for example, dirt on a road surface or a puddle) may be recognized as an obstacle. In this device, such an erroneous recognition is reduced by the obstacle recognition device 400. The principle is as follows. That is, when a true obstacle N exists, a plurality of difference images obtained by defining a plurality of different reference planes are based on the position of the reference plane.
Mutually change as shown in FIG. In other words, when a difference image showing a mutual change as shown in FIG. 9 is obtained, it can be said that a true obstacle exists. The obstacle recognition device 400 recognizes a mutual change of the difference images DR0 to DR4 given from the image difference devices 310 to 314, and obtains the mutual change if the mutual change is based on a preset change pattern. The difference image is recognized as an obstacle. Specifically, by changing the reference plane, the difference image is changed from a pattern in which two triangles are overlapped as shown in FIGS. 9A and 9B to an inverted triangle as shown in FIG. 9C. If the pattern changes to a pattern in which two triangles are superimposed as shown in FIGS. 9D and 9E, it is determined that a true obstacle has been recognized.
Conversely, if no such pattern change is seen,
It can be determined that the obtained image of the obstacle is due to erroneous recognition.

In each of the embodiments described above, a virtual right image is created by converting an actual left image, and a difference image is obtained by calculating a difference between the virtual right image and the actual right image. However, on the contrary, a virtual left image is created by converting an actual right image, and a difference image is obtained by taking a difference between the virtual left image and the actual left image. It does not matter, or both may be performed.

§4. In the device shown in FIG. 10, the image conversion processing in the image conversion means 200 and the difference calculation processing in the image difference means 300 are processings that handle a large number of pixels.
This is a process in which the higher the resolution of the image, the greater the computational burden. Therefore, here, a method for making these image operations more efficient will be disclosed. The right image PR and the left image PL input by the stereo image input means 100 change moment by moment, and are synchronized with each other at regular intervals, respectively, and the image conversion means 200
And the image difference means 300. In other words, the right image PR and the left image PL are given as time-series data. The technique shown here is intended to reduce the overall calculation load by omitting the processing of the same part of the image to be processed at the present time as the previously processed image.

Then, according to the specific example shown in FIG.
This technique will be described. The four images shown in the upper part of FIG. 11 are time-series image data of the left image PL obtained from the left camera 12. That is, the left images are images PL1 to PL4
Has changed over time. In the image PL1, only the distant vehicle 1 is present as an obstacle, but in the images PL2 to PL4, a new vehicle 2 enters the vicinity,
Moving left and right. Now, taking these images PL1 to PL4 as original images and taking difference images between adjacent time-series data, four images shown in the lower part of FIG. 11 are obtained (the difference images here are: An image corresponding to the difference between the current image and the previously obtained image;
0 does not indicate the difference between the left and right images obtained at 0). That is, since the preceding image is not defined for the image PL1, the difference image PL1 'cannot be obtained, but for the next image PL2, the difference from the preceding image PL1 is obtained. An image PL2 'is obtained.
Similarly, a difference image P3 is calculated as a difference between the image PL3 and the image PL2.
L3 'is obtained, and a difference image PL4' is obtained as a difference between the image PL4 and the image PL3. In the difference image PL2 ', the vehicle 2 appears as an image as it is, but in the difference image PL3', parts 2a and 2b of the vehicle 2 are shown as images, and in the difference image PL4 ', parts 2c and 2d of the vehicle 2 are shown as images. Is appearing. These difference images indicate temporal changes in the original image. Therefore, the vehicle 1 that does not appear in the difference image is an obstacle that is at a fixed position in time, and it is not necessary to repeat the same calculation for each time. That is, for the images PL2 to PL4, the image calculation process needs to be performed only for the vehicle 2 appearing in the difference image.

In order to perform efficient image processing based on the above concept, the following operation may be performed in the apparatus shown in FIG. Here, for convenience of explanation, the operation in the image conversion device 210 and the image difference device 310 will be described as an example. First, a frame memory capable of storing data of a plurality of images (at least two images) is prepared in the image conversion device 210. Therefore, the left images PL1 to PL4 given momentarily are
It can be temporarily stored as data. In addition, the image conversion device 210 is provided with a function of performing a difference on the time-series image data and determining a target area based on the obtained difference image. For example, when the difference image PL2 'is obtained based on the left images PL1 and PL2, the closed region including the image of the vehicle 2 in the difference image PL2' is determined as the target region. If a rectangular area including the image of the vehicle 2 is taken as the target area, the target area AL shown in FIG. 12 can be determined for the left image PL2. Therefore, the image conversion device 210 performs the image conversion process on the entire region for the left image PL1 and outputs the virtual right image IR0, but performs the image conversion process only on the target region AL for the left image PL2. become. A target area AR corresponding to the target area AL defined on the left image PL2 is obtained on the right image PR2. Image difference device 31
The left / right difference calculation at 0 is performed only within this target area. Obstacles for the remaining area (vehicle 1)
As the information about, the result based on the left image PL1 and the right image PR1 obtained last time may be used as it is.

In the difference images PL3 'and PL4',
The image of the vehicle 2, which is a single obstacle, has been separated into portions 2a and 2b and portions 2c and 2d. These separated images can be recognized as being part of a single obstacle by referring to the previous difference image. For example, portions 2a and 2b in the difference image PL3 '
Are in contact with the image of the vehicle 2 in the difference image PL2 '. Therefore, the portions 2a and 2b can be recognized as being a part of a single obstacle, and a rectangular region including both the portions 2a and 2b may be defined as the target region. Similarly, the difference image PL4 ′
Are the difference images PL3 '.
2a, 2b in (recognized as part of vehicle 2)
It is located in contact with. Therefore, the portions 2c and 2d can be recognized as being a part of a single obstacle, and a rectangular region including both the portions 2c and 2d is defined as the target region. .

By employing such a method, it is possible to perform a calculation for an obstacle having a large moving amount at a high speed, and it is possible to perform a very efficient image calculation. That is, in the example shown in FIG. 11, the calculation for the vehicle 2 having a large moving amount is performed more frequently than the vehicle 1 having a small moving amount, and the calculation is performed at a high speed. In the above embodiment, the difference image of the time-series image data is
Although the difference is obtained by calculating a difference between the previous image and the previous image, the difference may be obtained by weighting the previous image and the image before the previous image, respectively.

§5. Travel Road Surface Recognition Process I In the environment recognition device according to the present invention, a travel road surface or a surface at a predetermined height from the travel road surface is used as a reference surface.
Therefore, a process of recognizing the traveling road surface, more specifically, a process of obtaining an equation of a plane of the traveling road surface in a coordinate system defined in the device is required. Therefore, an example of the recognition processing of the traveling road surface will be described below.

In the apparatus shown in FIG. 10, information necessary for recognizing the traveling road surface includes a right image PR obtained from the right camera 11 and a left image P obtained from the left camera 12.
L. Here, a method for recognizing the traveling road surface based on the two images on the left and right will be described. Now, Figure 1
As shown in FIG. 3 (a), two unique points P on the left image PL
1. Determine P2. This may be determined, for example, as a predetermined point on a white line W that can be easily identified on the road surface. Next, two points on the virtual right image corresponding to the two points P1 and P2 are obtained by image conversion. However, since the road surface has not been recognized yet, no reference plane is defined. As described above, if an attempt is made to obtain a virtual right image from a left image without defining a reference plane, one point on the left image is not obtained as one point on the virtual right image, and one line is obtained. (Epipolar line). Therefore, the projection of the two points P1 and P2 on the left image PL onto the right image PR by image conversion becomes epipolar lines Q1 and Q2, respectively, as shown in FIG. 2 points P1, P2
Are the corresponding points P1 'and P2' on the lines Q1 and Q2, respectively. These true corresponding points P1 'and P2' can be easily recognized on the right image PR. That is, it can be recognized as the intersection of the white line W and the lines Q1 and Q2. More specifically, pixels on the lines Q1 and Q2 are searched, a pixel having a pixel value corresponding to the white line W is found, and the positions of these pixels may be set as true corresponding points P1 'and P2'.

By the above procedure, two corresponding points on the traveling road surface are plotted on the left image PL and the right image PR. In order to determine the plane, at least three points are needed, so for the third point on the traveling road surface,
It is necessary to find corresponding points in the left image PL and the right image PR, respectively. Therefore, first, as shown in FIG.
A third point P3 is determined on the left image PL, and a line segment S including this point P3 is defined. The point P3 may be anywhere, as long as it is not on a straight line connecting the points P1 and P2. However, it is preferable that the point P3 be located at a position where a pixel on the line segment S has a characteristic pixel value. Subsequently, the point P3 on the left image PL is converted into a right image PR. As described above, if the reference plane is not determined, the point corresponding to the point P3 is not a point but an epipolar line. That is, as shown in FIG. 14B, the epipolar line Q3 is obtained. The true corresponding point P3 'for the point P3 is one point on the line Q3.
Therefore, in order to determine the point P3 ', the line segment S on the left image PL is moved along the epipolar line Q3, and the pixels on the line segment S and the pixels on the epipolar line Q3 are correlated. The point P3 'is determined at the position where the correlation becomes highest. That is, as shown in FIG. 15, each pixel on the line Q3 is compared with each pixel on the line segment S, and the line segment S is fixed at a position where a comparison is obtained such that the pixel value is most approximated. The position of the center pixel at that time is determined as the position of the point P3 '.

Thus, the three corresponding points on the road surface are plotted on the left image PL and the right image PR. Therefore, the equation of the traveling road surface can be determined from these three points. In order to determine the traveling road surface with higher accuracy, it is preferable to perform the following error correction. That is, once the equation of the traveling road surface can be determined by the above-described procedure, the left image PL in a state where there is no obstacle is image-converted using the plane corresponding to this equation as a reference plane to obtain a virtual right image. View. If the correct road surface equation is obtained, the virtual right image obtained in this way is the right camera 1 in the absence of an obstacle.
It should match the real right image obtained from 1. Here, in the case where a mismatch has occurred, it is considered that the equation of the obtained traveling road surface includes an error. Therefore, two points P1,
With the line connecting P2 as the rotation axis, the rotation correction for slightly rotating the obtained traveling road surface is performed. In this way, if the correction is repeatedly performed until the virtual right image matches the real right image, an equation of the road surface with no error is finally obtained.

§6. Travel Road Surface Recognition Processing II The above-described travel road surface recognition processing is for determining a travel road surface based on an image obtained from the stereo image input unit 100 shown in FIG. Here, another method of performing the recognition processing of the traveling road surface using the laser light will be described. This method requires a measurement system as shown in FIG.
The laser beam created by the laser oscillator 500 is modulated by the modulator 510, and the half mirror 52
1,522,523 and total reflection mirrors 524,525
Is reflected on the traveling road surface as four beams. Although not shown in the drawing, it is preferable to provide an irradiation angle adjusting mechanism for each beam.

The laser beam reflected from the traveling road surface is received by the light receiving element 530, and the detection signal is given to the distance calculation device 540. The laser beam reflected from the traveling road surface is transmitted to the stereo image input means 1.
00 (ie, right camera 11 and left camera 12). As a result, as shown in FIGS. 17A and 17B, the irradiation points of the four laser beams on the traveling road surface can be recognized as high-luminance pixels in the left image PL and the right image PR. If the beam is modulated by the modulator 510, the laser beam can be identified with respect to other light sources. That is, among the high-luminance pixels on each image, only the pixels whose luminance changes in a predetermined modulation pattern can be recognized as the pixels corresponding to the irradiation point of the laser beam.

If such a measurement system is prepared, the traveling road surface can be determined by two methods. The first method is a method using left and right images as shown in FIG. That is, since the four points on the images in FIGS. 17A and 17B all correspond to the same point on the actual traveling road surface, when the four points on one image are image-converted, If the reference plane that matches the four points of the image is calculated by the road plane calculation device 550, the reference plane is determined as the traveling road surface. Since the plane can be determined if there are three points, only three laser beams may be used.

The second method is to use the distance obtained by the distance calculation device 540. Distance calculation device 54
0 calculates the distance between the irradiation point on the road surface and the vehicle based on the laser beam that travels back and forth between the two, and gives the distance to the road plane calculation device 550. The road plane calculation device 550 is:
Since the distance to each irradiation point is given in addition to the image of the irradiation point as shown in FIG. 17, the formula of the traveling road surface can be calculated based on these information and the information on the irradiation angle of the laser beam.

[0042]

As described above, according to the environment recognition apparatus of the present invention, the obtained stereo image is subjected to a conversion process defining a predetermined reference plane to create a virtual image, and the original image and the virtual image are created. A difference image is obtained as a difference from the image, and the obstacle is recognized in consideration of the shape of the image on the difference image, so that the obstacle can be reliably recognized together with its height position. .

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a principle of capturing a stereo image in an environment recognition device according to the present invention.

FIG. 2 is a diagram showing a three-dimensional coordinate system defined in left and right cameras when imaging is performed by the method shown in FIG. 1;

3 is a diagram showing a two-dimensional coordinate system obtained by two-dimensionally projecting the three-dimensional coordinate system shown in FIG.

FIG. 4 is a diagram showing the principle of obtaining a two-dimensional coordinate system by two-dimensionally projecting a three-dimensional coordinate system.

FIG. 5 is a diagram showing that a position in a three-dimensional space is determined by defining a virtual plane for an obtained two-dimensional image.

FIG. 6 is a diagram illustrating the principle of recognition of an obstacle N by the environment recognition device according to the present invention.

FIG. 7 is a diagram showing each image obtained in the environment recognition device according to the present invention in the environment shown in FIG. 6;

FIG. 8 is a diagram illustrating the principle of recognizing the height position of an obstacle N by the environment recognition device according to the present invention.

9 is a diagram showing a difference image obtained in the environment recognition device according to the present invention in the environment shown in FIG. 8;

FIG. 10 is a block diagram showing a basic configuration of an environment recognition device according to the present invention.

FIG. 11 is a diagram illustrating a principle for improving the efficiency of image calculation processing in the environment recognition device according to the present invention.

FIG. 12 is a diagram showing a state in which a region to be calculated is limited by the method shown in FIG. 11;

FIG. 13 is a diagram illustrating a method of determining two points for recognizing a traveling road surface in the environment recognition device according to the present invention.

FIG. 14 is a diagram illustrating a method of determining a third point for recognizing a traveling road surface in the environment recognition device according to the present invention.

FIG. 15 is a diagram illustrating a method for obtaining a correlation between pixels in the method shown in FIG. 14;

FIG. 16 is a block diagram showing a basic configuration of a measurement system for recognizing a traveling road surface in the environment recognition device according to the present invention.

17 is a diagram showing left and right images obtained by using the measurement system shown in FIG.

[Explanation of symbols]

 1, 2 ... vehicle (obstacle) 10 ... object 11 ... right camera 12 ... left camera 100 ... stereo image input means 110 ... conversion information storage means 200 ... image conversion means 210-214 ... image conversion device 300 ... image difference means 310 to 314 image difference device 400 obstacle recognition device 500 laser oscillator 510 modulator 521 to 523 half mirror 524, 525 total reflection mirror 530 light receiving element 540 distance calculation device 550 road plane calculation device AR , AL: Target area M: Vehicle N: Obstacle RR: Three-dimensional right coordinate system LL: Three-dimensional left coordinate system R: Two-dimensional right coordinate system L: Two-dimensional left coordinate system S: Line segment S0-S4: Reference plane P1 to P3 ... Points for plane determination P1 'to P3' ... Corresponding points of P1 to P3 Q1 to Q3 ... Epipolar line PR ... Right image PL ... Left image IR ... Virtual right Image DR: Difference image W: White line

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) G05D 1/02

Claims (8)

(57) [Claims] 1. A stereo image input means having a right image pickup device and a left image pickup device to obtain a two-dimensional stereo image composed of a right image and a left image of a recognition target; Conversion information storage means for storing conversion information between the left and right images determined based on a positional relationship; and a predetermined reference plane defined, and the right image is defined based on an expression indicating the reference plane and the conversion information. Image conversion means for performing at least one of right image conversion for creating a virtual left image corresponding to the left image conversion for creating a virtual right image corresponding to the left image, and For at least one of the combination and the combination of the left image and the virtual left image, an image difference unit that creates a difference image and a plurality of different reference planes are defined. A plurality of differential images obtained by comparison with one another, to select a preset reference plane difference image is obtained which is most approximate to the reference pattern,
An obstacle recognition device that recognizes an obstacle corresponding to the difference image at the height position of the selected reference plane. 2. A stereo image input means having a right image pickup device and a left image pickup device to obtain a two-dimensional stereo image composed of a right image and a left image of a recognition target; Conversion information storage means for storing conversion information between the left and right images determined based on a positional relationship; and a predetermined reference plane defined, and the right image is defined based on an expression indicating the reference plane and the conversion information. Image conversion means for performing at least one of right image conversion for creating a virtual left image corresponding to the left image conversion for creating a virtual right image corresponding to the left image, and For at least one of the combination and the combination of the left image and the virtual left image, an image difference unit that creates a difference image and a plurality of different reference planes are defined. An obstacle recognition device that recognizes a mutual change of a plurality of difference images obtained as a result and, when the mutual change is based on a preset change pattern, recognizes the obtained difference image as an obstacle. An environment recognition device, comprising: 3. The environment recognition apparatus according to claim 1, wherein different reference planes are defined, and a plurality of image conversion apparatuses capable of performing independent image conversion processing. Means for each of the virtual images converted by the plurality of image conversion devices, the image difference means comprising a plurality of image difference devices each independently generating a difference image, and for a plurality of different reference planes. An environment recognition device characterized in that each difference image can be calculated and obtained simultaneously. 4. The environmental certification according to claim 1,
In the recognition device, The stereo image input means separates the right image and the left image
Has the function of continuously capturing as time-series data, The image conversion means is provided for converting between images given before and after in time.
A time-series difference image showing the difference is obtained, and the time-series difference
Predetermined target area including temporal change based on minute image
And perform image conversion processing only for this target area
An environment recognition device, characterized in that: 5. A stereo image input means having a right image pickup device and a left image pickup device for obtaining a two-dimensional stereo image composed of a right image and a left image of a recognition target; A conversion information storage unit that stores conversion information between the left and right images determined based on a positional relationship, and a virtual left image corresponding to the right image based on an expression indicating a predetermined reference plane and the conversion information. Image conversion means for performing at least one of right image conversion to be created, and left image conversion to create a virtual right image corresponding to the left image, and a combination of the right image and the virtual right image, and the left image the combination of the virtual left image, for at least one of a combination of, and recognition and image difference means for generating a difference image, a partial feature on road surface, Oyo said right image
Three points P1, P2, and P3 are defined at positions indicating the feature on an actual image formed of one of the left images, and the conversion information is defined without defining a reference plane for these three points.
By performing image conversion processing based on the real image
The three points P1 and P
2, P3 corresponding to epipolar lines Q1, Q2, Q3
, And using the feature, the epipolar line Q
The corresponding points P1 ', P2', P3 'for the three points P1, P2, P3 are obtained on 1, Q2, Q3.
A plane determined based on P2 and P3 and their corresponding points P1 ', P2' and P3 'is defined as a reference plane corresponding to the traveling road surface, and this reference plane is given to the image conversion means.
Reference plane defining means for performing processing , and the image conversion means indicating a reference plane corresponding to the traveling road surface.
Difference image obtained by performing image conversion using
Environment recognition apparatus characterized by comprising, an obstacle recognition device for recognizing an obstacle based on. 6. The environment recognition apparatus according to claim 5, wherein the reference plane defining means temporarily sets a plane defined based on the three points P1, P2, P3 and their corresponding points P1 ', P2', P3 '. In the environment where there is no obstacle , the image conversion unit uses the temporary reference plane to perform image conversion.
Then, the virtual image and the real image obtained by the above are compared, and the correction is performed on the temporary reference plane until they match, and the corrected reference plane is defined as the reference plane corresponding to the traveling road surface .
Environment recognition apparatus characterized by that. 7. A stereo image input means having a right image pickup device and a left image pickup device for obtaining a two-dimensional stereo image composed of a right image and a left image of a recognition target; A conversion information storage unit that stores conversion information between the left and right images determined based on a positional relationship, and a virtual left image corresponding to the right image based on an expression indicating a predetermined reference plane and the conversion information. Image conversion means for performing at least one of right image conversion to be created, and left image conversion to create a virtual right image corresponding to the left image, and a combination of the right image and the virtual right image, and the left image An image difference means for creating a difference image for at least one of the combinations of the virtual left images, and an irradiation point on the road surface can be recognized as a point having a high luminance. Means for irradiating the traveling road surface with a light beam, and the image conversion means comprises:
Image conversion or matching of the irradiation point on the right image
The irradiation point on the left image and the irradiation point on the virtual left image are
It is possible to perform image conversion that matches
Define the reference plane as the reference plane corresponding to the traveling road surface
Then, a process of giving the reference plane to the image conversion means is performed.
Reference plane definition means and the image conversion means indicate a reference plane corresponding to the traveling road surface.
Difference image obtained by performing image conversion using
Environment recognition apparatus characterized by comprising, an obstacle recognition device for recognizing an obstacle based on. 8. A stereo image input means having a right image pickup device and a left image pickup device to obtain a two-dimensional stereo image composed of a right image and a left image of the recognition target; A conversion information storage unit that stores conversion information between the left and right images determined based on a positional relationship, and a virtual left image corresponding to the right image based on an expression indicating a predetermined reference plane and the conversion information. Image conversion means for performing at least one of right image conversion to be created, and left image conversion to create a virtual right image corresponding to the left image, and a combination of the right image and the virtual right image, and the left image An image difference means for creating a difference image for at least one of the combinations of the virtual left images, and an irradiation point on the road surface can be recognized as a point having a high luminance. Means for irradiating a light beam at a predetermined irradiation angle with respect to road surface as, by receiving the light beam which has been reflected from the irradiation point, and means for determining the distance between the irradiation point and the vehicle, the A reference plane corresponding to the traveling road surface is defined based on the position of the irradiation point on the right image or the left image, the irradiation angle, and the distance, and the reference plane is defined as the image conversion point.
A reference plane defining means for performing a process given to a step, and the image conversion means indicating a reference plane corresponding to the traveling road surface.
Difference image obtained by performing image conversion using
Environment recognition apparatus characterized by comprising, an obstacle recognition device for recognizing an obstacle based on.