JP2000293693A - Obstacle detecting method and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect an object existing on a road surface even when vibration during traveling is generated or inclination on the road surface itself exists without operating any calibration. SOLUTION: An obstacle detecting device 1 which detects an obstacle region on a road plane having plural lines is provided with two TV cameras 2a and 2b which photograph a read surface, a picture storing part 3 which stores right and left pictures photographed by the TV cameras 2b and 2a, a feature extracting part 4 which extracts plural lines expressed on the right and left pictures stored in the picture storing part 3, and which calculates corresponding points between the right and left pictures based on the extracted plural lines, a relational formula parameter calculating part 5 which calculates the parameter of a relational formula established between the projecting positions of arbitrary points of the road surface to the right and left pictures based on the corresponding points calculated by the feature extracting part 4, and a detecting part 6 which detects a region having height different from that of the road surface as an obstacle range based on the relational formula decided according to the parameter calculated by the relational formula parameter calculating part 5.

Description

DETAILED DESCRIPTION OF THE INVENTION

BACKGROUND OF THE INVENTION In order to assist the driving of a vehicle such as an automobile running on a surface such as a road, a TV camera is used to control other vehicles such as a preceding vehicle and a vehicle other than the own vehicle existing on the road such as a pedestrian. The present invention relates to an obstacle detection method and apparatus for detecting an object (hereinafter, defined as an obstacle).

2. Description of the Related Art Techniques for detecting obstacles on a road are roughly classified into those using a laser or an ultrasonic wave as an obstacle detecting means, and those using a TV camera. The method using a laser as an obstacle detecting means is impractical because the laser itself is expensive. Further, the method of using an ultrasonic wave as an obstacle detecting means has a problem in the accuracy of detecting an obstacle because the resolution of the ultrasonic wave is low, which also hinders practicality. For the above two methods,
The method of using a TV camera as an obstacle detection means is as follows.
It is known that the V-camera itself is relatively inexpensive and is suitable for obstacle detection in terms of its resolution, measurement accuracy, measurement range, and the like. When a TV camera is used as an obstacle detecting unit on a road, there are a method using one TV camera and a method using a plurality of cameras (stereo cameras).
In a method using one TV camera, a road area and an obstacle area are separated from one image captured by the camera using information such as luminance, color, and texture (pattern). For example, a medium-luminance region with low saturation in the image, that is, a gray region is extracted to obtain a road region, or a region with little texture is extracted to extract a road region, and other regions are determined as obstacle regions. And However, since there are many obstacles on the road having similar brightness, color, or texture to the road, it is difficult to distinguish between the obstacle area and the road area by this method. On the other hand, the method using a plurality of TV cameras is 3
An obstacle on the road is detected using the dimensional information as a clue.
This is a detection method generally called stereo vision. Stereo vision refers to, for example, arranging two cameras on the left and right,
The same point in the dimensional space is associated with the left and right images,
In the manner of triangulation, the three-dimensional position of the point is determined. If the position and posture of each camera with respect to the road plane are determined in advance, the height of an arbitrary point in the image from the road plane can be obtained by stereo vision. And the road area can be separated. According to the obstacle detection method using stereo vision, an obstacle having a brightness, color, or texture similar to a road, which is difficult to distinguish when one camera is used, is distinguished from the road area. It is possible to detect separately. By the way, the usual stereo vision technique uses three points with respect to the world coordinate system (absolute coordinate system) of an arbitrary point on an image.
This is a technique for obtaining a three-dimensional position. For this purpose, it is necessary to previously obtain parameters (relational expression parameters) relating to the relationship of each camera with respect to the world coordinate system {position, posture (photographing direction, etc.), focal length of camera lens, etc.}. is there.
Hereinafter, the operation (process) for obtaining the relational expression parameters is referred to as calibration. Calibration prepares a number of points whose positions with respect to the world coordinate system are known, finds the projection positions of these points on the image, and obtains the relational expressions regarding the position and orientation of the camera with respect to the world coordinate system, the focal length of the camera lens, and the like. This is the task of calculating the parameters. However, there is a problem that a great deal of time and labor is required to perform the calibration work. Therefore, focusing only on separating the road area and the obstacle area on the image, the presence or absence of a height from the road plane is determined without performing calibration, and the height is present. Object, no height → A method of detecting an obstacle as a road plane has been devised. At this time, the presence or absence of the height from the road plane can be determined as follows. The projection points of points on the road plane onto the left and right images are (u, v), (u ', v')
given that,

(Equation 1) ★

[Outside 1] ★ ● Using this relational expression, any point P (u, v) on the left image
Is found on the right image assuming that exists on the road plane, the corresponding point P ′ (u ′, v ′) is obtained. If the point P exists on the road plane, the points P and P 'form a correct set of corresponding points, and the difference between the luminances of the two points becomes small. Therefore, when the difference in luminance between the points P and P ′ is large, it can be determined that the point P belongs to the obstacle region, and the obstacle can be determined without performing calibration.

[Outside 2] ★

As described above, an obstacle detecting apparatus for detecting an obstacle without performing calibration based on the presence or absence of a height from a road plane using a plurality of TV cameras is an ultrasonic detecting apparatus. It has many advantages in that it is inexpensive, has high detection accuracy, and reduces the time and effort required for obstacle detection by eliminating the need for calibration work, as compared with a device that uses a laser as a detection means. .

[Outside 3] ★ ● The present invention has been made in view of the above circumstances, and extracts a projection image of two or more lines on a surface from a plurality of images taken using a plurality of TV cameras whose only ebipolar constraints are known. By calculating the geometric relationship between the surface and each camera from the extracted projection image, calibration is not required, and obstacles existing on the road surface even if the road surface It is an object of the present invention to provide an obstacle detection method and apparatus capable of detecting an obstacle.

According to a first aspect of the present invention, there is provided an obstacle detecting apparatus for detecting an obstacle area on a surface having a plurality of lines, the apparatus comprising: A plurality of image capturing devices, image storing means for storing a plurality of images respectively captured by the plurality of image capturing devices, and extracting the plurality of lines represented on the plurality of images stored in the image storing means And extracting means for obtaining a corresponding point between the plurality of images based on the plurality of extracted lines, based on the corresponding points obtained by the extracting means,
Parameter calculating means for calculating a parameter of a relational expression that holds between the projection positions of the arbitrary points on the surface to the respective images, and the surface based on a relational expression determined by the parameter calculated by the parameter calculating means. Detecting means for detecting areas having different heights as obstacle areas. In the first invention, when an image photographed by one of the plurality of photographing devices is set as a reference image, and an image photographed by the remaining photographing devices is set as a reference image, A corresponding point on the reference image corresponding to an arbitrary point is constrained on a straight line connecting the arbitrary point on the reference image and the viewpoint of the reference imaging device onto a projection straight line on the reference image.
In the first invention, it is preferable that the extraction unit includes a corresponding point between the reference image and the reference image based on a plurality of lines on the reference image and a straight line projected on the reference image with respect to the reference image. The parameter calculation unit calculates the reference image of an arbitrary point on the surface based on a set of a plurality of corresponding points between the reference image and the reference image obtained by the extraction unit. And means for obtaining a parameter of a relational expression that is established between the projection position with respect to the reference image. In the first invention, in particular,
The extracting means determines an intersection of a first plurality of points on one line 1 of the plurality of lines on the reference image and a projection line on the reference image as a first plurality of corresponding points. Determining the intersection of a second plurality of points on one line other than one of the plurality of lines on the reference image and a projection line on the reference image as a second plurality of corresponding points, The set of the first plurality of points and the first plurality of corresponding points and the set of the second plurality of points and the second plurality of corresponding points are determined. In the first invention, preferably, the detecting means determines a corresponding point on the reference image corresponding to the arbitrary point, assuming that an arbitrary point on the reference image exists on the surface. By comparing the luminance of an arbitrary point on the reference image with the calculated luminance of the corresponding point on the reference image, the arbitrary luminance on the reference image is calculated based on a relational expression determined by the relational expression parameter. Is determined to belong to an obstacle area having a different height with respect to the plane. In the first invention, the detection unit converts the reference image into an image obtained by the reference imaging device, assuming that an arbitrary point of the reference image exists on the surface. Calculating a difference between the reference image and the reference image converted by the conversion unit, and based on the calculated difference, any point on the reference image having an obstacle area having a different height with respect to the plane; And a difference calculation means for determining whether or not the data belongs to In the first aspect, the surface is a road surface on which a moving body that moves in a real space travels, and the obstacle detection device is mounted on the moving body. According to a second aspect of the present invention, there is provided an obstacle detection method for detecting an obstacle area on a plane having a plurality of lines, comprising: a step of photographing a plurality of images; Storing a plurality of captured images, extracting the plurality of lines represented on the plurality of images stored by the storing step, and performing correspondence between the plurality of images based on the plurality of extracted lines. Obtaining a point; calculating parameters of a relational expression that holds between the projection positions of any points on the surface onto each image based on the corresponding points obtained in this step; Detecting regions having different heights with respect to the surface as obstacle regions based on a relational expression determined by parameters calculated by the above.

Embodiments of the present invention will be described with reference to the accompanying drawings. In the present embodiment, two left and right image photographing devices (TV camera and stereo camera) mounted on a vehicle running on a road surface (road plane) such as an automobile receive a pedestrian or a preceding vehicle in the traveling direction. Assuming a situation where an obstacle existing on a road plane such as is detected, an obstacle detection process performed by the obstacle detection device in the assumed situation will be described. FIG. 1 is a block diagram illustrating a schematic configuration of an obstacle detection device 1 according to the present embodiment. The obstacle detection device 1 according to the present embodiment is mounted on a vehicle traveling on a road plane. The obstacle detection device 1 is connected to each of the TV cameras and a road plane that changes every moment due to vibration of the own vehicle when traveling or an inclination of the road plane. A geometric relationship is obtained, an obstacle existing on a road plane is detected using the geometric relationship, and the detected obstacle is displayed. That is, the obstacle detection device 1 is configured to capture an image on a common plane in a three-dimensional space related to the host vehicle, for example, a road plane in front of the traveling direction.
An image photographing unit 2 having two TV cameras (left TV camera 2a, right TV camera 2b), and accumulating an image photographed by the TV camera 2a (left image) and an image photographed by the TV camera 2b (right image) An image storage unit 3 for storing (memory), a feature extraction unit 4 for extracting two or more characteristic lines on the left and right images, and a relational expression parameter based on the extracted characteristic lines. A parameter calculating unit 5 for calculating and a detecting unit 6 for obtaining points having different heights with respect to the road plane based on the calculated relational expression parameters and detecting these points as obstacles (obstacle areas). Have. In the present embodiment, as shown in FIG. 2, the two lines on the road plane are defined as two white lines (1, 1 ') at both ends of the road plane, and the directions of the straight lines 1, 1' are Y.
A world coordinate system (absolute coordinate system) is set in which a direction orthogonal to the Y axis along the road plane with respect to the Y axis is the X axis.
The image photographing unit 2 uses the two left and right TV cameras 2a and 2b to form two images (left image IL and right image IR) on the road plane.
Are to be shot respectively. Each TV camera 2
a and 2b are mounted side by side at predetermined intervals, for example, along the X-axis direction of the world coordinate system, and their shooting directions are not limited to the front of the traveling vehicle, but may be the rear and side directions. You may. In addition, each TV camera 2
By switching the focal lengths a and 2b, wide-angle shooting and telephoto shooting can be performed according to the condition of the road plane (road surface). The two TV cameras 2a and 2b according to the present embodiment are configured so that their positions and postures with respect to the world coordinate system are unknown and only the epipolar constraint is known, and do not change during traveling. Here, the epipolar constraint is a constraint that holds for a general stereo camera, and corresponds to an arbitrary point k on one image (the left image in this embodiment, which is also referred to as a reference image). Points (corresponding points) on the other image (the right image in this embodiment, which is also referred to as a reference image)
k 'means being constrained on a certain straight line on the right image IR as shown in FIG. This straight line is called an epipolar line EL. This epipolar line EL
Is determined by projecting the point k on the left image IL and the viewpoint (center point position of the lens) of the left TV camera 2a on the right image IR and connecting them. For example, when the optical axes of the TV cameras 2a and 2b are arranged in parallel, a corresponding point on the right image IR of an arbitrary point on the left image IL exists on the same scanning line on the right image IR. Therefore, the epipolar line and the scanning line coincide with each other. Epipolar restraint
Relative position / posture relationship between stereo cameras (left TV camera 2a, right TV camera 2b) and each camera 2
The fact that the ebipolar constraint is invariable depends on the internal parameters a and 2b, that is, the focal length of each camera lens, the image origin, and the like.
This means that the relative positional relationship of b does not change during traveling. The epipolar constraint is obtained as follows. Now, any of the corresponding point set of stereo image (left image IL, right image _{IR) (u i, v i} ), (u i ', v i')
(I = 1, 2,..., N)

(Equation 2) ★ (where F is a 3 × 3 matrix). The matrix F can be obtained from eight or more sets, that is, N = 8 or more corresponding points. That is, since the matrix F can be obtained using only a set of corresponding points, there is no need to prepare a point whose three-dimensional position is known with respect to the world coordinate system. The matrix F can be obtained by simple processing. When the matrix F is obtained in this way, the corresponding point of a certain point (u, v) on the left image IL becomes a straight line on the right image IR.

(Equation 3) ★ It is above, and this is the shrimp polar line EL. That is, equation (2) represents the epipolar constraint. Here, F _ij (i, j = 1, 2, 3) is the i of the matrix F
Element of row j column. The image storage unit 3 has an image memory, and stores two images (left images IL and IR) captured by the TV cameras 2a and 2b of the image capturing unit 2 in the image memory, respectively. ing. The feature extraction unit 4 has an arithmetic processing processor and a memory,
First, on the two images (the left image IL and the right image IR) stored by the image storage unit 2, two straight lines (white lines) 1 and 1 'at both ends of the road are respectively extracted. The extraction of the two straight lines is performed using Hough transform or the like.
The straight lines 1 and 1 ′ have intersections (vanishing points) V and V ′ on the left and right images IL and IR, respectively, and the calculation amount can be reduced by using that V ′ is on the epipolar line Lv of V. Is possible. At this time, if any two points on the straight line 1 on the left image IL are A and C, and any two points on the straight line 1 ′ are B and D, the feature extraction unit 4
The corresponding points A ', B', C ', D' on the right image IR of the point are
It is determined by using the epipolar constraint. That is,
The corresponding point A 'of the point A can be obtained as an intersection of the straight line 1 and the epipolar line LA of the point A on the right image IR. Similarly, the points B', C ', and D' are respectively , C and D as the intersections of the shrimp polar lines LB, LC and LD. Here, the obtained point A,
The coordinate data of B, C, D and the points A ′, B ′, C ′, D ′ are respectively expressed as (u _A , v _A ), (u _B , v _B ), (u
_{C, v C), (u} D, v D), (u 'A, v' A),
_{(U 'B, v' B} ), (u 'C, v' C), (u 'D,
v ′ _D ). The parameter calculation unit 5 has an arithmetic processor and a memory, and projects any point on the road plane on the left image IL based on the correspondence between the four sets of points obtained by the feature extraction unit 4. A parameter of a relational expression (relational expression parameter) that is established between the point (u, v) and the projection point (u ′, v ′) on the right image IR is calculated. Now, assuming that the projection points of the arbitrary point P (X, Y) on the road plane on the left and right images IL and IR are (u, v) and (u ', v'),

(Equation 4) ★ The following relational expression holds.

[Outside 4] ★ ●

[Outside 5] ★ ●

[Outside 6] ★

(Equation 5) ★ ●

[Outside 7] ★ ● The detection unit 6 has an arithmetic processing processor and a memory, and calculates the luminance of an arbitrary point P (u, v) on the left image IL as B
Let _L (u, v) be the corresponding point P '(u, v) on the right image IR of the point P (u, v) assuming that this point P (u, v) exists on the road plane. ) Is obtained based on the above equation (4). Now, the point obtained P '(u', v ' ) luminance _B R (u, v) and when the point P (u, v) is if actually present on the road plane, and the point P P Since 'is a correct set of corresponding points, the brightness of the points P and P' is basically the same. That is,

(Equation 6) In the case of ★, the detection unit 6 determines that the point P satisfying D> Thr (Thr is a preset threshold) belongs to the obstruction book area in consideration of D ≠ 0 or an error, and determines this point P as an obstacle. It can be detected as an object.

[Outside 8] ★ ● Therefore, it is possible to improve the detection accuracy of the obstacle (obstacle detection area), and to contribute to further improvement of the safety and reliability of the vehicle. For example, as shown in FIG. 5, an alarm generation unit 7 that performs an alarm generation process based on obstacle area information detected by the detection unit 6 and driving direction instruction information corresponding to, for example, a steering wheel operation of the driver in the own vehicle is provided. By providing this, when the driver's steering wheel operation sends driving direction instruction information for directing the vehicle to an obstacle area (obstacle), the alarm generation section 7 Based on the area information and the driving direction instruction information, it is determined that the own vehicle is traveling toward the obstacle area, and an alarm is generated. As a result, since the driver of the traveling vehicle can recognize that the own vehicle is traveling toward the obstacle, it is possible to contribute to further improving the safety and reliability of the vehicle. Further, as a modified example of the present embodiment, as shown in FIG. 6, a display 8 mounted at a position where the driver in the vehicle can be visually recognized and, for example, a left image IL stored in the image storage unit 3 are displayed on the display. 8 on the basis of the obstacle area information detected by the detection unit 6, a display process (e.g., different display modes of the safety area other than the obstacle area and the obstacle area with respect to the left image IL) Image processing for performing a process of setting the safety area to a green display color and an obstacle area to a red display color, a process of superimposing a marker representing the obstacle area on the coordinate position of the obstacle area on the left image IL, and the like. The processor 9 may be provided.With such a configuration, the obstacle region and the safety region existing in front of the traveling vehicle can be clearly identified on the display 8, so that the display 8 can be visually recognized. An obstacle or the like present in the blind spot of the field of view of Utatete can be recognized easily and reliably, it is possible to contribute to further improvement of the safety and reliability of the vehicle. In the present embodiment,
Two TV cameras 2a and 2b of the image photographing unit 2 are mounted on a vehicle side by side and mounted on a vehicle, and two images are photographed by the TV cameras 2a and 2b.
May be arranged in any manner as long as the epipolar constraint is maintained and a common plane relating to the vehicle can be photographed. It is also possible to arrange three or more TV cameras on the vehicle. When the failure detection processing of the present embodiment is performed using three or more TV cameras, the epipolar constraint
One of the three or more captured images is used as a reference image,
This means that a corresponding point k ′ on the remaining images (reference images) corresponding to an arbitrary point k on the reference image is constrained on a certain straight line on each reference image. Also, the feature extraction unit 4
Has determined the correspondence between four pairs of points on the left image IL and the right image IR, but may also determine the correspondence between five or more pairs in the same manner. In this case, the parameter calculator 5
It is also possible to solve ten or more simultaneous equations obtained from the correspondences of pairs or more using the least square method or the like. Also,
The detection unit 6 can be further configured as shown in FIG. In this modified example, the detection unit 6 includes an image conversion unit 10 and a difference calculation unit 11, and the image conversion unit 10 and the difference calculation unit 11 are embodied as processing functions of an arithmetic processing processor. The image conversion unit 10 performs image conversion of the right image IR as a reference image according to the following procedure.
Generally, an image can be expressed as a function f (u, v) in which points (u, v) on the image are variables and a luminance value is defined for each point. Hereinafter, an image is represented in this manner. For example, suppose that a stereo image (left image, right image) including a preceding vehicle as shown in FIG.
Assuming that (u, v), the right image is g (u, v), and the converted image of the right image is g ′ (u, v), the image converter 10 converts the converted image g ′ as follows. Find (u, v).

G ′ (u, v) = g (u ′, v ′) (7) where (u ′, v ′) is obtained by the above equation (4).
g ′ (u, v) is an image obtained by the left TV camera 2a assuming that an arbitrary point on the image g (u, v) exists on the road plane. For example, from a right image g (u, v) shown in FIG. 9, a converted image g ′ as shown in FIG.
(U ', v') is obtained. As shown in FIG. 10, the projection point of a point on the road plane is the same in the left image f (u, v) and the transformed image g ′ (u ′, v ′), whereas the projection point is on the road plane. The missing point, that is, the point on the obstacle (in this case, the preceding vehicle) is projected to different positions depending on the height from the road. Therefore, an obstacle on the road plane can be detected by taking the difference (inter-pixel difference) between the left image f (u, v) and the converted image g ′ (u ′, v ′). It is. That is, the difference calculation unit 11

(Equation 8) The difference D ′ represented by ★ is calculated, and this D ′ is D ′ ≠ 0,
Alternatively, considering an error, a point (u, v) where D '> Thr (Thr is a preset threshold) is determined to belong to the obstacle area, and this point (u, v) is detected as an obstacle. Can be. In addition, the difference detection unit of the detection unit 6 detects the difference between the two images (the left image and the right image) by calculating the difference between the pixels, but the present modification is not limited to this. For example, the difference detection unit of the detection unit 6 sets a (2ω + 1) × (2ω + 1) window for each point (each pixel) corresponding to each image (left image, right image).
It is also possible to detect the difference by calculating the normalized cross-correlation C of the luminance values in the window. In this case, the point (u, v) C of two images (left image, right image) F (u, v) and G (u, v) is

(Equation 9) ★ is represented by Here, N = (2ω + 1) × (2ω +
1), a1, a2 are two images F (u, v), G (u,
The average brightness in the window of v), σ ² ₁ , σ ² _2, is
This is the variance of the luminance in the windows of the two images F (u, v) and G (u, v). At this time, the difference calculation unit 11 determines that a point (u, v) satisfying the calculated value C <Thr (Thr is a preset threshold) belongs to the obstacle area. As described above, even if the detection unit is configured as shown in the modification, the obstacle area can be easily detected. In this embodiment, the two ends of the road plane are
Although the white lines are extracted as straight lines, if the road is curved, the white lines are curved. In this case, if the white line is extracted as a curve, an obstacle can be detected in the same manner. Further, in the present embodiment, the description has been made assuming that the road surface is a plane. However, even in the case of a curved surface, an obstacle can be detected as in the case of a plane. Furthermore, in the present embodiment, it has been described that each of the block components 4 to 6 shown in FIG. 1 has an arithmetic processing processor and a memory, but the present invention is not limited to this. The configuration may be such that the feature extraction processing of the feature extraction unit 4, the relational expression parameter calculation processing of the relational expression parameter calculation unit, and the obstacle detection processing of the detection unit 6 are performed by one arithmetic processing processor and memory. By the way, in the present embodiment, the detection of an obstacle from a TV camera mounted on a vehicle traveling on a road surface has been described, but the present invention is not limited to this.
The present invention can also be applied to obstacle detection when a moving object such as an aircraft or a helicopter lands on a road surface such as a runway.
On the other hand, according to the present embodiment, a case where two or more TV cameras are mounted on a moving body such as a vehicle has been described, but the present invention is not limited to this. For example, as shown in FIG. 11, one TV camera 2 is mounted on a moving body (vehicle 15), and a road monitor for road monitoring is installed in advance, for example, at predetermined intervals above a road side of a road on which the vehicle 15 runs. Camera 16
It is also possible to perform the obstacle detection processing based on the above-described stereo vision technology by using the image of (1). That is, the obstacle detection device 1a having one TV camera 2A mounted on the vehicle 15 receives the image of the road monitoring camera 16 installed near the traveling position via the wireless communication device 17, and receives the image. Image (first image) and own vehicle 1
By executing the processing based on the above-described stereoscopic vision technology based on the image (second image) captured by Step 5, an obstacle on the road can be detected. Then, the obstacle detection device 1a transmits the first image transmitted from the monitoring camera 16 closer to the current traveling position of the own vehicle 15 via the wireless communication device 17 in accordance with the traveling of the own vehicle 15. By using this, continuous obstacle detection processing can be performed. With this configuration, in addition to the above-described effects, the number of TV cameras mounted on the vehicle can be reduced, and the system of the obstacle detection device is simplified.
When the size of the TV camera mounted on the vehicle and the size of the road monitoring camera are different, the obtained first and second
It is necessary to perform the correction processing between the images. Others
Any modification is possible without departing from the gist of the present invention.

As described above, according to the obstacle detecting method and apparatus of the present invention, an obstacle can be detected based on the presence or absence of a height from a surface such as a road surface. An obstacle such as a preceding vehicle or a pedestrian can be detected from the image without being affected by the shadow. In particular, in the present invention, since the geometric relationship between the road surface and each photographing device is obtained by simple processing such as detection of a straight line, even when the vehicle is traveling or is inclined on a road plane, the surface can be stably obtained. The above obstacle can be detected, and its practicality can be greatly improved.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of an obstacle detection device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a world coordinate system set on a road plane and a white line on the road plane.

FIG. 3 is a diagram for explaining epipolar constraint.

FIG. 4 is a view for explaining a white line extraction process and a process of obtaining a set of corresponding points between left and right images by the feature extraction unit shown in FIG. 1;

FIG. 5 is a block diagram showing an obstacle detection device having an alarm generation processing function in addition to the configuration of FIG. 1;

FIG. 6 is a block diagram showing an obstacle detection device having display processing of an obstacle area and a safety area for a display in addition to the configuration of FIG. 1;

FIG. 7 is a block diagram showing another configuration of the detection unit shown in FIG. 1;

FIG. 8 is a diagram showing an example of a stereo image.

FIG. 9 is a diagram showing a right image and its converted image.

FIG. 10 is a diagram showing a converted image of a left image and a right image.

FIG. 11 shows a modification of the present invention, in which one moving body is provided with one TV.
The figure which shows the structure which mounts a camera and performs stereoscopic vision using the camera for road monitoring installed in the road side of the road.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Obstacle detection device 2 Image photographing part 2a, 2b TV camera 3 Image storage part 4 Feature extraction part 5 Relational expression parameter calculation part 6 Detection part 7 Alarm generation part 8 Display 9 Image processor 10 Image conversion part 11 Difference calculation part

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] July 13, 1999 (1999.7.1)
3)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Full text

[Correction method] Change

[Correction contents]

[Document Name] Statement

Patent application title: Obstacle detection method and device

[Claims]

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION In order to assist the driving of a vehicle such as an automobile running on a surface such as a road, a TV camera is used to control other vehicles such as a preceding vehicle and a vehicle other than the own vehicle existing on the road such as a pedestrian. The present invention relates to an obstacle detection method and apparatus for detecting an object (hereinafter, defined as an obstacle).

[0002]

2. Description of the Related Art Techniques for detecting obstacles on a road are roughly classified into those using a laser or an ultrasonic wave as an obstacle detecting means, and those using a TV camera.

The method using a laser as an obstacle detecting means is impractical because the laser itself is expensive.
Also, the method of using ultrasonic waves as obstacle detection means is as follows:
Since the resolution of the ultrasonic wave is low, there is a problem in the accuracy of detecting an obstacle, which also hinders practicality.

[0004] In contrast to the above two methods, the method using a TV camera as an obstacle detecting means is relatively inexpensive for the TV camera itself, and also detects an obstacle in terms of its resolution, measurement accuracy, and measurement range. Has proven to be suitable for

[0005] When a TV camera is used as an obstacle detecting means on a road, there are a method using one TV camera and a method using a plurality of cameras (stereo cameras).

A method using one TV camera separates a road area and an obstacle area from one image taken by the camera by using information such as brightness, color, and texture (pattern) as clues. I do. For example, a medium-luminance region with low saturation in the image, that is, a gray region is extracted to obtain a road region, or a region with little texture is extracted to extract a road region, and other regions are determined as obstacle regions. And

[0007] However, there are many obstacles on the road having luminance, color, or texture similar to the road, and it is difficult to distinguish between the obstacle area and the road area by this method.

On the other hand, a method using a plurality of TV cameras detects an obstacle on a road using three-dimensional information as a clue. This is a detection method generally called stereo vision. Stereo vision refers to, for example, arranging two cameras on the left and right, associating a point that is the same point in a three-dimensional space between left and right images, and finding a three-dimensional position of the point in the manner of triangulation. .

If the position, posture, and the like of each camera with respect to the road plane are obtained in advance, the height of an arbitrary point in the image from the road plane can be obtained by stereo vision. The obstacle area and the road area can be separated. According to the obstacle detection method using stereo vision, an obstacle having a brightness, color, or texture similar to a road, which is difficult to distinguish when one camera is used, is distinguished from the road area. It is possible to detect separately.

By the way, the ordinary stereo vision technique is a technique for obtaining a three-dimensional position of an arbitrary point on an image with respect to a world coordinate system (absolute coordinate system). It is necessary to find parameters (relational expression parameters) related to the relation {position, posture (photographing direction, etc.), focal length of camera lens, etc.}. Hereinafter, the operation (process) for obtaining the relational expression parameters is referred to as calibration.

In the calibration, a number of points having known positions with respect to the world coordinate system are prepared, the projection positions of these points on the image are obtained, the position and orientation of the camera with respect to the world coordinate system, the focal length of the camera lens, and the like. This is the operation of calculating the relational expression parameters related to.

However, there is a problem that a great deal of time and labor is required to perform the above-mentioned calibration work.

Therefore, by focusing only on separating the road area and the obstacle area on the image, the presence or absence of the height from the road plane is determined without performing calibration, and the height is determined. Above obstacle, no height → A method of detecting an obstacle as a road plane has been devised.

At this time, the presence or absence of the height from the road plane is determined by
It can be determined as follows.

If the projection points of points on the road plane onto the left and right images are (u, v) and (u ', v'),

(Equation 1)

[Outside 1]

Using this relational expression, the corresponding point P '(u', v ') on the right image, assuming that an arbitrary point P (u, v) on the left image exists on the road plane, is calculated. Ask. If the point P exists on the road plane, the points P and P 'form a correct set of corresponding points, and the difference between the luminances of the two points becomes small. Therefore, when the difference in luminance between the points P and P ′ is large, it can be determined that the point P belongs to the obstacle region, and the obstacle can be determined without performing calibration.

[0017]

[Outside 2]

[0018]

As described above, an obstacle detecting apparatus for detecting an obstacle without performing calibration based on the presence or absence of a height from a road plane using a plurality of TV cameras is an ultrasonic detecting apparatus. It has many advantages in that it is inexpensive, has high detection accuracy, and reduces the time and effort required for obstacle detection by eliminating the need for calibration work, as compared with a device that uses a laser as a detection means. .

[0019]

[Outside 3]

The present invention has been made in view of the above-mentioned circumstances, and it is possible to form a projection image of two or more lines on a surface from a plurality of images photographed using a plurality of TV cameras having only known ebipolar constraints. By extracting the geometric relationship between the surface and each camera from the extracted projected images, calibration is not required, and obstacles existing on the road surface even if there is vibration or inclination on the road surface during traveling It is an object of the present invention to provide an obstacle detection method and apparatus capable of detecting an object.

[0021]

According to a first aspect of the present invention, there is provided an obstacle detecting apparatus for detecting an obstacle area on a surface having a plurality of lines, the apparatus comprising: A plurality of image capturing devices, image storing means for storing a plurality of images respectively captured by the plurality of image capturing devices, and extracting the plurality of lines represented on the plurality of images stored in the image storing means And extracting means for obtaining a corresponding point between the plurality of images based on the plurality of extracted lines, based on the corresponding points obtained by the extracting means,
Parameter calculating means for calculating a parameter of a relational expression that holds between the projection positions of the arbitrary points on the surface to the respective images, and the surface based on a relational expression determined by the parameter calculated by the parameter calculating means. Detecting means for detecting areas having different heights as obstacle areas.

In the first invention, when an image photographed by one of the plurality of photographing devices is used as a reference image, and an image photographed by the remaining photographing devices is used as a reference image, A corresponding point on the reference image corresponding to an arbitrary point on the image is constrained on a projection straight line on the reference image connecting a point on the reference image and a viewpoint of the reference imaging device. You.

[0023] In the first invention, preferably, the extracting means includes a plurality of lines on the reference image and a straight line projected on the reference image with respect to the reference image. The parameter calculation means calculates a set of corresponding points on the surface based on a set of a plurality of corresponding points between the reference image and the reference image obtained by the extraction means. This is means for obtaining a parameter of a relational expression that holds between the projection position with respect to the reference image and the reference image.

[0024] In the first invention, in particular, the extraction means is configured to determine a relationship between a first plurality of points on one line l of the plurality of lines on the reference image and projection lines on the reference image. Intersection is first
Of the plurality of lines on the reference image, and the intersection of the second plurality of points on one line other than l and the projection line on the reference image By obtaining the corresponding points, the first plurality of points and the first plurality of corresponding points and the second plurality of points and the second plurality of corresponding points are obtained.

[0025] In the first invention, preferably, when it is assumed that an arbitrary point on the reference image exists on the surface, the detecting means may detect the point on the reference image corresponding to the arbitrary point. The corresponding point is determined based on a relational expression determined by the relational expression parameter, and the brightness of an arbitrary point on the reference image is compared with the calculated brightness of the corresponding point on the reference image to obtain the reference image. It is determined whether or not the above arbitrary point belongs to an obstacle area having a different height with respect to the plane.

In the first invention, the detecting means converts the reference image into an image obtained by the reference photographing device, assuming that an arbitrary point of the reference image exists on the surface. Means, calculating a difference between the reference image and the reference image converted by the conversion means, and based on the calculated difference, any point on the reference image has a different height with respect to the plane A difference calculating means for determining whether or not the object belongs to the obstacle area.

In the first aspect, the surface is a road surface on which a moving body moving in a real space travels, and the obstacle detecting device is mounted on the moving body.

According to a second aspect of the present invention, there is provided an obstacle detecting method for detecting an obstacle area on a plane having a plurality of lines, comprising: Storing a plurality of images photographed by the photographing step, extracting the plurality of lines represented on the plurality of images stored by the storing step, and extracting the plurality of images based on the plurality of extracted lines. Calculating a corresponding point between the steps; calculating, based on the corresponding point obtained in this step, parameters of a relational expression that holds between the projection positions of the arbitrary points on the surface onto the respective images; Detecting a region having a different height with respect to the surface as an obstacle region based on a relational expression determined by the parameter calculated in the parameter calculation step.

[0029]

Embodiments of the present invention will be described with reference to the accompanying drawings. In the present embodiment, two left and right image photographing devices (TV camera and stereo camera) mounted on a vehicle running on a road surface (road plane) such as an automobile receive a pedestrian or a preceding vehicle in the traveling direction. Assuming a situation where an obstacle existing on a road plane such as is detected, an obstacle detection process performed by the obstacle detection device in the assumed situation will be described.

FIG. 1 is a block diagram showing a schematic configuration of an obstacle detection device 1 according to the present embodiment.

The obstacle detecting device 1 according to the present embodiment is mounted on a vehicle traveling on a road plane. Find the geometric relationship with the camera, detect obstacles on the road plane using the geometric relationship,
The detected obstacle is displayed.

That is, the obstacle detection device 1 includes two left and right TV cameras (left TV camera 2a) for capturing an image on a common plane in the three-dimensional space relating to the host vehicle, for example, a road plane in front of the traveling direction. , An image capturing unit 2 having a right TV camera 2b) and an image storing unit for storing (storing) an image (left image) captured by the TV camera 2a and an image (right image) captured by the TV camera 2b. 3 and
A feature extraction unit 4 for extracting two or more feature lines on the left image and the right image, a parameter calculation unit 5 for calculating a relational expression parameter based on the extracted feature lines, and a calculated relationship A detection unit that obtains points having different heights with respect to the road plane based on the equation parameters and detects the points as obstacles (obstacle areas);

In the present embodiment, as shown in FIG. 2, the two lines on the road plane are two white lines (l, l ') at both ends of the road plane, and the directions of the straight lines l, l' are A world coordinate system (absolute coordinate system) is set in which the X axis is a direction orthogonal to the Y axis along the road plane with respect to the Y axis.

The image photographing unit 2 includes two left and right TV cameras 2
The two images (left image IL,
The right image IR) is photographed.

The TV cameras 2a and 2b are mounted side by side at predetermined intervals, for example, along the X-axis direction of the world coordinate system, and their shooting directions are not limited to the front of the traveling vehicle. It may be in the rear or side direction. Further, by switching the focal length of each of the TV cameras 2a and 2b, it is possible to perform wide-angle shooting and telephoto shooting according to the condition of a road plane (road surface).

It is assumed that the two TV cameras 2a and 2b in the present embodiment have unknown positions and orientations with respect to the world coordinate system and only known epipolar constraints.
It is configured not to change during running.

Here, the epipolar constraint is a constraint that holds for a general stereo camera, and is set at an arbitrary point k on one image (the left image in this embodiment, which is also referred to as a reference image). (The right image in this embodiment, this image is also referred to as a reference image) on the other image (corresponding point) k ′ is a straight line on the right image IR as shown in FIG. Means being restrained on.

This straight line is called an epipolar line EL. The epipolar line EL is represented by a point k on the left image IL and the viewpoint of the left TV camera 2a (center point position of the lens).
Are projected on the right image IR and connected to each other.

For example, when the optical axes of the TV cameras 2a and 2b are arranged in parallel, the corresponding point on the right image IR of an arbitrary point on the left image IL is the same scan on the right image IR. Since they are on the line, the epipolar line and the scan line coincide.

The epipolar constraint is based on the relative position and orientation relationship between the stereo cameras (left TV camera 2a, right TV camera 2b) and the internal parameters of each camera 2a, 2b, ie, the focal length of each camera lens, The fact that the ebipolar constraint is invariable because it depends on the image origin and the like means that the relative positional relationship and the like of the stereo cameras 2a and 2b do not change during traveling.

The epipolar constraint is obtained as follows. Now, any of the corresponding point set of stereo image (left image IL, right image _{IR) (u i, v i} ), (u i ', v i')
(I = 1, 2,..., N)

(Equation 2) (However, F is a 3 × 3 matrix).

The matrix F can be obtained from eight or more sets of corresponding points, that is, N = 8 or more. That is, since the matrix F can be obtained using only a set of corresponding points, there is no need to prepare a point whose three-dimensional position is known with respect to the world coordinate system. The matrix F can be obtained by simple processing.

When the matrix F is obtained in this manner, a point corresponding to a certain point (u, v) on the left image IL becomes a straight line on the right image IR.

(Equation 3) It is above, and this is the shrimp polar line EL. That is, equation (2) represents the epipolar constraint. Here, F _ij (i, j = 1, 2, 3) is the i of the matrix F
Element of row j column.

The image storage unit 3 has an image memory, and stores two images (left images IL, IR) captured by the TV cameras 2a, 2b of the image capturing unit 2 in the image memory, respectively. It has become.

The feature extraction unit 4 has an arithmetic processor and a memory. First, on the two images (left image IL, right image IR) stored by the image storage unit 2, the feature extraction unit 4 Two straight lines (white lines) l and l ′ are extracted.

The above two straight line extractions are based on Hough
This is performed using conversion or the like. The straight lines l and l 'are left and right images IL,
On the IR, each has intersections (vanishing points) V and V ′,
By utilizing that V ′ is on the epipolar line Lv of V, the calculation amount can be reduced.

At this time, if any two points on the straight line 1 on the left image IL are A and C, and any two points on the straight line l 'are Β and D, the feature extracting unit 4 The corresponding points A ′, Β ′, C ′, D ′ on the right image IR of the point are obtained by using the epipolar constraint.

That is, the point A ′ corresponding to the point A is the right image I
On R, it can be obtained as the intersection of the epipolar line LA of the straight line 1 and the point A, and similarly, the points Β ′, C ′,
D ′ is also the shrimp polar line L of points Β, C and D, respectively.
It can be obtained as the intersection of B, LC, and LD.

Here, the coordinate data of the obtained points A, Β, C, D and points A ′, B ′, C ′, D ′ are respectively expressed as (u _A , v _A ), (u _B , v _B ), (U _C , v _C ),
_{(U D, v D),} (u 'A, v' A), (u 'B, v'
_B), and _{(u 'C, v' C} ), (u 'D, v' D).

The parameter calculation unit 5 has an arithmetic processing processor and a memory. Based on the correspondence between the four sets of points obtained by the feature extraction unit 4, the left image IL of an arbitrary point on the road plane is obtained. Upper projection point (u, v) and right image IR
A parameter of a relational expression (relational expression parameter) that is established between the above projected points (u ′, v ′) is calculated.

Now, the projection points of an arbitrary point P (X, Y) on the road plane onto the left and right images IL and IR are respectively (u,
v), (u ', v'),

(Equation 4) The following relational expression holds.

[0052]

[Outside 4]

[0053]

[Outside 5]

[0054]

[Outside 6]

(Equation 5)

[0055]

[Outside 7]

The detecting section 6 has an arithmetic processor and a memory, and has an arbitrary point P (u,
The brightness of v) is set to B _L (u, v), and this point P (u, v)
Point P assuming that exists on the road plane
The corresponding point P ′ (u, v) of (u, v) on the right image IR is obtained based on the above equation (4).

[0057] Now, the point obtained P '(u', v ' ) luminance _B R (u, v) of When the point P (u, v) if actually present on the road plane, the points Since P and P 'are a correct set of corresponding points, the brightness of the points P and P' is basically the same.

That is,

(Equation 6) In this case, the detection unit 6 determines that the point P satisfying D> Thr (Thr is a preset threshold) belongs to the obstruction obstacle area in consideration of D ≠ 0 or an error, and determines this point P as an obstacle. Can be detected as

[0059]

[Outside 8]

Therefore, the detection accuracy of an obstacle (obstacle detection area) can be improved, and it can contribute to further improvement of the safety and reliability of the vehicle.

For example, as shown in FIG. 5, an alarm generation process for performing an alarm generation process based on obstacle region information detected by the detection unit 6 and driving direction instruction information corresponding to, for example, a driver's steering operation of the own vehicle. By providing the unit 7, when the driver's steering wheel operation sends driving direction instruction information for directing the vehicle to the obstacle area (obstacle) to the alarm generation unit 7, the alarm generation unit 7 Based on the obstacle area information and the driving direction instruction information, it is determined that the host vehicle is traveling toward the obstacle area, and an alarm is issued.

As a result, the driver of the traveling vehicle can recognize that the own vehicle is traveling toward the obstacle, which contributes to further improvement of the safety and reliability of the vehicle. it can.

Further, as a modification of this embodiment, FIG.
As shown in the figure, the display 8 attached to a position where the driver in the own vehicle can be visually recognized and, for example, the left image IL stored in the image storage unit 3 are displayed on the display 8.
Based on the obstacle area information detected by the detection unit 6,
A display process for making the display mode of the safety region other than the obstacle region and the obstacle region different from each other with respect to the left image IL (for example, a process of setting the safety region to a green display color and the obstacle region to a red display color) Alternatively, an image processor 9 may be provided which performs a process of superimposing and displaying a marker representing an obstacle region on the coordinate position of the obstacle region on the left image IL.

With this configuration, the obstacle area and the safety area existing in front of the traveling vehicle can be clearly distinguished on the display 8. Existing obstacles and the like can be easily and reliably recognized, which contributes to further improvement of the safety and reliability of the vehicle.

In the present embodiment, two TV cameras 2a and 2b of the image photographing section 2 are mounted on the vehicle side by side, and two images are photographed by the TV cameras 2a and 2b. These two cameras 2a and 2b may be arranged in any manner as long as the two cameras 2a and 2b maintain the epipolar constraint and can capture a common plane relating to the vehicle. It is also possible to arrange three or more TV cameras on the vehicle.

When the failure detection processing of the present embodiment is performed using three or more TV cameras, the epipolar constraint uses one of three or more captured images as a reference image, This means that the corresponding point k ′ on the remaining images (reference images) corresponding to the arbitrary point k above is constrained on a certain straight line on each reference image.

Although the feature extracting unit 4 has determined the correspondence between the four sets of points on the left image IL and the right image IR, the correspondence may be determined between five or more sets in the same manner. In this case, the parameter calculation unit 5 can solve ten or more simultaneous equations obtained from the five or more sets of correspondences using the least square method or the like.

Further, the detecting section 6 can be further configured as shown in FIG. In this modified example, the detection unit 6 includes an image conversion unit 10 and a difference calculation unit 11, and the image conversion unit 10 and the difference calculation unit 11 are embodied as processing functions of an arithmetic processing processor.

The image conversion section 10 performs image conversion of the right image IR as a reference image according to the following procedure.

In general, an image uses points (u, v) on the image as variables, and a function f in which a luminance value is defined for each point.
It can be expressed as (u, v). Hereinafter, an image is represented in this manner.

For example, assume that a stereo image (left image, right image) including a preceding vehicle as shown in FIG.
Assuming that the left image is f (u, v), the right image is g (u, v), and the converted image of the right image is g ′ (u, v), the image conversion unit 10 calculates A converted image g '(u, v) is obtained.

[0072]

G ′ (u, v) = g (u ′, v ′) (7) where (u ′, v ′) is obtained by the above equation (4).

G '(u, v) is an image obtained by the left TV camera 2a assuming that an arbitrary point on the image g (u, v) exists on the road plane.

For example, from the right image g (u, v) shown in FIG. 9, a converted image g ′ (u ′, v ′) as shown in FIG. 9 is obtained. As shown in FIG. 10, the projection points of points on the road plane are the left image f (u, v) and the transformed image g ′ (u ′,
In contrast to v ′), points that are not on the road plane, that is, points on the obstacle (in this case, the preceding vehicle) are projected to different positions depending on the height from the road.

Therefore, the difference (inter-pixel difference) between the left image f (u, v) and the converted image g ′ (u ′, v ′)
, It is possible to detect an obstacle on the road plane.

That is, the difference calculation unit 11

(Equation 8) Is calculated, and a point (u, v) where D ′> 0 or D ′> Thr (Thr is a preset threshold value) is considered as an obstacle in consideration of an error. It is determined that they belong to the area, and this point (u, v) can be detected as an obstacle.

The difference detector of the detector 6 detects the difference between the two images (left image and right image) by calculating the difference between the pixels. However, the present modification is not limited to this. Absent.

For example, the difference detection unit of the detection unit 6 sets a (2ω + 1) × (2ω + 1) window for each point (pixel) corresponding to each image (left image, right image). It is also possible to detect the difference by calculating the normalized cross-correlation C of the luminance value of the.

In this case, two images (left image, right image)
The point (u, v) C of F (u, v) and G (u, v) is

(Equation 9) It is represented by

Here, N = (2ω + 1) × (2ω +
1), a1 and a2 are two images F (u, v), G (u,
The average brightness in the window of v), σ ² ₁ , σ ² _2, is
This is the variance of the luminance in the windows of the two images F (u, v) and G (u, v).

At this time, the difference calculator 11 calculates the calculated value C <
A point (u, v) satisfying Thr (Thr is a preset threshold value) is determined to belong to the obstacle area.

As described above, even if the detection unit is configured as shown in the modification, the obstacle area can be easily detected.

In this embodiment, the two white lines at both ends of the road plane are extracted as straight lines. However, when the road is curved, the white lines are curved. In this case, if the white line is extracted as a curve, an obstacle can be detected in the same manner.

Further, in the present embodiment, the description has been made assuming that the road surface is a plane. However, even in the case of a curved surface, an obstacle can be detected as in the case of a plane.

Further, in the present embodiment, it has been described that each of the block components 4 to 6 shown in FIG. 1 has an arithmetic processor and a memory, but the present invention is not limited to this. Instead, the configuration may be such that the feature extraction processing of the feature extraction unit 4, the relational expression parameter calculation processing of the relational expression parameter calculation unit, and the obstacle detection processing of the detection unit 6 are performed by one arithmetic processing processor and memory. It is.

In this embodiment, the detection of an obstacle from a TV camera mounted on a vehicle traveling on a road surface has been described. However, the present invention is not limited to this. For example, an aircraft or a helicopter may be used. It can also be applied to obstacle detection when a moving body such as lands on a road surface such as a runway.

On the other hand, according to the present embodiment, the case where two or more TV cameras are mounted on a moving body such as a vehicle has been described, but the present invention is not limited to this.

For example, as shown in FIG. 11, one TV camera 2 is mounted on a moving body (vehicle 15), and
By using the images of the road monitoring cameras 16 installed at, for example, predetermined intervals above the roadside of the road on which the vehicle 5 travels, it is also possible to perform the above-described obstacle detection processing based on the stereo vision technique.

That is, one T mounted on the vehicle 15
The obstacle detection device 1a having the V camera 2A receives the image of the road monitoring camera 16 installed near the traveling position via the wireless communication device 17, and receives this image (first image) and the own vehicle 15 By executing the processing based on the above-described stereoscopic vision technology based on the image (second image) captured by, it is possible to detect an obstacle on the road.

Then, the obstacle detecting device 1a
5, the continuous obstacle detection process is performed by using the first image transmitted from the monitoring camera 16 closer to the current traveling position of the host vehicle 15 via the wireless communication device 17 in accordance with the traveling of the vehicle 5. It can be carried out.

With this configuration, in addition to the above-described effects, the number of TV cameras mounted on the vehicle can be reduced, and the system of the obstacle detection device is simplified. When the size of the TV camera mounted on the vehicle and the size of the road monitoring camera are different, it is necessary to perform a correction process between the obtained first and second images.

In addition, any modifications can be made without departing from the gist of the present invention.

[0093]

As described above, according to the obstacle detecting method and apparatus of the present invention, an obstacle can be detected based on the presence or absence of a height from a surface such as a road surface. An obstacle such as a preceding vehicle or a pedestrian can be detected from the image without being affected by the shadow.

In particular, according to the present invention, the geometric relationship between the road surface and each photographing device is obtained by simple processing such as detection of a straight line. Obstacles on the surface can be detected stably,
Its practicality can be greatly improved.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of an obstacle detection device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a world coordinate system set on a road plane and a white line on the road plane.

FIG. 3 is a diagram for explaining epipolar constraint.

FIG. 4 is a view for explaining a white line extraction process and a process of obtaining a set of corresponding points between left and right images by the feature extraction unit shown in FIG. 1;

FIG. 5 is a block diagram showing an obstacle detection device having an alarm generation processing function in addition to the configuration of FIG. 1;

FIG. 6 is a block diagram showing an obstacle detection device having display processing of an obstacle area and a safety area for a display in addition to the configuration of FIG. 1;

FIG. 7 is a block diagram showing another configuration of the detection unit shown in FIG. 1;

FIG. 8 is a diagram showing an example of a stereo image.

FIG. 9 is a diagram showing a right image and its converted image.

FIG. 10 is a diagram showing a converted image of a left image and a right image.

FIG. 11 shows a modification of the present invention, in which one moving body is provided with one TV.
The figure which shows the structure which mounts a camera and performs stereoscopic vision using the camera for road monitoring installed in the road side of the road.

[Description of Signs] 1 Obstacle detection device 2 Image photographing unit 2a, 2b TV camera 3 Image storage unit 4 Feature extraction unit 5 Relational expression parameter calculation unit 6 Detection unit 7 Alarm generation unit 8 Display 9 Image processing processor 10 Image conversion unit 11 Difference calculator

[Procedure amendment 2]

[Document name to be amended] Drawing

[Correction target item name] Fig. 1

[Correction method] Change

[Correction contents]

FIG.

[Procedure amendment 3]

[Document name to be amended] Drawing

[Correction target item name] Figure 2

[Correction method] Change

[Correction contents]

FIG. 2

[Procedure amendment 4]

[Document name to be amended] Drawing

[Correction target item name] Figure 3

[Correction method] Change

[Correction contents]

FIG. 3

[Procedure amendment 5]

[Document name to be amended] Drawing

[Correction target item name] Fig. 4

[Correction method] Change

[Correction contents]

FIG. 4

[Procedure amendment 6]

[Document name to be amended] Drawing

[Correction target item name] Fig. 5

[Correction method] Change

[Correction contents]

FIG. 5

[Procedure amendment 7]

[Document name to be amended] Drawing

[Correction target item name] Fig. 6

[Correction method] Change

[Correction contents]

FIG. 6

[Procedure amendment 8]

[Document name to be amended] Drawing

[Correction target item name] Fig. 7

[Correction method] Change

[Correction contents]

FIG. 7

[Procedure amendment 9]

[Document name to be amended] Drawing

[Correction target item name] Fig. 8

[Correction method] Change

[Correction contents]

FIG. 8

[Procedure amendment 10]

[Document name to be amended] Drawing

[Correction target item name] Fig. 9

[Correction method] Change

[Correction contents]

FIG. 9

[Procedure amendment 11]

[Document name to be amended] Drawing

[Correction target item name] FIG.

[Correction method] Change

[Correction contents]

FIG. 10

[Procedure amendment 12]

[Document name to be amended] Drawing

[Correction target item name] FIG.

[Correction method] Change

[Correction contents]

FIG. 11

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Kazunori Onoguchi 8-6-26 Motoyama Minami-cho, Higashinada-ku, Kobe-shi, Hyogo F-term in Toshiba Kansai Research Institute Co., Ltd. 5B057 AA06 BA02 BA15 CE16 DA08 DA15 DA16 DC03 DC22 DC34 5H180 AA01 CC04 CC24 LL01 LL02 LL08

Claims (12)

[Claims] 1. An obstacle detecting apparatus for detecting an obstacle area on a plane having a plurality of lines, comprising: a plurality of photographing devices for photographing images; and a plurality of images photographed by the plurality of photographing devices, respectively. And a plurality of lines extracted from the plurality of images stored in the image storage unit, and a corresponding point between the plurality of images is determined based on the plurality of extracted lines. Extraction means; parameter calculation means for calculating parameters of a relational expression between projection positions of arbitrary points on the surface on each image based on the corresponding points obtained by the extraction means; Detecting means for detecting areas having different heights with respect to the surface as obstacle areas based on a relational expression determined by parameters calculated by the means. Detection device. 2. An image photographed by one reference photographing device of the plurality of photographing devices as a reference image,
When an image photographed by the remaining photographing device is used as a reference image, a corresponding point on the reference image corresponding to an arbitrary point on the reference image is an arbitrary point on the reference image and the corresponding point on the reference photographing device. 2. The system according to claim 1, wherein a straight line connecting a viewpoint is constrained on a straight line projected onto the reference image.
An obstacle detection device as described in the above. 3. The method according to claim 1, wherein the extracting unit calculates a set of corresponding points between the reference image and the reference image based on a plurality of lines on the reference image and a straight line projected on the reference image with respect to the reference image. Wherein the parameter calculation unit is configured to determine, for the reference image and the reference image at an arbitrary point on the surface, based on a set of a plurality of corresponding points between the reference image and the reference image obtained by the extraction unit. 3. The obstacle detection device according to claim 2, wherein the means for obtaining a parameter of a relational expression that holds between the projection positions. 4. The method according to claim 1, wherein the extracting unit determines an intersection between a first plurality of points on one line 1 of the plurality of lines on the reference image and a projection line on the reference image. And the intersection of a second plurality of points on one line other than one of the plurality of lines on the reference image with a projection line on the reference image is defined as a second plurality of corresponding points. And the second set of the first plurality of points and the first plurality of corresponding points
4. The obstacle detecting device according to claim 3, wherein a plurality of points and a set of a second plurality of corresponding points are obtained. 5. The method according to claim 1, wherein the detecting unit determines a corresponding point on the reference image corresponding to the arbitrary point on the reference image, assuming that the arbitrary point on the reference image exists on the surface. By comparing the brightness of an arbitrary point on the reference image with the calculated brightness of the corresponding point on the reference image, an arbitrary point on the reference image can be obtained based on the relational expression determined by 5. The obstacle detecting device according to claim 3, wherein it is determined whether the object belongs to an obstacle region having a different height. 6. The converting means for converting the reference image into an image obtained by the reference photographing device, assuming that an arbitrary point of the reference image exists on the plane, Calculate the difference between the reference image and the reference image converted by the conversion means, and based on the calculated difference, any point on the reference image is in an obstacle area having a different height with respect to the plane. The obstacle detection device according to claim 3, further comprising a difference calculation unit that determines whether the object belongs. 7. The obstacle detection device according to claim 6, wherein the difference calculation means calculates the difference by taking a difference between the reference image and the converted reference image. 8. The difference calculation means sets a window centered on an arbitrary point on the reference image, and sets a luminance between the reference image and the converted reference image in the set window. 7. The obstacle detecting device according to claim 6, wherein the difference is calculated by calculating a normalized cross-correlation of the value. 9. The vehicle according to claim 1, wherein the surface is a road surface on which a moving body moving in a real space travels, and the obstacle detection device is mounted on the moving body.
The obstacle detection device according to the item. 10. When the movement instruction information for moving the moving body with respect to the obstacle area detected by the detection means is transmitted, the moving body is configured to avoid moving the moving body to the obstacle area. 10. The obstacle detecting device according to claim 9, further comprising means for performing a predetermined process. 11. A display unit for displaying at least one image stored by the image storage unit in a manner different from a display mode of another safety area on the image, the obstacle area detected by the detection means. The obstacle detection device according to claim 10, further comprising: 12. An obstacle detection method for detecting an obstacle area on a plane having a plurality of lines, comprising: a step of photographing a plurality of images; and a step of storing a plurality of images photographed in the photographing step. Extracting the plurality of lines represented on the plurality of images stored by the storage step, and obtaining a corresponding point between the plurality of images based on the extracted plurality of lines; Calculating the parameters of a relational expression that holds between the projection positions of the arbitrary points on the surface onto the respective images based on the corresponding points; and Detecting a region having a different height with respect to the surface as an obstacle region.