JP2001331787A - Road shape estimating device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To secure sufficient shape estimating precision and reliability in a road shape estimating device mounting only a monocular camera on a vehicle as an image pickup device for picking up the image of the foreground. SOLUTION: This road shape obtaining means 1 is for extracting the shape of a road in an image and consists of an intra-screen road shape extracting means for extracting a white line and a road end by processing the image. A method for extracting a white line can be a method for converting a front image to be monochromatic to obtain the change of luminance by a differential filter to extract a part where luminance changes remarkably, as the white line. A road map obtaining means 1 is for extracting the coordinate point of the road end on which its own vehicle travels from the obtained map information and consists of a road shape in a map surface extracting means. When estimating a three-dimensional road shape by using image road information and map road information, the projection error of the image is excluded, thereby the shape of the road can be estimated more precisely than it is estimated by single information. A road shape estimating device like this is useful for judging a precedent vehicle and automatic navigation, e.g.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a road shape (or a road shape) extracted from a monocular camera image of a foreground of a vehicle.
By comparing the road shape estimated from the radar signal of the radar) with the road shape of a normal digital road map,
The present invention relates to a road shape estimating device that estimates a rough shape of a road or a lane during traveling. In addition, the present invention provides, for example, an adaptive control that performs speed control such as acceleration control and brake control.
Useful for cruise control systems and the like.

[0002]

2. Description of the Related Art An apparatus for estimating the shape of a running road and the position and orientation of a host vehicle is described in, for example, Japanese Patent Laid-Open Publication No. Hei 7-27541: Measuring Device for Road Shape and Host Vehicle Position. Some devices are generally known. In the related art (hereinafter, sometimes referred to as “conventional technology 1”), the position and orientation of the host vehicle are estimated by comparing the road shape with the map information and the image information.

[0003] In the prior art 1, a road shape obtained from an image and map information including data corresponding to a three-dimensional road shape are used. That is, the prior art 1 detects a road shape from an image obtained by photographing a road in the traveling direction of a vehicle, projects a three-dimensional road shape obtained from map information on an image plane, and outputs a road shape based on a map and a road shape of the image. And adjust the position and orientation parameters (camera height, yaw angle, pitch angle, roll angle, distance from the road edge, etc.) of the own vehicle so that both overlap. Are sequentially estimated.

The above-mentioned map information includes horizontal curvature, vertical curvature,
Data indicating a three-dimensional shape of a road such as a bank angle is stored in advance, and according to the related art 1, a three-dimensional road shape model of a forward road can be calculated based on the vehicle position data. . That is, in the prior art 1, the road shape is three-dimensionally stored in advance in the map information, and the road shape is basically restored using the map.

[0005] As another apparatus for recognizing a road shape with a small calculation load, for example, an apparatus described in Japanese Patent Laid-Open Publication No. 11-203458: Road Shape Recognition Apparatus is generally known. I have. In this conventional technology (hereinafter, sometimes referred to as “prior art 2”), a target object existing on a road such as a reflector whose size and installation height are known is detected from an image obtained by a camera. I do. Further, the position of the target object on the image when the road surface is flat is stored, and the height of the target object detected as needed from the image at a predetermined size is compared with the stored height.

As a result of this comparison, if the difference between the detected position and the stored position is within a predetermined range, it is determined that the road ahead is flat, and if the detected position is below the stored position, the road is downhill and the detected position is the stored position. If it is higher, it is determined to be uphill. According to the related art 2, it is not necessary to estimate the road gradient from the shape of the white line, the road gradient can be easily detected, and by comparing the position of the target object, the road shape such as a curve or a straight line can be detected. Can be detected.

[0007]

The above prior art 1 has the following problems. (Problem 1) The map information used in the above-described conventional technology 1 includes three-dimensional road shape information such as information on longitudinal curvature, but recently, most of the map information including the three-dimensional road shape is other than the expressway. Is difficult to obtain, and the creation of a new map including three-dimensional road shape information is enormous and costly. That is, the above-mentioned prior art 1 basically supplements image information using a high-accuracy map, and uses a map having an accuracy of about several meters, which is currently used in car navigation systems and the like. It is not possible.

More specifically, for example, map data having three-dimensional data such as the vertical curvature of a road is generally difficult to obtain at present, but even if relatively detailed three-dimensional data is available, In order to actually determine the preceding vehicle 50 m away, for example, in the case of the prior art 1, the accuracy of the map road shape needs to be at least about 0.3 m in both the horizontal and vertical directions. However,
At present, the accuracy of a generally available map in the height direction is limited to about several meters. Therefore, in order to secure sufficient road shape restoration accuracy according to the above-described related art 1, a highly accurate map must be obtained. They need to be created, which requires enormous costs.

Further, the above-mentioned prior art 2 has the following problems. (Problem 2) In the above-described related art 2, gradient estimation is performed on an object whose size, installation height, and the like are known in advance, but in this method, the environment where the object to be detected is always present is considered. Is good, but in an environment where there is no object to be detected, it is difficult to estimate the distance and the height, and it is difficult to estimate the gradient. Further, in the above-described conventional technology 2, since the gradient is estimated by performing the object detection based on the image obtained from the camera, it is difficult to obtain information on the road ahead by the camera in bad weather or the like. Therefore, in such a case, It is difficult to perform forward gradient estimation and the like.

When the road ahead has a longitudinal curvature or a transverse curvature, the road shape is projected on a plane by using an image.
There is a possibility that the actual road shape is greatly different from the actual road shape due to a projection error, and it is difficult to estimate the road shape with the accuracy required for the preceding vehicle determination with the currently available map accuracy.

That is, when estimating a road shape on a plane from a shape obtained by projecting a road shape extracted on the screen onto a plane and a road shape obtained from a map, when a road ahead has a longitudinal curvature or a transverse curvature, In addition, the projection error may be very large, and it is difficult to accurately estimate a distant shape using only image data. In addition, the accuracy of a generally available road map and the position of the own vehicle is currently about several meters, and it is not possible to estimate with high accuracy only information on the map.

In particular, regarding the own vehicle position estimation, for example, a global position estimation of the own vehicle is performed by detecting a landmark by an image and aligning the landmark with a landmark described on a map. Other prior art is also known. However, such a method cannot perform position estimation in bad weather or in a place where no landmark exists.

There are GPS and dead reckoning as ordinary position estimating methods that do not use special map information. However, the positioning accuracy of GPS is limited, positioning intervals are limited, and dead reckoning also has a large accumulated error. It is difficult to perform highly accurate position estimation.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and has as its object to improve the shape estimation accuracy of a vehicle-mounted device for estimating a rough road shape or a lane (lane) shape, Provided is a road shape estimating device capable of ensuring sufficient shape estimation accuracy and reliability even when only a monocular camera is mounted as an imaging device for imaging a foreground. The purpose is to do.

[0015]

In order to solve the above-mentioned problems, the following means are effective. That is, the first means includes an imaging device that acquires a foreground of the own vehicle as image data, and a vehicle-mounted road shape estimating device that estimates a shape of a road on which the own vehicle is running or is expected to run,
Image road shape extracting means for extracting a first road shape candidate from image data, map data obtaining means for obtaining map data around the own vehicle, and a map for extracting a second road shape candidate from the map data Road shape extracting means, and projecting the first shape candidate and the second shape candidate into one three-dimensional or two-dimensional logical space, and an overlapping state of the two road shapes projected in the logical space And a road shape estimating means for estimating the shape of the road, the attitude of the own vehicle with respect to the road surface, or the absolute position of the own vehicle with respect to the road based on the above.

Further, a second means is the road shape estimating means of the first means, wherein the attitude of the own vehicle with respect to the road surface is estimated in advance, or the absolute position of the own vehicle with respect to the road is determined. This is to provide a vehicle position estimating means for estimating in advance.

Further, the third means may be the first or the second means.
The road shape estimating means is provided with a matching means for stochastically evaluating the overlapping state for each point of the first shape candidate and the second shape candidate that should correspond to each other.

Further, the fourth means includes the first to third means.
The road shape estimating means of any one of
A reliability calculating means for calculating the reliability R1 of the shape candidate and the reliability R2 of the second shape candidate for each point of the first shape candidate and the second shape candidate which should correspond to each other;
An internal dividing point calculating means for calculating an internal dividing point between the two points by a coordinate weighted average calculation for each point based on R2.

Further, the fifth means includes the first to fourth means.
In any one of the means, the inconsistency in the positional relationship between the representative points constituting the shape of the road estimated by the road shape estimating means is eliminated based on predetermined physical constraints, so that an appropriate An appropriate road shape extracting means for extracting a road shape is provided.

Further, the sixth means comprises the first to fifth means.
In the road shape estimating means of any one of the above, the camera viewpoint of the imaging device is set as the projection center (projection origin),
Projection means for projecting a shape candidate onto the logical space is provided, and the camera coordinate system and the map coordinate system are aligned based on the position and orientation of the host vehicle.

Further, the seventh means includes the first through sixth means.
In any one of the means, an estimation area extending means for estimating the shape of the road in the blind spot area of the imaging device is provided based on the estimation result of the road shape estimation means and the second shape candidate.

Further, the eighth means has a radar for searching ahead of the own vehicle to obtain a radar signal, and the vehicle-mounted road for estimating the shape of the road on which the own vehicle is running or expected to run. In the shape estimating device, a position of an object ahead of the own vehicle is detected from the radar signal, and road alignment estimating means for estimating a road alignment of the road based on the arrangement of the objects is provided. An image capturing apparatus that acquires the first road shape candidate from the image data ”or“ a map data acquiring unit that acquires map data around the own vehicle ”; At least one of a map road shape extracting means for extracting shape candidates from map data, and the first shape candidate or the second shape candidate and the road alignment are configured in three dimensions or two dimensions. And a road shape estimating means for estimating the shape of the road in the logical space based on an overlapping state in the logical space between the projected road shape and the road alignment. It is.

The ninth means has an imaging device for acquiring the foreground of the own vehicle as image data and a radar for exploring the front of the own vehicle to obtain a radar signal. Then, an on-vehicle road shape estimating device for estimating a predicted road shape, an image road shape extracting means for extracting a first shape candidate of the road from the image data, and a distance from a radar signal to an object ahead of the own vehicle. Distance estimating means for estimating the height of the road, road surface height estimating means for estimating the road surface height of the road from the image data based on the distance to the object, and the first shape candidate and the road surface height of the road ,
Road shape estimating means for estimating the three-dimensional shape of the road. With the above means, the above-mentioned problem can be solved.

[0024]

The operation of the present invention will be described below. The image road shape extraction means of the present invention can detect a road edge by detecting a white line, for example, and acquire a road alignment from the detected white line set. In the map road shape extraction means,
The road shape (road alignment) is extracted from the map around the acquired vehicle position. For example, if the map is provided with road lane markings, the road alignment can be obtained by extracting the center of the lane marking set, and if node data for a navigation system is provided, The road alignment can be obtained by connecting the nodes.

Further, according to the road shape estimating means of the present invention, for example, the obtained image road shape is projected onto one two-dimensional logical space with the camera viewpoint as the projection center, and this operation is performed. The obtained map road plane shape can be further vertically projected onto a three-dimensional logical space. In this three-dimensional logical space, the image road shape has certain candidate regions, and the above-described map road shape also has certain candidate regions. According to the road shape estimating means of the present invention, for example, both candidate regions are superimposed, and the road shape position having the highest matching degree can be output as the estimated road shape.

(Function 1) For example, according to the above-described means of the present invention, by performing estimation using a plurality of road shape information (image data, map data), the reliability of the estimation result is improved, Even in a situation in which the estimation error becomes large only by the image processing, by estimating the road shape having a high degree of matching by superimposing the map road shape and the image road shape, accurate road shape estimation becomes possible.

(Function 2) In addition, three-dimensional or two-dimensional data is used according to each numerical value of a predetermined parameter group representing the position and orientation of the host vehicle.
Since the projection result on one logical space composed of the dimensions changes, the estimated road shape also changes. Therefore, by changing the position and orientation of the own vehicle so that the degree of collation becomes higher by the means of the present invention, the estimated values of the position and attitude of the own vehicle and the road shape can be more accurately obtained.

(Function 3) The road shapes of the image and the map are projected into a three-dimensional space, and calculated as the existence probability in the three-dimensional space in consideration of the overlap between the two. By originally estimating the road shape, it is possible to eliminate a projection error generated in the plane estimation and to increase the estimation accuracy. Further, since the road shape is estimated in the three-dimensional space, it is possible to estimate the gradient of the road ahead.

(Function 4) In addition, a situation where an invisible area is generated on a road ahead due to a large longitudinal curvature or a transverse curvature of the road, that is, a road shape such as a pass or a curve, or a blind spot area generated by a preceding vehicle or the like. In such a situation that an area where image data cannot be obtained from the imaging device occurs,
By using map information, it is possible to estimate the road shape,
The reliability of the estimation result of the road shape is improved. Furthermore,
By superimposing (collating) the road shape estimated in the visible range (the range in which the image data could be obtained) with the map information and extending the road shape, the road shape in the invisible range beyond that is also predicted Then, it becomes possible to estimate the road shape with relatively high accuracy compared to the case of estimating the map alone.

(Function 5) In estimating the road shape close to the host vehicle, image information is effective because the projection error is small. For estimating the distant road shape, the error in the map information is the distance from the host vehicle. Therefore, the map information is effective because it is almost constant regardless of the above. By changing the weights of the image and the map in accordance with the distance from the camera and matching both information, the estimation accuracy of the road shape is higher than in the case of individually estimating each.

(Function 6) According to the eighth means of the present invention, the arrangement of the detected objects and their temporal position changes based on the results of the distance, direction and speed of the objects detected by the radar. A rough road alignment of the road ahead can be estimated by a change in the position of the own vehicle or the like. Since the road alignment can be estimated only by the radar, it is possible to robustly estimate the road shape even in a situation where external environmental conditions are poor.

(Function 7) According to the ninth means of the present invention, the distance to the obstacle ahead can be measured by detecting the object by the radar, and the distance to the object becomes clear. Since the distance can be detected, ambiguity in the distance and height directions of information obtained in the image can be reduced, and the three-dimensional position of the object in the image can be clarified. Since the position of the object existing on the road is known, the height of the road surface can be estimated, and the three-dimensional shape of the road ahead can be estimated even when there is no map information.

[0033]

(Effect 1) (a) Depending on the shape of the road ahead, even in a situation where the road surface in front is not visible in the image or in a situation where the estimation error becomes large when the estimation is made on the assumption of a horizontal plane in the monocular image, the image can be reproduced. By using other information such as maps, the road shape and the position of the vehicle
The posture can be estimated with high accuracy (the reliability of the estimation result) and robustly without being affected by disturbance or the like. (B) Also, based on the map road shape and the image road shape,
By estimating the road shape and estimating the attitude of the host vehicle with respect to the road surface, it is expected that the accuracy of the host vehicle attitude estimation is improved as compared with the attitude estimation based on the image road shape alone.

(C) Further, by estimating the road shape based on the map road shape and the image road shape and estimating the absolute position of the vehicle with respect to the map, special landmarks and three-dimensional information are described. Even without a map, the position of the host vehicle can be estimated. (D) In addition, GPS and INS (Inertial Navigation Sy
It can be expected that the estimation accuracy is improved as compared with the estimation accuracy by the stem). (E) Further, since the accuracy of estimating the posture and position of the vehicle is improved, the accuracy of estimating the road shape can be expected to be improved.

(Effect 2) (a) The road shape estimating means can project each road shape into a three-dimensional space even in a situation where sufficient estimation accuracy cannot be obtained with only one of an image and a map. By superimposing, it is possible to estimate the road shape three-dimensionally and to increase the road shape estimation accuracy. Therefore, it is possible to estimate the road gradient, which is useful for speed control and the like.

(B) In addition, road shapes such as curves and passes,
Even when an area (such as a blind spot area) that cannot be detected by image data occurs due to a preceding vehicle or the like, a road in an area (such as a blind spot area) that cannot be seen in an image is determined based on the estimated road shape or a road shape obtained from a map. The shape can be estimated. Therefore, the road shape can be accurately and robustly estimated, and the preceding vehicle determination and the automatic steering can be realized.

(C) That is, if the estimation area expanding means of the present invention is used, for example, even in a range where image information cannot be obtained by the three-dimensional road shape estimating means, data of a visible area such as image data can be obtained. It is also possible to realize a three-dimensional road shape estimating means for estimating a road shape based on a three-dimensional road shape estimated based on the above (and / or a road map shape obtained from a map: a second shape candidate).

(D) The above three-dimensional road shape estimating means compares the estimated road with a real road model, so that the road shape can be more accurately estimated. .

(Effect 3) (a) The logical space in the road shape estimating means can be constituted by a two-dimensional space.
Such a road shape estimating means (hereinafter sometimes referred to as “in-plane road shape estimating means”) also generates a road shape from a road shape obtained from an image projected on a plane and a road shape obtained from a map. It is possible to increase the shape estimation accuracy. Therefore, in particular, by configuring the logical space in two dimensions, that is, not performing shape estimation in a three-dimensional space, but performing shape estimation in a certain plane, the amount of calculation required for the road shape estimation process is increased. Can be reduced.

(B) When the estimated area expanding means of the present invention is used, image information can be obtained by the above-mentioned road shape estimating means in the same manner as in the case of the three-dimensional road shape estimating means, for example. Even in a non-existent range, the in-plane road shape estimated based on the visible region data such as image data (and / or the road map shape obtained from the map: second shape candidate)
It is also possible to realize an in-plane road shape estimating means for estimating the road shape based on.

(C) Further, the above-mentioned in-plane road shape estimating means compares the estimated road with a real road model, so that the road shape can be more accurately estimated. .

(Effect 4) (a) According to the eighth means of the present invention, when image information cannot be obtained due to bad weather, or when the position of the own vehicle such as in a tunnel or under a highway cannot be obtained, etc. Also, it is possible to robustly estimate the shape of a road by using radar.

(B) Further, according to the ninth means of the present invention, the distance and height of the object detected in the image become clear from the result of distance measurement to the object by the radar. As a result, the three-dimensional position of the object in front can be estimated without using a stereo camera, so that the three-dimensional shape of a road can be estimated without using a map.

[0044]

DESCRIPTION OF THE PREFERRED EMBODIMENTS By using the above-mentioned means of the present invention,
For example, as in the following [Main Configuration Example 1] to [Main Configuration Example 20], a road shape estimating apparatus and the like include “an imaging device that acquires a foreground of the own vehicle as image data, and the own vehicle has The main configuration of the "vehicle shape estimating device for vehicle that estimates the shape of a road that is running or is expected to run" can be configured. According to these configurations (basic embodiments), the above-described functions and effects of the present invention can be obtained.

[Principal Configuration Example 1] An imaging device for acquiring road information as an image, an image road shape extracting means for extracting a road shape in a screen from the acquired image, and map data for acquiring map data around the own vehicle. An acquisition unit, a map road shape extraction unit that extracts a road shape on a map on which the vehicle is expected to travel from the acquired map data, and a map road shape based on the extracted image road shape and the map road shape. A road shape estimating means for estimating the “road shape” and the “position / posture of the own vehicle with respect to the road” so that the degree of collation is calculated by projecting onto the logical space and calculating the degree of collation based on the shape overlap; And a road shape estimating device that outputs “road shape”.

[Main Configuration Example 2] [Main Configuration Example 1]
A road shape estimating device, comprising: the image pickup device described in 1 above, an image road shape extracting unit, a map data acquiring unit, a map road shape extracting unit, and a road shape estimating unit, and outputs the “posture of the own vehicle with respect to the road surface”.

[Main Configuration Example 3] [Main Configuration Example 1]
A road shape estimating device, comprising: the image pickup device described in 1 above, an image road shape extracting unit, a map data acquiring unit, a map road shape extracting unit, and a road shape estimating unit, and outputs the “absolute position of the vehicle relative to the map”.

[Main Configuration Example 4] [Main Configuration Example 1] and / or [Main Configuration Example 2] and / or
A road shape estimating device in which [Main Configuration Example 3] is arbitrarily combined.

[Main Configuration Example 5] [Main Configuration Example 1]
-In the road shape estimating apparatus according to [Main Configuration Example 4], instead of the image pickup device and the image road shape extracting means, or the map data obtaining means and the map road shape extracting means, the position and speed of an object existing in the vicinity are determined. A radar device for detecting,
A road shape estimating apparatus having a radar road shape extracting means for estimating a road shape from a sequence of detected objects.

[Main Configuration Example 6] [Main Configuration Example 1]
In the road shape estimating apparatus according to any one of the first to fifth main configuration examples, a self-vehicle posture estimating unit that preliminarily estimates the “posture of the own vehicle with respect to the road surface” based on the image road shape obtained by the image road shape extracting unit is A road shape estimating apparatus that has a self-vehicle attitude estimation result with respect to a road surface and inputs the result to a road shape estimating means to simplify collation.

[Main Configuration Example 7] [Main Configuration Example 1]
In the road shape estimating apparatus according to any one of [Main Configuration Example 5], based on the image road shape acquired by the image road shape extracting means and the map road shape acquired by the map road shape extracting means, the “ A road shape estimating device which has own vehicle position estimating means for preliminarily estimating "absolute position with respect to the map", and simplifies matching by inputting the own vehicle position estimating result for the map to the road shape estimating means.

[Main Configuration Example 8] [Main Configuration Example 6]
A road shape estimating device that combines the above and [Main Configuration Example 7].

[Main Configuration Example 9] [Main Configuration Example 1]
-"Road shape" estimated by the road shape estimating means in the road shape estimating device described in [Main Configuration Example 8];
A part or all of the estimation results of the "attitude of the host vehicle relative to the road surface" and the "absolute position of the host vehicle relative to the map" are compared with a road model that satisfies the physical shape constraints of the real world. A road shape estimating device that has a road shape model matching unit that outputs a shape to be matched most as a road shape, and that enables more accurate road shape estimation.

[Principal Configuration Example 10] A front image capturing device for acquiring a forward image, road image acquiring means for extracting a road shape on an image plane from the acquired forward image, and a host vehicle for acquiring the position of the host vehicle A position acquisition unit, a road map acquisition unit that acquires a road shape on a map around the acquired own vehicle position, and a pitch angle with respect to a road surface, a yaw angle of a camera with respect to the map, and a lateral angle, based on the acquired road image and the road map. Position, traveling direction position, own vehicle attitude estimating means for estimating the attitude of the own vehicle such as the inclination angle, and projecting the road shape to a three-dimensional space from the obtained road image and the attitude of the own vehicle with respect to the road map and the map. A road shape estimating device comprising: a three-dimensional road shape estimating unit that calculates an existence probability and extracts an appropriate three-dimensional road shape candidate.

[Main Configuration Example 11] [Main Configuration Example 1]
[0], the road image obtaining means includes an in-screen road shape extracting means for performing image processing on the obtained forward image and extracting shapes such as white lines and road edges. Means for processing the acquired map information around the own vehicle and extracting a road shape indicating a road edge on which the own vehicle travels.

[Main Configuration Example 12] [Main Configuration Example 1]
The road image acquiring means according to 1), further comprising: an in-screen road shape extracting means for extracting a white line or the like; and an in-screen road shape estimating means for estimating a linear shape of a road in the screen from the extracted in-screen road shape. In the road map acquisition means described in [Main Configuration Example 11] above, the in-map road shape extraction means for extracting the road shape on which the vehicle travels, and the estimation in the image plane from the extracted map road shape are performed. A road shape estimating means for estimating a road alignment at a lateral position equivalent to the determined alignment.

[Main Configuration Example 13] [Main Configuration Example 1]
0], the camera coordinate system and the map coordinate system are aligned based on the acquired road image, the acquired road map, and the attitude of the own vehicle with respect to the map estimated by the own vehicle attitude estimating means. And a road shape existence probability calculating means for calculating an existence probability from an overlap of a candidate region of a road image projected to a three-dimensional space from a camera viewpoint and a candidate region of a road map projected to a three-dimensional space from a map surface. Has a proper road shape extracting means for extracting a proper road shape in consideration of the existence probability and physical constraints of the real world,
A road shape estimation device for estimating a three-dimensional road shape in a visible region.

[Principal Configuration Example 14] A front image capturing apparatus for obtaining a front image, road image obtaining means for extracting a road shape on an image plane from the obtained front image, and a host vehicle for obtaining the position of the host vehicle A position obtaining means, a road map obtaining means for obtaining a road shape on a map around the obtained own vehicle position, and a yaw angle of a camera with respect to the map, a pitch angle with respect to a road surface, and a lateral angle, based on the obtained road image and the road map. Self-vehicle posture estimating means for estimating the posture of the own vehicle such as the position, the traveling direction position, etc., and estimating the shape of the front road in one plane from the obtained road image, the road map, and the posture of the own vehicle with respect to the map. A road shape estimating device comprising: an in-plane road shape estimating means.

[Main Configuration Example 15] [Main Configuration Example 1]
The road image obtaining means of [4] includes in-screen road shape extracting means for performing image processing on the obtained front image and extracting shapes such as white lines and road edges. Means for processing the acquired map information around the own vehicle and extracting a road shape indicating a road edge on which the own vehicle travels.

[Main Configuration Example 16] [Main Configuration Example 1]
5] The road image acquiring unit according to 5), further comprising: an in-screen road shape extracting unit for extracting a white line or the like; and an in-screen road shape estimating unit for estimating a line shape of a road in the screen from the extracted in-screen road shape. Further, in the road map obtaining means described in [Main Configuration Example 15] above, an in-map road shape extracting means for extracting a road shape on which the own vehicle travels, and an in-image road shape extracting method based on the extracted in-map road shape. A road shape estimating device comprising: an in-map road shape estimating means for estimating a road alignment at a lateral position equivalent to the estimated alignment.

[Main Configuration Example 17] [Main Configuration Example 1]
4] The in-plane road shape estimating means aligns the camera coordinate system and the map coordinate system based on the result estimated by the own-vehicle attitude estimating means, and maps the road shape projected onto a certain plane from the camera viewpoint and a map superimposed on the road shape. Shape estimating device for estimating the road shape in the plane of the visible area from the road shape according to.

[Principal Configuration Example 18] An imaging device for acquiring an image of a road ahead, a radar device for detecting an object on the road ahead and measuring the distance, and extracting a road shape from image data by image processing such as white line extraction. An in-plane road shape acquiring means for detecting an object in the image plane, an in-plane object detecting means for detecting an object from the image data, and an object obtained in the radar by comparing the object obtained in the image with the object obtained in the radar. Object position estimating means for estimating the position of the object from the distance and the possibility area of the object obtained from the image, road surface height estimating means for estimating the height of the road surface from the position of the object, and the estimated road surface height And a road shape estimating means for estimating a three-dimensional shape of a forward road based on a road shape in an image plane.

[Principal Configuration Example 19] An imaging device for acquiring an image of a road ahead, a radar device for detecting an object on the road ahead and measuring the distance, azimuth, and speed, and an image such as white line extraction from image data. An in-plane road shape extracting means for extracting a road shape by processing, a road alignment estimating means for estimating a road alignment from an arrangement of objects detected by radar,
Based on the road shape in the image plane and the road alignment by radar,
A road shape estimating device having road shape estimating means for estimating a three-dimensional shape of a road ahead.

[Principal Configuration Example 20] A radar device that detects an object on a road ahead and measures the distance, azimuth, and speed of the vehicle, own vehicle position obtaining means for obtaining the position of the own vehicle, A road map shape obtaining means for obtaining a road map, a road alignment estimating means for estimating a road alignment from an array of objects detected by radar, and a road shape obtained by the map and a road shape obtained by radar are superimposed. A road shape estimating device having a road shape estimating means for estimating a shape of a front road in a plane.

For example, according to the main configuration as described above, the operation and effect of the present invention can be obtained. Less than,
An outline of a basic and representative embodiment of the present invention will be illustrated more specifically. The basic and representative embodiments exemplified here have common points or relations in the basic idea with more detailed and specific examples described later.

(Basic and Representative Embodiment) A road shape estimating apparatus according to the present invention will be described with reference to the drawings. FIG. 1 schematically shows the basic components of the present invention. An imaging device for acquiring information on a road ahead as an image, an image road shape extracting unit for extracting a road shape from the acquired image, a map data acquiring unit for acquiring map data around the own vehicle, and the acquired map data Map road shape extraction means for extracting the road shape of the map from the map road shape, and projecting the acquired image road shape and the map road shape into one logical space composed of three or two dimensions, A road that estimates a position having a high degree of matching among the candidate regions of the shape and the map road shape as the road shape, and simultaneously performs the estimation of the road shape, the posture estimation of the own vehicle with respect to the road surface, and the position estimation of the own vehicle with respect to the map. A road shape estimating device that has a shape estimating unit and outputs a road shape.

As a result of this output, the attitude of the own vehicle may be output as shown in FIGS. 2 and 3, the position of the own vehicle may be output, and the road shape and / or the own vehicle may be output. Any combination of various information such as the attitude and / or the position of the host vehicle may be output. The imaging device is basically configured with a single camera,
The focal length of the camera and the size of the imaging surface are known in advance. However, these imaging devices can be configured using a plurality of cameras.

The map data acquisition means is a method for acquiring map data around the own vehicle. DGPS (Di
fferential Global Positioning System) and peripheral map data may be obtained from the map database and the vehicle position and the peripheral map data estimated by the INS etc. in addition to DGPS. INS based on the own vehicle position acquired in the past
For example, the surrounding map data may be acquired from the own vehicle position and the map database estimated up to now. That is, in the map data acquisition means of the present invention, the method of acquiring map data is not particularly limited.

The map database is basically data in which two-dimensional road information is described, and may be a map database in which road links for a navigation system are described, or the coordinate values of road division lines. May be described, or a map database in which the coordinate values of the road center line may be recorded. Further, a dedicated map database may be used, or data in which three-dimensional information is described may be used.

As means for acquiring road shape information, in addition to the combination of the imaging device and the map data, the combination of the imaging device and the radar device, the radar device and the map data, the imaging device, the radar device, and the map database may be used. good. FIG. 4 illustrates a road shape estimating device in which the map database obtaining means of FIG. 1 is replaced by a radar device and the map road shape extracting means is replaced by a radar road shape extracting device.

A specific method of each means in the road shape estimation device shown in FIG. 1 will be described. The image road shape extraction means extracts a road shape on the screen by image processing from a forward road image acquired by the imaging device. FIG.
An example is shown below. For example, a luminance change of the image is obtained from the acquired image by a differential filter or the like, and a portion having a large luminance change is extracted as a white line candidate. A portion corresponding to the center line is extracted from the extracted left and right white line sets and is set as a road shape on the image.

The map road shape extracting means extracts the road shape on the map from the own vehicle position acquired by the own vehicle position acquiring means and the map data around the own vehicle position. FIG. 6 shows an example. For example, when using a map database in which coordinate values of road lane markings are used, points indicating road lane markings around the vehicle are extracted, and points indicating road lane markings on which the vehicle travels are extracted. . The road shape located at the center of the traveling lane is extracted from the points indicating the extracted road lane markings and the lane where the host vehicle is expected to be traveling. The road shape may be an interior dividing point of the points indicating the left and right road division lines, a shape in which the interior dividing points are connected by a line segment, or a shape in which the interior dividing point is approximated by a curve.

The road shape estimating means projects the acquired image road shape and map road shape onto one logical space, and calculates a matching degree C indicating the degree of overlap between both candidate regions obtained as the projection result. By estimating the parameters so that the matching degree C increases, the road shape, the attitude of the own vehicle relative to the road surface, and the position of the own vehicle relative to the map surface are estimated.

FIG. 7 exemplifies a coordinate system of the image road shape and the map road shape. The road shape of the image is obtained on the imaging plane, and is represented by an image coordinate system (Ximage, Yimage). Map road shape is obtained on the map, and the map coordinate system (Xm, Ym, Zm)
Indicated by The imaging plane is perpendicular to the optical axis of the camera,
Let f be the focal length. The camera coordinate system is (Xc, Yc,
Zc), the camera viewpoint is the origin of the camera coordinate system,
The optical axis direction of the camera is defined as a Yc axis direction.

An angle between the Yc axis of the camera and the Xm-Ym plane of the map is defined as a camera pitch angle θ with respect to the map plane.
Xm-Ym is the angle between the Yc axis of the camera and the Ym axis of the map.
The angle projected on the plane is defined as the yaw angle of the camera with respect to the map as φ. Further, assuming that the deviation between the camera origin and the map origin is the Xm-axis direction d, the Ym-axis direction L, and the Zm-axis direction h, the map coordinate system (Xm, Ym, Xm) in the camera coordinate system (Xc, Yc, Zc; origin Oc). The position coordinates of the origin Om of Zm)
(Xc, Yc, Zc) = (d, L, h).

The relationship between the camera coordinate system (Xc, Yc, Zc) and the image coordinate system (Ximage, Yimage) is expressed by the following equation (1).

(Xc, Yc, Zc) = (Ximage, f, Yimage) (1)

FIG. 8 exemplifies a candidate area based on an image road shape and a candidate area based on a map road shape. In the camera coordinate system, the actual point projected on the point (Ximage, Yimage) of the image road shape is a point (Xim
age, Yimage) direction vector (Ximage, f, Y)
image), the projected actual point is assumed to be at the position indicated by the constant α times the map coordinate system (Xm, Ym).
m, Zm), it can be represented by the equation shown in FIG.

As shown in FIG. 8, a projection candidate area obtained by perspectively projecting a point of an image road shape onto a map coordinate system and a projection candidate area obtained by vertically projecting a point of a map road shape are superimposed to calculate the degree of matching between the two. I do. Here, the parameters (independent variables) that change the matching degree C are α, d, L, φ, θ, and the matching degree C is expressed as a function C (α, d, L, φ, θ). However, the camera height h with respect to the map surface was known. As a method of calculating the collation degree C, any appropriate method may be used. For example, the values of these parameters can be specifically determined (optimized) by a well-known numerical analysis technique such as the steepest descent method so that the above-mentioned two values best match.

FIG. 9 illustrates a simple parameter determination method. First, the acquired image road shape and map road shape are projected onto one three-dimensional or two-dimensional logical space. For parameters necessary for projection, unknown parameters are given an initial value and projection is performed. An unknown parameter can be estimated by sequentially updating the parameters within the limited range so that the calculated matching degree C becomes the maximum. As for the parameter estimation method, the parameter may be updated sequentially, or the parameter may be updated roughly at first, and the parameter may be changed more finely in a range where the degree of matching increases. The parameter estimation method does not need to be particularly limited.
These parameters can be optimized using a known appropriate numerical analysis technique.

As described above, by estimating the parameters so that the matching degree C becomes large, it becomes possible to estimate the road shape, the attitude of the own vehicle with respect to the road surface, and the position of the own vehicle with respect to the map surface. Here, the estimation example is shown assuming that the focal length f of the camera and the shift h in the z direction of the camera coordinate system with respect to the map coordinate system are known. However, even when h is an unknown number, estimation is performed in substantially the same manner as described above. It is possible. Also,
The pitch angle θ with respect to the map plane is the sum of the camera pitch angle θ1 with respect to the road plane and the road inclination angle θ2 with respect to the map plane, and may be separately estimated.

FIG. 10 shows a method for preliminarily estimating the attitude of the host vehicle with respect to the road surface from the image road shape acquired by the image road shape extraction means. By inputting the estimation result to the road shape estimation means, it is possible to limit the parameter change range. In the road shape estimating means, it is not necessary to perform the update by using the posture with respect to the road surface estimated in advance as it is, or the update may be performed.

FIG. 11 shows a method of estimating the position of the host vehicle with respect to the map surface in advance from the image road shape obtained by the image road shape extraction means and the map road shape obtained by the map road shape extraction means. By inputting the estimation result to the road shape estimation means, it is possible to limit the parameter change range. In the road shape estimating means, the position with respect to the map plane estimated in advance may be adopted as it is and may not be updated, or may be updated.

FIG. 12 is obtained by adding a road shape model collating means to the road shape estimating apparatus shown in FIG.
The road shape model matching means is configured to determine whether the road shape estimated by the road shape estimation means is physically restricted by a road near the site such as "there is no steep slope (gradient) exceeding a specified value". Modifications may be made depending on conditions, or the road shape may be expressed in a parameter space such as curvature, and DP
(Dynamic Programming) It may be modified by matching, Kalman filter, or the like.

FIG. 13 is an example of a method for estimating the attitude of the host vehicle with respect to the road surface from the image road shape shown in FIG. The extension of the road edge near the vehicle is shown as straight line 1 and straight line 2 on the screen. Assuming that the left and right road edges near the host vehicle are parallel, the vanishing point is located at Yimage = 0 when the pitch angle with respect to the road surface is 0 deg. Also,
When the yaw angle is 0 deg, the position becomes Ximage = 0.
From the position of the vanishing point on the screen, the pitch angle with respect to the road surface and the yaw angle with respect to the road center line can be estimated.

Further, from the straight line 1 and the straight line 2, the lateral position deviation d of the own vehicle with respect to the road center line and the lane width 2W can be obtained. If the slope of the straight line 1 is a0 and the y-intercept is b0, and the slope of the straight line 2 is a1 and the y-intercept is b1, the pitch angle θ
1, the yaw angle φ, the lateral position deviation d, and the lane width 2W can be uniquely obtained by a geometric method. That is,
These numerical values (θ1, φ, W, d) are shown in FIG.
, Respectively, can be obtained by the four equations described collectively.

For example, as described above, the road shape estimating apparatus of the present invention collates the extraction results of the means for extracting the road shape from the image data obtained by capturing the image in front of the vehicle and the means for extracting the road shape from the map data. Thus, the position / posture of the vehicle and the road shape ahead of the vehicle are estimated. Further, a radar can be used as one of means for extracting a road shape to be collated. According to these configurations (embodiments), it is possible to estimate the shape of a distant road with high accuracy.

Further, according to the road shape estimating apparatus of the present invention as described above, even in a situation where the front road surface cannot be detected from the image data due to the road shape ahead such as a pass or a curve, or the preceding vehicle. Alternatively, even in a situation where the estimation error in the horizontal plane shape estimation using a single-eye image is large, robust estimation of the road shape and the position / posture of the own vehicle can be accurately performed by using information such as a map in addition to the image data. Becomes possible.

Further, by estimating the road shape based on the map road shape and the image road shape and estimating the attitude of the own vehicle with respect to the road surface, the own vehicle attitude estimation is performed in comparison with the attitude estimation based on the image road shape alone. The accuracy can be expected to improve.

As described above, by comparing the road end position obtained from the image with the digital road map, the position / posture of the own vehicle on the map and the road shape ahead of the own vehicle are more accurately estimated than in the past. be able to. Further, according to the road shape estimating device of the present invention, a distant road shape can be estimated with relatively high accuracy, so that, for example, danger determination of a target detected by radar can be performed. Further, the road shape estimation device of the present invention can also be used as a position / posture estimation device for estimating the position / posture of the vehicle.

[0090]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to specific embodiments. However, the present invention is not limited to the embodiments described below. (First Embodiment) A road shape estimating apparatus according to the first embodiment includes a road shape (first shape candidate) extracted from image data obtained by capturing an image in front of a vehicle and a road shape (first shape candidate) extracted from digital map data. (Two shape candidates) to estimate the three-dimensional shape of the traveling road.

According to the road shape estimating apparatus of the first embodiment, a logical form is obtained based on the extracted road shape information (first shape candidate and second shape candidate) and the estimated position and orientation of the vehicle. An appropriate three-dimensional road shape can be estimated by projecting a road shape onto a three-dimensional space, calculating the existence probability, and checking physical constraints to be satisfied by the estimated representative point of the road edge. Hereinafter, the road shape estimation device according to the first embodiment will be described with reference to the drawings. FIG.
4 shows the outline of the components of the present invention.

[0092] The road shape estimating device includes an image pickup device that obtains forward information as an image, a road image obtaining device that obtains a road shape in the image from the image, a device that obtains the position of the own vehicle, and a device that obtains the vicinity of the own vehicle position. A road map obtaining means for obtaining a road shape on the map from the map; and a means for estimating the position and orientation of the host vehicle. In the three-dimensional road shape estimating means, the road image and the road map are projected and superimposed in the three-dimensional space based on the result of the estimation of the position and orientation of the own vehicle, and a robust road in the three-dimensional space is determined based on the existence probability and physical constraints. The shape is estimated with high accuracy.

The image pickup device is a camera whose installation height, focal length, and viewing angle are known in advance, and a plurality of cameras may be used. However, since accurate distance information is not required from an image, the image pickup device is basically used. Use a single camera. The position of the host vehicle may be obtained by sequentially using positioning results obtained by a satellite such as DGPS, or may be estimated by a gyro or INS from the result of the positioning once. Further, the map database may be digital map data for a navigation system, numerical map data on the market, or a specially created database.

FIG. 15 shows a road image / road map acquisition means 1.
1 shows an outline. The road image acquiring means 1 is for extracting a road shape in an image, and includes an in-screen road shape extracting means for extracting white lines and road edges by image processing. As a method of extracting a white line, there is a method of converting a front image into a monochrome image, obtaining a luminance change by a differential filter, and extracting a portion having a large luminance change as a white line as shown in FIG. The road map acquisition means 1 extracts, from the acquired map information, the coordinate points of the road edge on which the vehicle travels as a road shape, and is composed of a road shape extraction means within a map plane.

FIG. 16 shows the outline of the road image / road map acquisition means 2. The road image obtaining means 2 further comprises an in-screen road shape estimating means for estimating a shape indicating a road alignment in the screen based on the shape of the white line or the like extracted by the in-screen road shape extracting means. As a method of estimating the road shape in the screen, there is a method of estimating as a continuation of the middle point of a horizontal line connecting the extracted white lines as shown in FIG.

The road map obtaining means 2 further estimates the shape of the road in the map plane based on the road edge coordinate values extracted by the above-described road shape extraction means in the map plane. Consisting of means. As a method of estimating the road shape in the map plane, there is a method of estimating as a continuation of internally dividing points of the road width as shown in FIG.

As the road image and the road map used for estimating the road shape, the result obtained by extracting the road edge or the like may be used as it is, or the center line of the road or the center line of the lane in which the vehicle is running may be obtained from the extracted result. Alternatively, a line on which the host vehicle is expected to travel may be estimated and used. FIG. 17 illustrates an outline of a three-dimensional road shape estimation unit. The three-dimensional road shape estimating unit estimates a road shape in a three-dimensional space from the road shape acquired by the road image acquiring unit and the road shape acquired by the road map acquiring unit.

In the road shape existence probability calculating means, the camera coordinate system and the map coordinate system are aligned based on the result of the estimation of the vehicle posture, and the road shape on the screen is projected from the camera viewpoint into a three-dimensional space. The road shape on the map surface is projected into the space, and the road shape information is superimposed. The situation where the road shape in the screen and the road shape in the map are projected in the three-dimensional space is as shown in FIG.

FIG. 18 is a graph showing an example of a method for calculating the existence probability. The road shape by the map as a set of coordinate points, the existence probability map at each coordinate point, n-th map coordinate point m _n (x, y) probability around the distribution Pm _n (x,
y). The portion where the projection range of the i-th pixel overlaps the spread range of the map coordinate point as shown in FIG. The existence probability of the overlapping portion is expressed, for example, by the following equation (2). P _i (x, y) is the existence probability at the coordinate point (x, y) when the i-th pixel is projected into the three-dimensional space.

[0100]

[Number 2] _{P n (x, y) =} ∬Pm n (x, y) · Pi (x, y) dxdy ... (2) In addition to the above, estimating the maximum value of the overlap region without taking a sum of products A point may be used, or a middle point of the x-axis may be used as an estimated point.

In the appropriate road shape extracting means, a position having a high probability is extracted as an estimated point from the road shape existence probability in the three-dimensional space calculated in the preceding stage. The extracted estimation points may be directly used as three-dimensional road shape estimation points, or a test may be performed on the estimation points to determine whether or not some conditions are satisfied. It may be adopted as a shape estimation point.

For example, a road shape that is smoothly continuous on the screen becomes closer to the camera as it goes down the screen. When the estimated points are projected on the image plane and there is an inconsistency in the order relationship between the screen and the map, it is desirable to adopt another estimated point. Further, when the change in the height direction of the adjacent estimation point is equal to or larger than the gradient change allowed on a general road, it is desirable to adopt another estimation point. FIG. 19 illustrates a flowchart of the three-dimensional road shape estimating means.

The three-dimensional shape may be estimated by directly connecting the estimated points or by a polynomial using least square approximation of all or some of the estimated points. Instead of directly determining individual estimation points, the three-dimensional road shape to be estimated may be expressed by parameters such as curvature, and the values of these parameters may be estimated to estimate the three-dimensional road shape. That is,
For details on how to determine the estimation points and how to estimate the three-dimensional shape,
It is not necessary to limit to a specific method, and an appropriate method can be selected from known numerical analysis methods.

By estimating the road shape in the three-dimensional space, even if the road ahead has, for example, a longitudinal curvature or a transverse curvature, and a projection error occurs in the estimation in a plane, FIG. Shape estimation can be performed while eliminating such projection errors, and highly accurate road shape estimation can be performed.

FIG. 21 shows a situation in which an invisible area (a blind spot area or the like) occurs due to the shape of the road and information on the road cannot be obtained from the image. If the image information ahead cannot be obtained due to such road conditions or weather, the visible range is estimated using both the image and map information. The estimated road shape can be extended by extending the road shape estimated within the range. Alternatively, the estimated road shape may be corrected by using the obtained map information, or may be corrected by referring to a road model that matches the actual road shape. For example, FIG. 22 illustrates a flowchart of shape estimation in a case where there is a range (such as a blind spot area) that cannot be seen in an image as in the above situation.

As described above, using the road shape estimating apparatus of the present invention for estimating a three-dimensional road shape using image road information and map road information eliminates the projection error of an image and makes it possible to estimate the shape alone. The road shape can be accurately estimated. Such a road shape estimation device is useful for, for example, preceding vehicle determination and automatic steering. Further, since the gradient of the road ahead can be estimated, the road shape estimation device of the present invention is also effective for speed control such as braking and accelerator control.
Further, according to the road shape estimation device of the present invention, it is possible to estimate a road shape ahead (a blind spot area or the like) even in a range where image data cannot be obtained.

(Second Embodiment) The road shape estimating apparatus according to the second embodiment estimates the shape of a road (or lane) ahead on a plane. According to the road shape estimating apparatus of the second embodiment, even in a range where information is not obtained from the image data or in a situation where the accuracy of estimating the road shape cannot be sufficiently ensured by the information of the image map unit, the required accuracy can be obtained. The road shape ahead can be estimated.

Hereinafter, the road shape estimating apparatus according to the second embodiment will be described with reference to the drawings. FIG. 23 schematically shows the components of the second embodiment. The road shape estimating device includes an imaging device that acquires information in front of the vehicle as an image, a road image acquiring unit that acquires a road shape in the image from the image, a unit that acquires the position of the own vehicle, and a map from a map around the own vehicle position. Road map acquisition means for acquiring the upper road shape, and means for estimating the position and orientation of the host vehicle.

The in-plane road shape estimating means superimposes the road image and the road map in one plane on the basis of the result of the estimation of the position and orientation of the own vehicle, and provides robustness in the plane based on information of both the image and the map. Accurate road shape estimation. The imaging device is a camera whose installation height, focal length, and viewing angle are known in advance, and a plurality of cameras may be used.
Since a precise distance information is not required from the image, a single camera is basically used.

The position of the host vehicle may be obtained by sequentially using positioning results obtained by satellites such as DGPS, or may be estimated by a gyro or INS from the results of the positioning once. The map database may be digital map data for a navigation system, numerical map data on the market, or a specially created database.

The outline of the road image / road map obtaining means 1 in the second embodiment is substantially the same as that shown in FIG. 15 of the first embodiment. However, it is assumed that the logical space on which the extracted road shape is projected is constituted by a plane (n = 2). The road image acquiring means 1 is for extracting a road shape in an image, and includes an in-screen road shape extracting means for extracting white lines and road edges by image processing. As a method of extracting the white line, as shown in FIG. 15, there is a method of converting the front image into monochrome, obtaining a luminance change by a differential filter, and extracting a portion where the luminance change is large as a white line. The road map acquisition means 1 extracts, from the acquired map information, the coordinate points of the road edge on which the vehicle runs, as a road shape, and comprises road shape extraction means within a map plane.

The outline of the road image / road map acquisition means 2 in the second embodiment is substantially the same as that shown in FIG. 16 of the first embodiment. However, it is assumed that the logical space onto which the extracted road shape is projected is constituted by a plane (n = 2). The road image obtaining means 2 further comprises an in-screen road shape estimating means for estimating a shape indicating a road alignment in the screen based on the shape of the white line or the like extracted by the in-screen road shape extracting means. As a method of estimating the road shape in the screen, there is a method of estimating as a continuation of the middle point of a horizontal line connecting the extracted white lines as shown in FIG.

The road map obtaining means 2 further estimates the shape of the road in the map plane based on the road end coordinate values extracted by the above-described road shape extraction means in the map plane. Consisting of means. As a method of estimating the road shape in the map plane, there is a method of estimating as a continuation of internally dividing points of the road width as shown in FIG.

As the road image and the road map used for estimating the road shape, the results obtained by extracting the road edges and the like may be used as they are, or from the extraction results, the center line of the road and the center line of the lane in which the vehicle is running. Alternatively, a line on which the host vehicle is expected to travel may be estimated and used.

FIG. 24 illustrates an outline of the in-plane road shape estimating means. The in-plane road shape estimating means estimates a road shape on a certain plane from the road shape on the screen obtained by the road image obtaining means and the road shape on the map surface obtained by the road map obtaining means. In the road shape coplanar projection part,
Based on the result of the estimation of the position and orientation of the host vehicle, the camera coordinate system and the map coordinate system are aligned, and the road shape on the screen is projected onto a plane from the camera viewpoint.
The road shape on the map plane is vertically projected and the road shape information is superimposed. When the road shape in the image plane is superimposed on the road shape in the map, the result is as shown in FIG.

When the front road has a longitudinal curvature or a transverse curvature, as shown in FIG. 20 of the first embodiment, the corresponding position in the plane is larger than the actual road position corresponding to the road shape in the image plane. May shift. This shift increases as the distance from the center of the camera increases, and the shape may be completely different from the actual shape. However, the range close to the camera,
Depending on the shape of the road, this deviation is not very large. The relative accuracy of a generally available map is about 1 m, which does not depend on the position from the camera. By combining the information based on the image and the information based on the map by using the nature of the error, it becomes possible to estimate the road shape with higher accuracy than the case where the information is individually estimated.

For example, as shown in FIG. 26, when it is estimated as an internal dividing point between a coordinate point of a map and a coordinate point of an image corresponding to the map, an internal dividing point is set such that an image is closer to the vehicle and closer to the map as far away. By changing the ratio (a / (a + b)) according to the distance from the camera, it becomes possible to estimate a road shape in consideration of image and map information.

That is, assuming that the reliability of the coordinate point of the image (first shape candidate) is R1 and the reliability of the coordinate point of the map (second shape candidate) is R2, as shown in FIG. The estimated coordinate point can be determined so that the equation (3) holds.

A / (a + b) = R2 / (R1 + R2) (3)

FIG. 27 shows a flow of the estimation shown in FIG.
The interior shape points may be connected to form a road shape, or the shape predicted based on the estimation results up to the middle and the road shape may be sequentially corrected from the information of the image and the map to obtain the actual road shape. The road shape may be corrected by comparing with a road model adapted to the road shape.

For example, as shown in FIG. 21 described above, there may be a situation where an invisible area is generated due to the shape of the road and information on the road cannot be obtained from the image. However, according to the present road shape estimation device, even in such a situation, the visible range is estimated using information of both the image and the map as shown in FIG. It is possible to estimate the preceding road shape from the road shape estimated within the range, and output the estimated road shape as the estimated road shape.

Further, as shown in FIG. 22, the estimated road shape may be corrected by using the obtained map information. However, the dimension number n of the logical space to which the extracted road shape is projected is set to two. Also,
The estimated road shape may be obtained by performing correction by collating with a road model that matches the actual road shape.

As described above, even when the imaging device (camera) is a single eye, the accuracy of the road shape is estimated by using the image road information and the map road information as compared with the case of estimating the road shape by itself (either information). The road shape can be estimated well. Also, according to the present invention, it is possible to estimate a road shape with relatively high accuracy in a range where image information cannot be obtained due to a longitudinal curvature in front of the vehicle.

(Third Embodiment) The road shape estimating apparatus according to the third embodiment estimates the height of an object (or a road surface) ahead of a vehicle from image data of an imaging device and radar signals of a radar.
Using the extracted road shape (first shape candidate or second shape candidate) and the height of the object (or road surface) ahead of the vehicle,
The three-dimensional shape of the road ahead is estimated. According to the road shape estimation device of the third embodiment, the road alignment is estimated from the arrangement of the objects extracted from the radar signal of the radar,
By performing a road shape estimation process based on the estimated road alignment and the above-described road shape (the first shape candidate or the second shape candidate), it is also possible to highly accurately estimate the three-dimensional shape of the road ahead. It is.

Hereinafter, the road shape estimating apparatus in the third embodiment will be described with reference to the drawings. FIG. 28 schematically shows the components of the present invention. In FIG. 28,
The road shape estimating device includes an imaging device that acquires a road ahead as an image and a radar device that detects an object ahead.

It has means for extracting a road shape in the image plane by image processing from an image obtained by the imaging device, and means for detecting an object in the image plane, and detects the distance to the object from the acquired radar signal. And an object detecting means. Further, an object height estimating means for estimating the height of the corresponding object from the object distance detection result of the radar and the object detection result of the image, and the height of the road surface at the object existence position from the object height Has a road surface height estimating means for estimating the road surface height. In addition, there is provided road shape estimating means for estimating the three-dimensional shape of the road from the estimated road surface height and the result of extracting the road shape in the image plane.

Since accurate distance information is not required from the image, the imaging device is basically constituted by a single camera. FIG. 31 and FIG. 32 schematically show the object position estimating means. A range in the image plane of the object detected by the object detection means in the image plane is projected into a three-dimensional space from a camera viewpoint to obtain a region where the object may exist.

The object position in the horizontal plane detected by the object distance detecting means by the radar is projected into a three-dimensional space to obtain a region where the object may exist. By obtaining a portion where both regions overlap, it becomes possible to estimate the three-dimensional position where the object exists. At this time, when a portion where both regions overlap is not uniquely determined, a plurality of possibilities are considered.

When it is determined that the object is present on the road surface, the road surface height estimating means estimates the height of the road surface by considering the lower end of the object position as the road surface. FIG. 33, FIG.
Shows an outline of the road shape estimating means. FIG.
As shown in (2), when there is no bank on the road ahead, a horizontal line indicating the road surface height at the three-dimensional existence possibility position of the object estimated by the radar and the image is drawn in the three-dimensional space. From the horizontal line indicating the height of the road surface in the three-dimensional space and the positional relationship of the camera, it is possible to estimate a straight line in the image plane corresponding to the horizontal line indicating the height in the three-dimensional space.

By searching for an intersection between a straight line in the image corresponding to the height at the object position in the three-dimensional space and the road shape in the extracted image, the road shape at the object position in the three-dimensional space is obtained. Can be determined in the image. The corresponding point is projected from a camera viewpoint to a three-dimensional space, and a portion overlapping with a horizontal line indicating a road surface height is a three-dimensional position of the road shape. When there are a plurality of points whose three-dimensional positions can be estimated, the three-dimensional road shape can be estimated by connecting them. Even when only one point can be estimated, the three-dimensional shape of the road can be estimated by using conditions such as a constant gradient and a gentle change.

When the actual road shape is estimated only with the road shape extracted in the image plane, when the road ahead has a longitudinal curvature or a transverse curvature, a large error occurs between the actual road position and the projected position on the horizontal plane. there is a possibility. By estimating the actual road surface height and estimating the road shape at the correct height,
The three-dimensional shape of the road can be estimated, and errors can be reduced.

FIGS. 29 and 30 show an outline of an apparatus for estimating a road shape using a radar and estimating a road shape in combination with a road shape obtained from an image or a road shape obtained from a map. Things. By performing road linear estimation by radar, even when the external environmental conditions are poor and the road shape cannot be obtained from the image data, or when the GPS reception state is continuously poor and the road shape cannot be obtained from the map, Although the accuracy is deteriorated, it is possible to continue the estimation of the road shape robustly.

[Brief description of the drawings]

FIG. 1 is a logical configuration diagram showing a basic configuration example of a road shape estimation device of the present invention.

FIG. 2 is a logical configuration diagram showing another typical configuration example of the road shape estimation device of the present invention.

FIG. 3 is a logical configuration diagram showing another typical configuration example of the road shape estimation device of the present invention.

FIG. 4 is a logical configuration diagram showing a basic configuration example of a road shape estimation device of the present invention equipped with a radar.

FIG. 5 is a data flow diagram illustrating an image road shape extraction unit of the road shape estimation device of the present invention.

FIG. 6 is a data flow diagram illustrating a map road shape extraction unit of the road shape estimation device of the present invention.

FIG. 7 is a three-dimensional image diagram illustrating a coordinate system of an image road shape and a map road shape of the road shape estimation device of the present invention.

FIG. 8 is an image diagram of a three-dimensional logical space including a candidate region based on an image road shape and a candidate region based on a map road shape in a matching unit of the road shape estimation device of the present invention.

FIG. 9 is a flowchart illustrating an example of a procedure for determining a parameter group for optimizing a matching degree C in the matching unit of the road shape estimation device of the present invention.

FIG. 10 is a logical configuration diagram showing a basic configuration example of a road shape estimating device of the present invention having a host vehicle posture estimating means.

FIG. 11 is a logical configuration diagram showing a basic configuration example of a road shape estimation device of the present invention having own vehicle position estimation means.

FIG. 12 is a logical configuration diagram showing a basic configuration example of a road shape estimating device of the present invention having a road shape model matching unit.

FIG. 13 is an explanatory diagram exemplifying a method of estimating a posture of a host vehicle with respect to a road surface by the road shape estimating device of the present invention.

FIG. 14 is a logical configuration diagram showing a basic configuration of a road shape estimating device having an appropriate road shape extraction unit of the first embodiment.

FIG. 15 is a road image / road map acquisition unit 1 of the first embodiment.
FIG. 3 is a data flow diagram showing a basic configuration of the embodiment.

FIG. 16 shows a road image / road map acquisition unit 2 according to the first embodiment.
FIG. 3 is a data flow diagram showing a basic configuration of the embodiment.

FIG. 17 is a logical configuration diagram showing a basic configuration of a three-dimensional road shape estimating unit having an appropriate road shape extracting unit of the first embodiment.

FIG. 18 is a graph showing an example of a method for calculating the existence probability according to the first embodiment.

FIG. 19 is a flowchart illustrating an information processing procedure of a three-dimensional road shape estimating unit of the first embodiment.

FIG. 20 is a vertical sectional view (a) and a horizontal plane view (b) of a front road, illustrating a situation where a road shape projection error occurs in the first embodiment.

FIG. 21 is a vertical cross-sectional view of a front road, illustrating a situation where an invisible area occurs in the first embodiment.

FIG. 22 is a flowchart illustrating a basic information processing procedure for estimating a road shape in an invisible area according to the first embodiment;

FIG. 23 is a logical configuration diagram showing a basic configuration of a road shape estimating device having an in-plane road shape estimating means of the second embodiment.

FIG. 24 is a logical configuration diagram showing a basic configuration of an in-plane road shape estimating means of the second embodiment.

FIG. 25 is a plan view for explaining an outline of an internally dividing point calculating means for calculating an internally dividing point by a coordinate weighted average calculation of the second embodiment.

FIG. 26 is an enlarged view of a road shape in a plane illustrating an arithmetic method of an internal dividing point calculating means for calculating an internal dividing point according to the second embodiment.

FIG. 27 is a flowchart illustrating a basic information processing procedure of an in-plane road shape estimation unit according to the second embodiment;

FIG. 28 is a logical configuration diagram (general configuration 1) showing a basic configuration example of a road shape estimation device of the present invention equipped with radar according to the third embodiment;

FIG. 29 is a logical configuration diagram (overall configuration 2) showing a basic configuration example of a road shape estimation device according to the third embodiment of the present invention equipped with radar.

FIG. 30 is a logical configuration diagram (general configuration 3) showing a basic configuration example of the road shape estimation device of the third embodiment, which is equipped with a radar, of the present invention;

FIG. 31 is a plan view illustrating the outline of a road shape estimation device according to a third embodiment;

FIG. 32 is a perspective view of a logical space illustrating an information processing image of the object position estimating means of the third embodiment.

FIG. 33 is a perspective view of a logical space illustrating an information processing image of the road shape estimating means of the third embodiment.

FIG. 34 is a flowchart illustrating an information processing procedure of a road shape estimation method of the estimation device 1 of the third embodiment.

[Explanation of symbols]

C: Degree of collation (Xc, Yc, Zc): Camera coordinates (Xm, Ym, Zm): Map coordinates (Ximage, Yimage): Image coordinates f: Focal length of imaging device θ: Pitch angle φ: Yaw angle α: Enlarge Ratio (d, L, h) ... origin of camera coordinates O in map coordinate system
m coordinates Oc: camera viewpoint (origin of camera coordinate system) (α, d, L, φ, θ): optimization parameter group 2W for matching degree C: lane width

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06T 11/60 300 G09B 29/10 A 5J070 G08G 1/16 G01B 11/24 K 5J084 G09B 29/10 G01S 17 / 88 A F term (for reference) 2C032 HB05 HB22 HC01 HC05 HC08 HC23 HD03 2F065 AA14 AA56 BB05 BB27 CC40 DD00 FF01 FF04 FF64 FF65 FF67 JJ03 JJ05 JJ26 MM06 QQ00 QQ32 QQ38 QQ45 RR06 SS10 A07 A07 A07 A05 A07 A07 A05 A CB13 DB03 5H180 AA01 CC04 CC12 CC14 FF05 5J070 AC01 AC02 AE01 AF03 AK22 BD08 BF19 5J084 AA02 AA05 AB20 AC02 DA01 EA40

Claims (9)

[Claims] 1. An on-vehicle road shape estimating device having an imaging device for acquiring a foreground of an own vehicle as image data and estimating a shape of a road on which the own vehicle is running or expected to run, An image road shape extraction unit for extracting a first shape candidate of the road from the image data; a map data acquisition unit for acquiring map data around the own vehicle; and a second shape candidate of the road from the map data. Map road shape extracting means to be extracted; three-dimensional or two-dimensional
Projecting onto one logical space composed of two dimensions, based on the overlapping state of the two road shapes projected in the logical space, the shape of the road, the attitude of the host vehicle with respect to the road surface of the road, A road shape estimating device comprising: a road shape estimating unit configured to estimate an absolute position of the vehicle with respect to the road. 2. The self-vehicle posture estimating means for preliminarily estimating the posture of the self-vehicle with respect to the road surface, or the self-vehicle position estimating means for presuming an absolute position of the self-vehicle with respect to the road. The road shape estimating device according to claim 1, comprising: 3. The road shape estimating means includes a matching means for stochastically evaluating the overlapping state for each point of the first shape candidate and the second shape candidate that should correspond to each other. The road shape estimation device according to claim 1 or 2, wherein 4. The road shape estimating means associates the reliability R1 of the first shape candidate and the reliability R2 of the second shape candidate with each other between the first shape candidate and the second shape candidate. A reliability calculation means for calculating each power point, and a reliability calculation means for each point based on the reliability R1 and R2.
The road shape estimating device according to any one of claims 1 to 3, further comprising an internal dividing point calculating means for calculating an internal dividing point between the points by a coordinate weighted average calculation. 5. An appropriate road shape is eliminated by eliminating inconsistencies in the positional relationship between respective representative points constituting the shape of the road estimated by the road shape estimating means based on predetermined physical constraints. The road shape estimating device according to any one of claims 1 to 4, further comprising an appropriate road shape extracting means for extracting the road shape. 6. The method according to claim 1, wherein the road shape estimating unit sets the first viewpoint based on a camera viewpoint of the imaging device as a projection center.
2. The image processing apparatus according to claim 1, further comprising a projection unit configured to project a shape candidate into the logical space, and performing positioning between a camera coordinate system and a map coordinate system based on the position and orientation of the host vehicle.
The road shape estimation device according to any one of claims 1 to 5. 7. An estimating area extending means for estimating a shape of the road in a blind spot area of the imaging device based on an estimation result of the road shape estimating means and the second shape candidate. The road shape estimation device according to any one of claims 1 to 6. 8. An on-vehicle road shape estimating apparatus, comprising: a radar for searching ahead of a host vehicle to obtain a radar signal, and estimating a road shape on which the host vehicle is running or expected to run. And detecting a position of an object ahead of the own vehicle from the radar signal, and estimating a road alignment of the road based on the arrangement of the objects. An image capturing device that acquires the data as data, an image road shape extracting unit that extracts the first shape candidate of the road from the image data, or “a map data acquiring unit that acquires map data around the own vehicle, A map road shape extracting means for extracting a second shape candidate of the road from the map data ", and the first shape candidate or the second shape candidate and the road shape are three-dimensionally or Projecting onto one two-dimensional logical space, and estimating the shape of the road in the logical space based on the overlapping state of the projected road shape and the road alignment in the logical space. A road shape estimating device comprising a road shape estimating means. 9. An image pickup apparatus for acquiring a foreground of a host vehicle as image data, and a radar for searching a front of the host vehicle to obtain a radar signal, wherein the host vehicle is running or expected to run. An on-vehicle road shape estimating device for estimating the shape of a road, comprising: an image road shape extraction unit configured to extract a first shape candidate of the road from the image data; and from the radar signal to an object in front of the host vehicle. Object distance estimating means for estimating the distance of the road; road surface height estimating means for estimating the road surface height of the road from the image data based on the distance to the object; Based on the road height and
And a road shape estimating means for estimating a three-dimensional shape of the road.