DE102011084674A1 - Construction of a lane map in multi-camera systems - Google Patents

Abstract

The invention relates to a method for creating a lane map, wherein a side camera (10, 11, 12, 13, 31, 32) of a vehicle (1) detects an environment of the vehicle (1), wherein from the images (22, 23, 24 ) of the side camera (10, 11, 12, 13, 31, 32) Fahrbahnbegrenzungselemente (3, 4, 5) of a roadway (2) are recognized and the recognized road boundary elements (3, 4, 5) are used for a lane map. The invention also relates to a driver assistance system, which is designed to create a lane map.

Description

State of the art

The invention relates to a method for creating a lane map and a driver assistance system.

In driver assistance systems, lane assistants are increasingly being used. Lane assistants cooperate with vehicle control systems, wherein they intervene to a defined extent independently in the drive train of the vehicle or in the control of the vehicle, such as to prevent impending collisions. They are also used, for example, in highway driving to relieve the driver on longer journeys without heavy traffic in the tracking. Lane assistants use information about the course of the lane in which the vehicle is located. The information is thereby provided by an environment recognition device, which comprises an environment detection device and a recognition device.

The lane assistants usually use a camera arranged on the vehicle as the environment detection device, because not only lane boundary elements that have a three-dimensional structure, such as guardrails or curbs, can be recognized by the camera, but also lane markings painted on the road that do not lie above the lane are substantially sublime. Currently, front cameras of the vehicle are used to pick up lane markings. DE 10 2009 020 300 A1 shows a method for displaying an environment of a vehicle on a display, wherein the environment has been detected by a front-mounted on a vehicle monocular camera. In this case, a selected lane section is output on a display to the driver of the vehicle as a virtual lane plane. An environment recognition device recognizes objects, such as lane boundaries, lane markings and objects or persons on the road, and outputs them on realistic positions on the virtual roadway plane.

From the DE 10 2005 037 273 A1 a lane assistant for vehicles is known, which comprises a vehicle-mounted environment detection device, with a detection device for the detection of road boundary elements, for. B. painted road markings. The surroundings detection device comprises two cameras, wherein a first camera in the direction of travel and another camera are aligned in the opposite direction to the direction of travel. The information obtained by the recognition device is used to give a warning to the driver or to actively intervene in the driving behavior of the vehicle.

The object of the invention is to provide a method for creating a lane map and a driver assistance system, which allow a more robust determination of the lane course and / or road course.

The object is achieved by a method having the features according to claim 1 and by a driver assistance system having the features according to claim 10. Advantageous developments of the invention are defined in the dependent claims.

Disclosure of the invention

Accordingly, it is provided that a side camera of a vehicle detects an environment of the vehicle, wherein from the images of the side camera lane boundary elements of a roadway are recognized and the recognized lane boundary elements are used for a lane map. The information obtained about the images of the side camera information about the lane boundary elements can be used in addition to the information obtained from the images of a front and / or rear camera on the lane boundary elements for the lane map.

The advantage with the additional use of side cameras is that a larger data set can be used than when using the data of the front camera or the rear camera, because a larger portion of road boundary elements can be detected, namely both the front and the rear side boundary elements and those side of the vehicle. Any bills in the combination of information from the front camera with information from the rear camera may be subject to errors, since the relative position of the cameras in the operation is not always very accurately determined and is subject to environmental influences. By using the side camera data, such calculations are eliminated and thus a potential source of systematic errors. Also, bills, which are required by the fact that with dashed lane markings, the gaps between the lane markings are so large that they are no longer detectable in a single image, can be omitted by the enlargement of the detected area by the additional use of side cameras for the detection of road boundary elements ,

Another advantage of the additional use of side cameras is that Points of the front camera are no longer so heavily tracked, ie must be tracked, as more data is available. The integration of data from the side camera enables a more robust determination of the road course and in particular the road curvature.

In one embodiment of the invention, the vehicle includes a plurality of image sensors. The image sensors may be front cameras, rear cameras, blind spot detection (BSD) cameras and / or sideview assistant (SVA) cameras and / or surround view system (SVS) cameras. which are used within other driver assistance systems for other purposes, for example, the so-called "back-over avoidance" or to detect a blind spot or help with an entrance to a poorly visible road. The image sensors can be monocular cameras or cameras from a stereo camera system, which enable the three-dimensional recognition of the environment.

The lane boundary elements may be lane markers, i. H. solid and offset lines, which can serve both to mark a road boundary, as well as to mark boundaries between different lanes, for example, to mark the demarcation of oncoming lanes. The roadway boundary elements can also be curbs or transitions caused by material change, such as from a road to grass. A roadway is understood to include multiple lanes, including lane for oncoming traffic.

The lane boundary elements are extracted by a recognizer from the image data received from the side camera and constructed, e.g. used to create a lane map, i. a two-dimensional or three-dimensional model of the detectable roadway and its lanes is calculated from the data and provided for further purposes, in particular for further driver assistance systems.

In lane assistance systems, the lane map can be used, for example, for calculations as to whether leaving the lane is imminent, for example, an acoustic signal can be output to the driver or an automatic steering intervention can be made.

In collision avoidance systems, for example, the lane map may be used for calculations as to whether a collision with a vehicle on the same lane is imminent, wherein, for example, a warning signal may be output to the driver or an automatic steering intervention may be performed.

The lane map can also be output to the driver on a display, such as in a head-up display or a head-down display. The display can be both a topview display and a perspective display of an inclined plane, it also being possible for these displays to be switchable and / or swiveled into one another. Furthermore, the display can be extended by further objects which have been detected by the cameras and recognized by a recognition device, for example using a technique of examining the optical flow for discontinuities. The lane map can be displayed purely virtual or combined with a real image, for example in the form of an "augmented reality", in particular under any highlighting, i. colored highlighting, certain areas.

The lane map can also be integrated into an existing, especially digital lane map. This can in particular include an embedding of the data in a navigation map, the data calculated from the camera images being compared with the data already present in a navigation map, and the data present in the navigation map can be enriched with additional information.

In a preferred embodiment of the invention, the side camera as a, for example, on a side mirror of the vehicle arranged, surround view system camera, hereinafter also SVS camera formed. The use of such a camera has the advantage that it is already installed as standard in many vehicle models and is used as part of a driver assistance system. In addition, such cameras are equipped with a wide-angle lens, which allows detection of a large section with road boundary elements.

In a further embodiment of the invention, the side camera is designed as a sideview assistant camera operated in a wide-angle mode, also referred to as SVA camera hereinafter, which is usually mounted in the vicinity of the vehicle front end and oriented in a lateral direction, ie perpendicular to the longitudinal axis of the vehicle to the left or to the right. Such cameras are often used when a driver wants to turn around from a house exit or from a side street in a busy street / view and the view, for example, by a series of parked vehicles is limited. The SVA cameras are so far at the top of the vehicle, that a look into the traffic already is possible if the vehicle is already inflected only a few centimeters on the road to be driven.

If the SVA cameras are used to detect lane boundary elements, they are preferably set to have the widest possible solid angle for detecting as large a road section as possible with lane boundary elements that can be used to build the lane map. In one embodiment of the invention, the SVA cameras are operable in two modes, namely, in a first mode in which they perform their well-known function when entering a busy highway and in which they can detect a lesser angle of the lateral environment and in one second mode wherein they serve a lane detection for detecting the largest possible portion of road boundary elements in a wide-angle operation. The various modes can be manually activated and / or interconverted by the driver, for example. In a wide-angle mode of the SVA camera, there are optical distortions, especially in the edge region of the image. Any optical distortions operated by side cameras in wide angle mode will be discounted by the recognizer.

In a further embodiment of the invention, the side camera as a, for example, on a side mirror of the vehicle arranged blind spot detection camera, hereinafter also BSD camera formed. The use of such a camera has the advantage that it is already installed in many available vehicle models and is used as part of a driver assistance system.

A particularly preferred embodiment of the invention relates to a vehicle having a multi-camera system in which the lane map can be constructed from the data of at least two cameras. According to the invention z. B. data of one or more front cameras in conjunction with one or more SVA cameras and / or data of one or more rear cameras in conjunction with one or more SVA cameras and / or data of one or more BSD cameras in conjunction with one or more SVA -Cameras and / or data from one or more front cameras connected to one or more BSD cameras and / or data from one or more rear view cameras in conjunction with one or more BSD cameras and / or data from one or more front cameras in conjunction with one or more SVSs Cameras and / or data from one or more rear-view cameras in conjunction with one or more SVS cameras and / or data from one or more BSD cameras in conjunction with one or more SVS cameras and / or data from one or more SVS cameras be used with one or more SVA cameras. In particular, the data of two opposing side cameras can be evaluated for lane detection. In a particularly preferred embodiment of the invention, the signals of all available cameras are combined in a central computer, which recognizes the lane boundary elements from the images and combines and processes them to create the lane map.

In the case of using the data of two or more cameras having overlapping detection areas, an overlapping area of the detection area of the cameras is determined such that an association of the points detected by one camera on the image area of one camera with those of the other camera defined points on the screen of the other camera is defined. Thus, objects that are simultaneously visible on two cameras can be located more reliably.

In the particularly advantageous case that an overlapping region of a side camera with a front camera and / or a rear camera, the detection areas of the cameras can be so mapped and continued, so that from the captured images information about the longest possible stretch of road boundary elements can be obtained ,

The information from areas with better image resolution is preferably used in the overlapping area of the detection area of the cameras. Objects that are on the edge of an image of a camera and may not be able to resolve properly here may be in the center of the image in the optimal resolution range of another camera.

From the road boundary elements detected by the side cameras, in one embodiment of the invention, a local curvature of the traffic lane and, therefrom, a curvature course in a detected section are determined. From the road boundary elements detected by a front camera or a rear camera, curvature values of the lanes can additionally be calculated and compared with the data of the side cameras. The advantage of using the side camera data is that a larger data set than when using the front camera or rear camera data can be used because a larger portion of road boundary elements can be detected. Information about the course of curvature is particularly in driver assistance systems, which a navigation device with a digital road map, particularly advantageous.

In a particularly preferred embodiment of the invention, data of a navigation map is matched with the determined traffic lane map, i. compared, checked and, if necessary, improved. A permanent storage of the lane information, for example, an overstore of the information of the navigation map should also be possible. The digital maps of navigation devices are often made from simple, idealized assumptions. Often, as is common in the planning of motorway routes, the distances are distinguished in straight lines, curves of constant curvature and clothoids for the transition between straight lines and curves of constant curvature. The clothoids form a transition from the path without curvature to the path of constant curvature, in which the curvature increases or decreases linearly. By calculating the local curvature from the road boundary elements of the side cameras, a map used in the navigation device can be better adapted to the real conditions. But this can also happen at construction sites, where there are changed track guides, which may not be present in the navigation map.

According to one embodiment of the invention, a position and orientation of the vehicle on the lane and / or the roadway is determined in particular from calculated curvature data of the traffic lane and / or the roadway. For this purpose, the identification device determines from the available image data information about the curvature of the roadway and / or the traffic lane and information on which lane the vehicle is located. Particularly advantageously, the information about the orientation of the vehicle on the traffic lane and / or the roadway is supplied to a navigation system, in particular a display of a navigation device, whereby the navigation system can also use the information to indicate in a stationary vehicle in which direction the vehicle is aligned Without, for example, information about where the vehicle comes from, which are not available to some available devices when turning on the navigation device.

According to one embodiment of the invention, the road boundary elements recognized by the side camera are used to determine a track width, in particular at the height of the vehicle. From knowing the position of the side camera and determining the location of the roadway boundary elements, i. in particular the height of the roadway boundary element in the middle of the image, a distance of the vehicle to the road boundary element can be determined.

• – Zumindest eine Seitenkamera des Fahrzeugs zum Erfassen einer Umgebung des Fahrzeugs,
• – eine Komponente zur Erkennung von Fahrbahnbegrenzungselementen einer Fahrbahn aus den durch die zumindest eine Seitenkamera erfassten Bildern, insbesondere aus den erfassten Bildern aller an dem Fahrzeug befindlichen Kameras und
• – eine Komponente zur Erstellung einer Fahrspurkarte.

The task is also solved by a driver assistance system with the components:

• At least one side camera of the vehicle for detecting an environment of the vehicle,
• A component for detecting lane boundary elements of a roadway from the images captured by the at least one side camera, in particular from the captured images of all the cameras located on the vehicle and
• A component for creating a lane map.

Preferably, further cameras are associated with the driver assistance system, in particular a front camera and / or rear camera whose images are evaluated on road boundary elements, and a fusion component in which the data of the cameras are combined and / or coordinated with each other.

Brief description of the drawings

1 shows a lane map with a vehicle disposed thereon in plan view,

2 an image taken by a front camera of a lane with lane boundary elements,

3 an image taken by a rear camera of a roadway with lane boundary elements,

4 a picture taken by a left side camera in a first embodiment of a lane with lane boundary elements,

5 an image taken by a right side camera in a first embodiment of a roadway with lane boundary elements and

6 an image taken by a left side camera in a second embodiment of a roadway with lane boundary elements.

Embodiments of the invention

1 shows a lane map in plan view. On the lane ticket is a vehicle 1 represented, as well as the course of a roadway 2 , which is bounded by Fahrbahnbegrenzungselemente, here by a solid lane marking 3 on the right side and through a solid lane marking 4 left side, the roadway 2 through the center of the road arranged dashed lane markings 5 in two lanes 6 . 7 is shared, being a first lane 6 a roadway of the vehicle 1 is and a second lane 7 is an oncoming lane.

Furthermore, on the vehicle 1 arranged image pickup devices shown, namely a front camera 8th , a back camera 9 , a left SVA camera 10 and a right SVA camera 11 , In the left side mirror of the vehicle 1 Can a left SVS camera 32 be arranged and / or a left BSD camera 12 , In the right side mirror of the vehicle 1 can be a right SVS camera 31 be arranged and / or a right BSD camera 13 ,

The detection areas of the front camera 8th , the back camera 9 , the left SVA camera 10 and the right SVA camera 11 form cones 8a . 9a . 10a . 11a , which are shown in dashed lines. The detection ranges of the left and right SVS cameras 31 . 32 are not shown for the sake of clarity. They essentially correspond to the cones 10a . 11a the left SVA camera 10 and the right SVA camera 11 with the difference that the cone tip would lie in the exterior mirror.

The detection areas of the left BSD camera 12 and the right BSD camera 13 are also shown in dashed lines, whereby it has already been considered that the detection areas by the vehicle 1 obscured, ie limited.

The cone 8a of the detection range of the front camera 8th and the cone 10a the detection range of the left SVA camera 10 or the left SVS camera 32 have a first overlap area 14 on which in 1 is shown as a hatched area. Likewise point the cone 8a of the detection range of the front camera 8th and the cone 11a the coverage of the right SVA camera 11 or the right SVS camera 31 an overlap area 15 on, which is hatched. Corresponding overlapping areas 16 . 17 exist also in the coverage areas of the SVA cameras 10 . 11 or the SVS cameras 31 . 32 with the rear camera 9 , which are also shown as hatched areas. The overlapping areas of the BSD cameras with the SVA cameras and with the rear cameras are not shown as hatched areas for the sake of clarity, but it is clear where these lie.

In the lane map is the vehicle 1 to N-W, ie oriented to the northwest, which is indicated by an arrow 18 in the vehicle 1 , which indicates the sitting position of a driver of the vehicle, is shown. The roadway 2 is in 1 curved to the left, ie, the vehicle 1 drives a left turn.

From all available cameras, ie from the front camera 8th , the back camera 9 , the left SVA camera 10 , the right SVA camera 11 , the left BSD camera 12 and the right BSD camera 13 , the left and right SVS cameras 31 . 32 Images are received by a central computer in the vehicle 1 can be arranged and having a detection device supplied. This evaluates the images with regard to the road boundary elements 3 . 4 . 5 from and calculated from the course of the road 2 and their lanes 6 . 7 , as well as the orientation of the vehicle in the roadway 2 and the lane 6 which is N-W here. The information obtained in this way can be compared with the data of a digital map and expanded with information from a navigation system. Subsequently, a picture of the roadway 2 and the vehicle arranged thereon 1 , such as in 1 shown, but also in other common perspectives, to a display device in the vehicle 1 be transmitted to the driver of the vehicle 1 outputs. In addition, the lane map obtained from the information is available for other driver assistance systems, such as a lane assistant and / or a collision avoidance assistant.

2 shows an example of the front camera 8th taken picture 20 with the roadway 2 out 1 , with the lanes 6 . 7 and the road boundary elements 3 . 4 . 5 , The roadway 2 has a curvature to the left, so the right lane mark is 3 , the left lane marking 4 and the middle lane mark 5 curved to the left.

Every point on the front camera 8th taken picture 20 is assigned an ordinate value y and an abscissa value x, for example the pixel value of the point. The own lane 6 is characterized by being in the case that the front camera 8th is located in the center of the vehicle, with respect to the abscissa, ie the x-position, in the center of the image is located. Is the front camera 8th not arranged in the center of the vehicle, the recognition device can also be a conversion of the received image on an image, which from the center of the vehicle 1 would be included.

3 shows an example of the rear camera 9 taken picture 21 the roadway 2 out 1 with the lanes 6 . 7 and the road boundary elements 3 . 4 . 5 , Every point on the back camera 9 taken picture 21 For example, an abscissa value x and an ordinate value y are assigned, for example, the pixel value of the dot. The rear camera 9 shows a curvature of the road 2 which is in the case that the front camera 8th and the rear camera 9 in the direction of the longitudinal axis of the vehicle 1 are aligned and the vehicle 1 is on a section of constant curvature, opposite to that of the front camera 8th represents. By the information of the front camera 8th and the rear camera 9 can the in 1 illustrated lane map are constructed. Here are the information of the front camera 8th and the rear camera 9 assembled and in a non-detectable area in which the vehicle 1 itself and which outside the cone 8a of the detection range of the front camera 8th and outside the cone 9a the coverage of the rear camera 9 is extended by calculation methods, for example, interpolated.

In 4 is a picture corresponding to the previous one 22 a left SVS camera 32 or a left SVA camera 10 , which is operated in a wide-angle mode, shown. The wide-angle mode results in a pillow-shaped distortion, which is based on a pillow-shaped distorted horizon 30 is shown recognizable. The left SVS camera 32 or the left SVA camera 10 takes the left turn, ie the curvature of the middle of the road 2 arranged dashed lane markings 5 after deducing the pincushion distortion as a substantially convex dot-dash line and the curvature of the left-side lane mark 4 as a convex line from which the curvatures are local, ie at each point of the lane markings 4 . 5 can be calculated. Through the left SVS camera 32 or the left SVA camera 10 In the case of the left turn, a very large road section can be surveyed, which is an accurate determination of the course of the road 2 allows.

In the picture 22 the left SVS camera 32 or the left SVA camera 10 would, after deduction of any caused by the wide-angle lens optical distortions, oblique lane boundary element, ie a road boundary element whose ordinate y on the abscissa x shows a substantially linear behavior, indicate that the vehicle 1 not exactly in the lane 6 lies. This information is also used to create the lane map according to 1 and in particular for illustrating the orientation of the vehicle 1 used in the lane ticket.

From the height x measured with respect to the abscissa x 25 the dashed lane marking 5 in the picture 22 , ie its ordinate y in the middle of the image 22 and knowing the position of the left SVS camera 32 or the left SVA camera 10 becomes a distance of the vehicle 1 to the lane marking 5 calculated on vehicle height. This information is also used to create the lane map according to 1 and in particular for the arrangement of the vehicle 1 used in the lane ticket.

In 5 is a picture 23 a right SVS camera 31 or a right SVA camera operated in wide-angle mode 11 represented, which to the in the 2 to 4 corresponds to images shown. The right SVS camera 31 or right SVA camera 11 takes the curvature of the road 2 in the form of a concave line 3 on. A recognition device of the vehicle 1 can out of the line 3 locally calculate the curvature and from the values obtained in conjunction with those from the in the 2 to 4 shown images in the 1 Create the lane map shown. In particular, in the case of a right turn, not shown, a very large area of the roadway 2 in the picture 23 the right SVS camera 31 or the right SVA camera 11 be measured.

In the picture 23 the right SVS camera 31 or the right SVA camera 11 would a, after deduction of any optical distortions by the wide-angle lens, oblique pavement marking, ie a pavement marking whose ordinate y on the abscissa x shows a substantially linear behavior, indicate that the vehicle 1 not exactly in the lane 6 lies. This information is also used to create the lane map according to 1 and in particular for illustrating the orientation of the vehicle 1 used in the lane ticket.

From the height x measured with respect to the abscissa x 27 the lane marking 3 in the picture 23 , ie its ordinate y in the middle of the image 23 and knowing the position of the right SVS camera 31 or the right SVA camera 11 becomes a distance of the vehicle 1 to the lane marking 3 calculated on vehicle height. This information is also used to create the lane map according to 1 and in particular for the arrangement of the vehicle 1 used in the lane ticket.

6 shows a picture 24 a left BSD camera 12 which corresponds to the previous ones. The picture 24 Corresponds in principle to that of the rear camera 9 taken picture 21 the roadway 2 in 3 with the difference that here a Lateralversatz and possibly a different orientation of the cameras are to be considered, because the BSD camera 12 . 13 is usually in a side mirror of the vehicle 1 looking towards the rear of the vehicle 1 built-in. The coverage of BSD cameras 12 . 13 is through the vehicle 1 restricted, what in 6 as a hatched left-sided area 26 is shown. The pictures of the BSD cameras 12 . 13 be like those the other cameras to detect the lane boundary elements 3 . 4 . 5 evaluated. The invention is not limited to the embodiments described herein and the aspects highlighted therein. Rather, within the scope given by the appended claims a variety of modifications are possible, which are within the scope of expert action.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102009020300 A1 [0003]
• DE 102005037273 A1 [0004]

Claims (10)

Method for creating a lane map, wherein a side camera ( 10 . 11 . 12 . 13 . 31 . 32 ) of a vehicle ( 1 ) an environment of the vehicle ( 1 ), whereby from the pictures ( 22 . 23 . 24 ) of the side camera ( 10 . 11 . 12 . 13 . 31 . 32 ) Lane boundary elements ( 3 . 4 . 5 ) of a roadway ( 2 ) and the recognized lane boundary elements ( 3 . 4 . 5 ) can be used for a lane ticket. A method according to claim 1, characterized in that the side camera as a particular on a side mirror of the vehicle ( 1 ) surround view system camera ( 31 . 32 ) is trained. Method according to Claim 2, characterized in that the side camera is used as a sideview assistant camera ( 10 . 11 ) and the sideview assistant camera ( 10 . 11 ) is operated in a wide angle mode. A method according to claim 1, characterized in that the side camera as a particular on a side mirror of the vehicle ( 1 ) arranged blind spot detection camera ( 12 . 13 ) is trained. Method according to one of the preceding claims, characterized in that the lane map from the data of at least two cameras ( 8th . 9 . 10 . 11 . 12 . 13 . 31 . 32 ), which overlapping detection areas ( 8a . 9a . 10a . 11a . 12a . 13a ) to have. Method according to one of the preceding claims, characterized in that from the at least one side camera ( 10 . 11 . 12 . 13 . 31 . 32 ) detected roadway boundary elements ( 3 . 4 . 5 ) a curvature course of the roadway ( 2 ) is determined in a detected section. A method according to claim 6, characterized in that a position and orientation of the vehicle ( 1 ) on the lane ( 6 ) and / or the roadway ( 2 ) is determined. Method according to one of the preceding claims, characterized in that data of a navigation map are compared with the determined lane map. Method according to one of the preceding claims, characterized in that that of the at least one side camera ( 10 . 11 . 12 . 13 . 31 . 32 ) detected roadway boundary elements ( 3 . 4 . 5 ) can be used to determine a track width. Driver assistance system with the following components: - a side camera ( 10 . 11 . 12 . 13 . 31 . 32 ) of the vehicle ( 1 ) for detecting an environment of the vehicle ( 1 ), - a component for detecting road boundary elements ( 3 . 4 . 5 ) of a roadway ( 2 ) from the through the side camera ( 10 . 11 . 12 . 13 . 31 . 32 ) captured images ( 22 . 23 . 24 ), in particular from the captured images ( 20 . 21 . 22 . 23 . 24 ) all on the vehicle ( 1 ) ( 8th . 9 . 10 . 11 . 12 . 13 . 31 . 32 ) and - a component for creating a lane map.

Images