WO2013056955A1

WO2013056955A1 - Construction of a lane map in multi-camera systems

Info

Publication number: WO2013056955A1
Application number: PCT/EP2012/068974
Authority: WO
Inventors: Henning Von Zitzewitz; Karsten Muehlmann; Jan Egelhaaf; Raphael Cano; Leo VEPA
Original assignee: Robert Bosch Gmbh
Priority date: 2011-10-18
Filing date: 2012-09-26
Publication date: 2013-04-25
Also published as: DE102011084674A1

Abstract

The invention relates to a method for creating a lane map, wherein a side camera (10, 11, 12, 13, 31, 32) of a vehicle (1) captures an environment of the vehicle (1), wherein roadway limit elements (3, 4, 5) of a roadway (2) are recognised from the images (22, 23, 24) of the side camera (10, 11, 12, 13, 31, 32) and the recognised roadway limit elements (3, 4, 5) are used for a lane map. The invention also relates to a driver assistance system which is configured for creating a lane map.

Description

Description title

PRIOR ART The invention relates to a method for creating a lane map and a vehicle

Driver assistance system.

In driver assistance systems, lane assistants are increasingly being used.

Lane assistants cooperate with vehicle control systems, wherein they intervene to a defined extent independently in the drive train of the vehicle or in the control of the vehicle, such as to prevent impending collisions. They are also used, for example, in highway driving to relieve the driver on longer journeys without heavy traffic in the tracking. Lane assistants use information about the course of the lane in which the vehicle is located. In this case, the information is provided by an environment recognition device, which comprises an environment detection device and a

Detection device comprises.

The lane assistants usually use a camera arranged on the vehicle as the environment detection device, since not only

Roadway boundary elements that have a three-dimensional structure, such as crash barriers or curbs are recognized, but also painted on the road lane markings that are not substantially raised above the roadway.

Currently, front cameras of the vehicle are used to pick up lane markings. DE 10 2009 020 300 A1 shows a method for displaying an environment of a vehicle on a display, the environment being detected by a monocular camera arranged on the front of a vehicle. In this case, a selected lane section on a display to the driver of the vehicle as a virtual

Road level output. From an environment recognition device, objects such as lane boundaries, lane markings and objects or persons the lane, recognized and issued on the virtual lane level at realistic positions.

From DE 10 2005 037 273 A1 a lane assistant for vehicles is known, which comprises an in-vehicle environment detection device, with a

Recognition device for the detection of road boundary elements, eg. B.

painted road markings. The environment detection device comprises two cameras, wherein a first camera in the direction of travel and another camera in

Counter direction are aligned to the direction of travel. The information obtained by the recognition device is used to give a warning to the driver or to actively intervene in the driving behavior of the vehicle.

The object of the invention is to provide a method for creating a lane map and a driver assistance system, which provides a more robust determination of

Lane course and / or road course allow.

The object is achieved by a method having the features according to claim 1 and by a driver assistance system having the features according to claim 10. Advantageous developments of the invention are defined in the dependent claims.

Disclosure of the invention

Accordingly, it is provided that a side camera of a vehicle detects an environment of the vehicle, wherein from the images of the side camera lane boundary elements of a roadway are recognized and the recognized lane boundary elements are used for a lane map. The information obtained from the images of the side camera information about the lane boundary elements can in addition to the information obtained from the images of a front and / or rear camera on the

Fahrbahnbegrenzungselemente be used for the lane map.

The advantage with the additional use of side cameras is that a larger data set can be used than when using the data of the front camera or the rear camera because a larger portion of road boundary elements can be detected, namely both the front and the rear

Roadway boundary elements and those on the side of the vehicle. Possible invoices when combining information from the front camera with information from the rear camera can be burdened with errors, since the relative position of the cameras in the operation is not always very accurately determined and is subject to environmental influences. By using the side camera data, such calculations are eliminated and thus a potential source of systematic errors. Also, bills, which are required by the fact that with dashed lane markings the gaps between the

Lane markings are so large that they are no longer detectable in a single image, by enlarging the detected area by the additional

Use of side cameras to detect lane boundary elements is eliminated.

Another advantage of using additional side cameras is that the front camera points are no longer so heavily tracked, d. H. must be tracked as more data is available. The integration of data from the side camera enables a more robust determination of the road course and in particular the road curvature.

In one embodiment of the invention, the vehicle includes a plurality of image sensors. The image sensors may be front cameras, rear cameras, blind spot detection (BSD) cameras, and / or SVA cameras (side view assist cameras) and / or surround view system (SVS) cameras be used within other driver assistance systems for other purposes, for example, the so-called "back-over avoidance" or to detect a blind spot or help with an entrance to a poorly visible street. The image sensors can be monocular cameras or cameras from a stereo camera system, which is the three-dimensional recognition of the environment

enable.

The lane boundary elements may be lane markers, i. H.

solid as well as offset lines, which are both for marking a

Fahrbahnbegrenzung can serve, as well as to mark boundaries between different lanes, for example, to mark the demarcation of

Against roadways. The roadway boundary elements can also be curbs or transitions caused by material change, such as from a road to grass. A roadway is understood to include multiple lanes, including lane for oncoming traffic. The lane boundary elements are extracted by a recognition device from the image data received from the side camera and used to build, ie to create a lane map, ie a two-dimensional or three-dimensional model of the detectable lane and its lanes from the data is calculated and used for other purposes, in particular for provided additional driver assistance systems.

In lane assistance systems, the lane map can be used, for example, for calculations as to whether leaving the lane is imminent, for example, an acoustic signal can be output to the driver or an automatic steering intervention can be made.

For example, in collision avoidance systems, the lane map may be for

Calculations are used, whether a collision threatens a vehicle on the same track, for example, a warning signal can be issued to the driver or an automatic steering intervention can be made.

The lane map can also be output to the driver on a display, such as in a head-up display or a head-down display. The display can be both a topview display and a perspective display of an inclined plane, it also being possible for these displays to be switchable and / or swiveled into one another. Furthermore, the display can be extended by further objects which have been detected by the cameras and recognized by a recognition device, for example using a technique of examining the optical flow for discontinuities. The lane map can be displayed purely virtual or combined with a real image, for example in the form of an "augmented reality", in particular with any highlighting, that is colored highlighting, of specific areas.

The lane ticket can also be in an existing, especially digital

Lane map to be integrated. This can in particular include an embedding of the data in a navigation map, the data calculated from the camera images being compared with the data already present in a navigation map and the data stored in the navigation map

Navigation map existing data can be enriched with additional information. In a preferred embodiment of the invention, the side camera as a, for example, on a side mirror of the vehicle arranged, surround view system camera, hereinafter also SVS camera is formed. The use of such a camera has the advantage that it is already installed as standard in many vehicle models and is used as part of a driver assistance system. In addition, such

Cameras equipped with a wide-angle lens, which allows detection of a large section with road boundary elements.

In another embodiment of the invention, the side camera is designed as a wide-angle sideview assistant camera, also referred to as an SVA camera hereinafter, which is usually mounted near the vehicle front end and oriented in a lateral direction, i. perpendicular to the longitudinal axis of the vehicle to the left or to the right. Such cameras are often used when a driver wants to turn around from a house exit or from a side street in a busy street / must and the view, for example, by a number parked

Vehicles, limited. The SVA cameras are so far at the top of the vehicle, that an inspection of the traffic is already possible if the vehicle is already inflected only a few inches on the road to be driven. If the SVA cameras are used to detect road boundary elements, they are preferably set so that they have the widest possible solid angle for detecting as large a road section as possible

Road boundary elements, which can be used to build the lane map overview. In one embodiment of the invention, the SVA cameras are operable in two modes, namely in a first mode in which they perform their well-known function when entering a busy highway and in which they can detect a lesser magnitude of the lateral environment and in one second mode wherein they serve a lane detection for detecting the largest possible portion of road boundary elements in a wide-angle operation. The various modes can be activated manually and / or into each other, for example, by the driver

be switchable. In a wide-angle mode of the SVA camera, there are optical distortions, especially in the edge region of the image. Any optical distortions operated by side cameras in wide angle mode will be discounted by the recognizer. In a further embodiment of the invention, the side camera as a, for example, on a side mirror of the vehicle arranged blind spot detection camera, hereinafter also BSD camera is formed. The use of such a camera has the advantage that it is already installed in many available vehicle models and is used as part of a driver assistance system.

A particularly preferred embodiment of the invention relates to a vehicle having a multi-camera system in which the lane map can be constructed from the data of at least two cameras. According to the invention z. B. data of one or more front cameras in conjunction with one or more SVA cameras and / or data of one or more rear cameras in conjunction with one or more SVA cameras and / or data of one or more BSD cameras in conjunction with one or more SVA - Cameras and / or data from one or more front cameras in conjunction with one or more BSD cameras and / or data from one or more rear cameras in conjunction with one or more BSD cameras and / or data from one or more front cameras in conjunction with one or more SVSs Cameras and / or data from one or more rear-view cameras in conjunction with one or more SVS cameras and / or data from one or more BSD cameras in conjunction with one or more SVS cameras and / or data from one or more SVS cameras be used with one or more SVA cameras. In particular, the data of two

opposite side cameras are evaluated for lane detection. In a particularly preferred embodiment of the invention, the signals of all available cameras are combined in a central computer, which the

Fahrbahnbegrenzungselemente recognizes from the images and to create the

Lane map merges and processes.

In the case of using the data of two or more cameras having overlapping detection areas, an overlapping area of the detection area of the cameras is determined such that an association of the points detected by one camera on the image area of one camera with those of the other camera defined points on the screen of the other camera is defined. Thus, you can

Objects that are simultaneously visible on two cameras can be localized more reliably.

In the particularly advantageous case that an overlapping area of a side camera with a front camera and / or a rear camera, the detection areas of the Cameras are so mapped and continued so that from the captured images information about the longest possible stretch of

Fahrbahnbegrenzungselementen can be obtained. The information from areas with better image resolution is preferably used in the overlapping area of the detection area of the cameras. Objects that are on the edge of an image of a camera and may not be able to resolve properly here may be in the center of the image in the optimal resolution range of another camera.

From the road boundary elements detected by the side cameras, in one embodiment of the invention, a local curvature of the traffic lane and therefrom a

Curvature determined in a detected section. From the road boundary elements detected by a front camera or a rear camera, curvature values of the lanes can additionally be calculated and compared with the data of the side cameras. The advantage of using the side camera data is that a larger data set than when using the front camera or rear camera data can be used because of a larger section

Fahrbahnbegrenzungselementen can be detected. Information about the

Curvature are especially in driver assistance systems, which a

Include navigation device with a digital road map, particularly advantageous.

In a particularly preferred embodiment of the invention, data of a navigation map is matched with the determined traffic lane map, i. compared, checked and, if necessary, improved. A permanent storage of the lane information, for example, an overstore of the information of the navigation map should also be possible. The digital maps of navigation devices are often made from simple, idealized assumptions. Often, as in the planning of

Motorway routes are common, the routes in straight lines, curves of constant curvature and clothoids for the transition between straight lines and curves of constant curvature distinguished. The clothoids form a transition from the path without curvature to the path of constant curvature, in which the curvature increases or decreases linearly. By calculating the local curvature from the

Lane boundary elements of the side cameras, a map used in the navigation device can be better adapted to the real conditions. But this can also be happen at construction sites, where there are changed track guides, which may not be present in the navigation map.

According to one embodiment of the invention is calculated in particular from

Curve data of the lane and / or the road surface determines a location and orientation of the vehicle on the lane and / or the road. For this purpose, the identification device determines from the available image data information about the curvature of the roadway and / or the traffic lane and information on which lane the vehicle is located. Particularly advantageously, the information about the orientation of the vehicle on the traffic lane and / or the roadway is supplied to a navigation system, in particular a display of a navigation device, whereby the navigation system can also use the information to indicate in a stationary vehicle in which direction the vehicle is aligned Without, for example, information about where the vehicle comes from, which are not available to some available devices when turning on the navigation device.

According to one embodiment of the invention, the road boundary elements recognized by the side camera are used to determine a track width, in particular at the height of the vehicle. From knowing the position of the side camera and determining the location of the roadway boundary elements, i. in particular the height of the roadway boundary element in the middle of the image, a distance of the vehicle to the road boundary element can be determined.

The task is also solved by a driver assistance system with the components:

At least one side camera of the vehicle for detecting an environment of the vehicle,

a component for detecting lane boundary elements of a

Lane from the captured by the at least one side camera images, in particular from the captured images of all cameras located on the vehicle and

a component for creating a lane map.

The driver assistance system is preferably associated with further cameras, in particular a front camera and / or a rear camera, whose images point towards road boundary elements be evaluated, and a fusion component in which the data from the cameras are brought together and / or matched.

Brief description of the drawings

FIG. 1 shows a lane map with a vehicle arranged thereon

Top view,

FIG. 2 shows an image of a roadway taken by a front camera

Road boundary elements,

FIG. 3 shows an image of a roadway taken by a rear camera

Road boundary elements,

Figure 4 shows a left side camera in a first embodiment

taken picture of a lane with Fahrbahnbegrenzungselementen, Figure 5 a from a right side camera in a first embodiment

taken picture of a roadway with Fahrbahnbegrenzungselementen and

Figure 6 shows a left side camera in a second embodiment

taken picture of a roadway with Fahrbahnbegrenzungselementen. Embodiments of the invention

FIG. 1 shows a lane map in plan view. On the lane map, a vehicle 1 is shown, as well as the course of a road 2, which by

Lane boundary elements is limited, here by a solid

Lane marking 3 on the right side and by a solid lane marking 4 left side, the roadway 2 is divided by centrally located the lane dashed lane markings 5 in two lanes 6, 7, wherein a first lane 6 is a lane of the vehicle 1 and a second lane 7 an oncoming lane is. Furthermore, image recording devices arranged on the vehicle 1 are shown, namely a front camera 8, a rear camera 9, a left SVA camera 10 and a right SVA camera 11. In the left side mirror of the vehicle 1, a left SVS camera 32 may be arranged and / or a left BSD camera 12. In the right side mirror of the vehicle 1, a right SVS camera 31 may be arranged and / or a right BSD camera 13. The detection areas of the front camera 8, the rear camera 9, the left SVA camera 10 and the right SVA camera 1 1 form cones 8 a, 9 a, 10 a, 1 1 a, which are shown in dashed lines. The detection areas of the left and right SVS cameras 31, 32 are not shown for the sake of clarity. They essentially correspond to the cones 10a, 11a of the left SVA camera 10 and the right SVA camera 11, with the difference that the conical tip would lie in the exterior mirror.

The detection areas of the left BSD camera 12 and the right BSD camera 13 are also shown in dashed lines, taking into account that the detection areas are obscured by the vehicle 1, i. are limited.

The cone 8a of the detection area of the front camera 8 and the cone 10a of the

Detection range of the left SVA camera 10 and the left SVS camera 32 have a first overlap region 14, which is shown in Figure 1 as a hatched area. Similarly, the cone 8a of the detection area of the front camera 8 and the cone 11a of the detection area of the right SVA camera 11 and the right SVS camera 31 have an overlap area 15, which is shown hatched. Corresponding overlap areas 16, 17 also exist in the

Detection areas of the SVA cameras 10, 1 1 and the SVS cameras 31, 32 with the rear camera 9, which are also shown as hatched areas. The

Overlapping areas of the BSD cameras with the SVA cameras and with the rear cameras are not shown as hatched areas for the sake of clarity, but it is clear where these lie.

In the lane map, the vehicle 1 is to N-W, i. oriented to the northwest, which is represented by an arrow 18 in the vehicle 1, which indicates the sitting position of a driver of the vehicle. The lane 2 is curved to the left in Fig. 1, that is, the vehicle 1 makes a left turn.

Of all available cameras, i. from the front camera 8, the rear camera 9, the left SVA camera 10, the right SVA camera 11, the left BSD camera 12 and the right BSD camera 13, the left and right SVS cameras 31, 32 become images received a central computer, which may be arranged in the vehicle 1 and the one

Recognition device has supplied. This evaluates the pictures in terms of Lane boundary elements 3, 4, 5 and calculated from the course of the lane 2 and its lanes 6, 7, and the orientation of the vehicle in the lane 2 and the lane 6, which is here NW. The information obtained in this way can be compared with the data of a digital map and information of a

Navigation system to be extended. Subsequently, an image of the roadway 2 and the vehicle 1 arranged thereon, as shown for example in FIG. 1, but also in other common perspectives, can be transmitted to a display device in the vehicle 1, which outputs the image to the driver of the vehicle 1. In addition, the lane map obtained from the information is available for other driver assistance systems, such as a lane assistant and / or a collision avoidance assistant.

FIG. 2 shows, by way of example, an image 20 taken by the front camera 8 with the roadway 2 from FIG. 1, with the lanes 6, 7 and the roadway boundary elements 3, 4, 5. The roadway 2 has a curvature to the left, therefore the right-hand ones

Lane marker 3, the left lane marker 4 and the middle lane marker 5 curved to the left.

Each point on the image 20 taken by the front camera 8 is assigned an ordinate value y and an abscissa value x, for example the pixel value of the point. The own lane 6 is characterized in that, in the case where the front camera 8 is located at the center of the vehicle, it is excellent in the abscissa value, i. the x-position, which is in the middle of the picture. If the front camera 8 is not arranged in the middle of the vehicle, the recognition device can also convert the received image to an image which would be taken from the center of the vehicle 1.

FIG. 3 shows, by way of example, an image 21 of the road 2 from FIG. 1 taken by the rear camera 9 with the lanes 6, 7 and the road boundary elements 3, 4, 5.

Each point on the image 21 taken by the rear camera 9 is assigned an abscissa value x and an ordinate value y, for example the pixel value of the point. The

Rear camera 9 indicates a curvature of the road 2 which, in the case that the front camera 8 and the rear camera 9 are aligned in the direction of the longitudinal axis of the vehicle 1 and the vehicle 1 is on a portion of constant curvature, is opposite to that of the front camera 8 represents. By the information of the front camera 8 and the rear camera 9, the lane map shown in Figure 1 can being constructed. Here, the information of the front camera 8 and the rear camera 9 are assembled and in a non-detectable area in which the vehicle 1 is itself and which outside of the cone 8 a of

Detection range of the front camera 8 and outside of the cone 9a of the

Detection range of the rear camera 9 is extended by calculation methods,

for example, interpolated.

FIG. 4 shows a previous image 22 of a left SVS camera 32 or a left SVA camera 10, which is operated in a wide-angle mode. The wide-angle mode results in a pillow-shaped distortion, which is recognizable by means of a pillow-shaped distorted horizon 30.

The left SVS camera 32 and left SVA camera 10, respectively, take the left turn, i. the curvature of the dashed lane markings 5 arranged centrally of the roadway 2 after a deduction of the pincushion distortion as a substantially convex

Dash line on and the curvature of the left-side lane mark 4 as a convex line, from which the curvatures locally, i. at every point of the

Lane markings 4, 5 can be calculated. By left SVS camera 32 and the left SVA camera 10 in the case of the left turn a very large road section can be surveyed, allowing an accurate determination of the course of the lane 2.

In the image 22 of the left SVS camera 32 and the left SVA camera 10 would, after deduction of eventual caused by the wide-angle lens optical

Distortions, sloping roadway boundary element, i. one

Lane boundary element whose ordinate y on the abscissa x shows a substantially linear behavior, indicate that the vehicle 1 is not straight in the

Lane 6 is located. This information is also used to create the lane map according to Figure 1 and in particular for illustrating the orientation of the vehicle 1 in the

Lane map used. From the x with respect to the abscissa x measured height 25 of the dashed

Lane marking 5 on the image 22, ie its ordinate y in the center of the image 22 and in the knowledge of the position of the left SVS camera 32 and the left SVA camera 10, a distance of the vehicle 1 is calculated to the lane marking 5 at vehicle height , This information is also used to create the lane map of Figure 1 and in particular for the arrangement of the vehicle 1 in the lane map. FIG. 5 shows an image 23 of a right-hand SVS camera 31 or a right SVA camera 11 operated in the wide-angle mode, which corresponds to the images shown in FIGS. 2 to 4. The right SVS camera 31 or right SVA camera 1 1 receives the curvature of the roadway 2 in the form of a concave line 3. A

Recognition device of the vehicle 1 can locally calculate the curvature from the line 3, and from the values obtained thereby, in conjunction with the values obtained from the images shown in FIGS. 2 to 4, those shown in FIG

Create lane map. In particular, in the case of a right turn, not shown, a very large area of the roadway 2 in the image 23 of the right SVS camera 31 and the right SVA camera 1 1 can be measured.

In the image 23 of the right SVS camera 31 and the right SVA camera 1 1 would, after deduction of any optical distortions through the wide-angle lens, oblique pavement marking, i. a lane marking whose ordinate y over the abscissa x shows a substantially linear behavior, indicate that the

Vehicle 1 is not just in the lane 6. This information will also become

Creation of the lane map according to Figure 1 and in particular for the representation of

Orientation of the vehicle 1 used in the lane map.

From the height 27 of the lane mark 3 centered on the abscissa x on the image 23, i. its ordinate y in the center of the image 23 and in the

Knowing the position of the right SVS camera 31 or the right SVA camera 1 1, a distance of the vehicle 1 to the lane marking 3 is calculated at the vehicle height. This information is also used to create the lane map of Figure 1 and in particular for the arrangement of the vehicle 1 in the lane map.

FIG. 6 shows an image 24 of a left BSD camera 12 which is one of the previous ones

corresponds. The image 24 corresponds in principle to that of the rear camera 9

taken picture 21 of the lane 2 in Figure 3 with the difference that here a lateral offset and possibly a different orientation of the cameras to

are considered, because the BSD camera 12, 13 is usually installed in a side mirror of the vehicle 1 with a view towards the rear of the vehicle 1. Of the

Detection range of the BSD cameras 12, 13 is limited by the vehicle 1, which is shown in Figure 6 as a hatched left-side region 26. The pictures of the BSD Cameras 12, 13 are like those of the other cameras for detecting the

Roadway boundary elements 3, 4, 5 evaluated. The invention is not limited to the embodiments described herein and the aspects highlighted therein. Rather, within the scope given by the appended claims a variety of modifications are possible, which are within the scope of expert action.

Claims

Claims 1. A method for creating a lane map, wherein a side camera (10, 1 1, 12, 13, 31, 32) of a vehicle (1) detects an environment of the vehicle (1), wherein from the images (22, 23, 24) of the side camera (10, 1 1, 12, 13, 31, 32)

Fahrbahnbegrenzungselemente (3, 4, 5) of a roadway (2) are recognized and the recognized lane boundary elements (3, 4, 5) are used for a lane map.

2. The method according to claim 1, characterized in that the side camera as a particular on a side mirror of the vehicle (1) arranged surround view system camera (31, 32) is formed.

3. The method according to claim 2, characterized in that the side camera is designed as a sideview assistant camera (10, 1 1) and the sideview assistant camera (10, 1 1) is operated in a wide angle mode.

4. The method according to claim 1, characterized in that the side camera as a particular on a side mirror of the vehicle (1) arranged blind spot detection camera (12, 13) is formed.

5. The method according to any one of the preceding claims, characterized in that the lane map from the data of at least two cameras (8, 9, 10, 1 1, 12, 13, 31, 32) is constructed, which overlapping detection areas (8a, 9a , 10a, 11a, 12a, 13a).

6. The method according to any one of the preceding claims, characterized in that from the of the at least one side camera (10, 1 1, 12, 13, 31, 32) detected

Lane boundary elements (3, 4, 5) a curvature of the road (2) is determined in a detected section.

7. The method according to claim 6, characterized in that a position and orientation of the vehicle (1) on the lane (6) and / or the roadway (2) is determined.

8. The method according to any one of the preceding claims, characterized in that data of a navigation map are compared with the determined lane map.

9. The method according to any one of the preceding claims, characterized in that the of the at least one side camera (10, 1 1, 12, 13, 31, 32) detected

Roadway boundary elements (3, 4, 5) are used to determine a track width.

10. Driver assistance system with the following components:

- A side camera (10, 1 1, 12, 13, 31, 32) of the vehicle (1) for detecting an environment of the vehicle (1),

- A component for the detection of road boundary elements (3, 4, 5) of a roadway (2) from the by the side camera (10, 1 1, 12, 13, 31, 32) detected

Images (22, 23, 24), in particular from the acquired images (20, 21, 22, 23, 24) of all the cameras (8, 9, 10, 11, 12, 13, 31, 32) and

a component for creating a lane map.