DE102016202594A1 - Method and device for interpreting a vehicle environment of a vehicle and vehicle - Google Patents

Abstract

The invention relates to a method for interpreting a vehicle environment (106) of a vehicle (100). The method includes a step of providing an image of the vehicle environment (106), a step of forming a relational information using a first position of a first object (115) in the image and a second position of a second object (116) in the image the relational information represents a spatial relationship between the first object (115) and the second object (116) in the vehicle environment (106), and a step of using the relational information to interpret the vehicle environment (106).

Description

State of the art

The invention is based on a device or a method according to the preamble of the independent claims. The subject of the present invention is also a computer program.

Driver assistance systems (FAS) typically use sensors with different physical characteristics to detect the situation in which one's own vehicle is located. The information from the sensors is processed in such a way that driver assistance functions - for example the longitudinal guidance of the vehicle, also called "ACC" (Adaptive Cruise Control) - can be realized.

Disclosure of the invention

Against this background, the method presented here introduces a method for interpreting a vehicle environment of a vehicle, furthermore a device which uses this method, a vehicle, and finally a corresponding computer program according to the main claims. The measures listed in the dependent claims advantageous refinements and improvements of the independent claim device are possible.

Using positions of objects in an image of a vehicle environment of a vehicle to form a relation information representing a spatial relation between the objects in the vehicle environment enables the introduction of relative object descriptions for the realization of driver assistance functions and / or automated driving functions and / or automatic driving functions.

The image may be a near-sensor image of a scene. The image can be understood as a description of the driving situation around the vehicle. The image may, for example, be a central perspective image or an image representing the scene from a bird's-eye view. The image may represent a representation of the scene as viewed by a video sensor.

In particular, the image may be an image of a video sensor of the vehicle which, in accordance with an embodiment of the concept proposed herein, analyzes the relative object descriptions and relationships, combines them in an internal interpretation, and the result of the interpretation and / or Objects and / or the relations between the objects on an external interface to a driver assistance system and / or driving functions of the vehicle provides.

A method is provided for interpreting a vehicle environment of a vehicle, the method comprising the steps of:
Providing an image of the vehicle environment;
Forming a relational information using a first position of a first object in the image and a second position of a second object in the image, the relational information representing a spatial relationship between the first object and the second object in the vehicle environment; and
Use the relationship information to interpret the vehicle environment.

This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

The vehicle environment may be an environment of the vehicle that is relevant for driver assistance functions of the vehicle. The vehicle environment may include infrastructure and / or busy or inanimate objects in the relative vicinity of the vehicle. The vehicle environment may vary depending on whether the vehicle is moving in a rural or urban environment and also depending on which road class a road currently occupied by the vehicle belongs to. The optical sensor may be part of a video camera associated with a driver assistance system of the vehicle, which is arranged behind the windshield of the vehicle. The image may represent a near-sensor image of a scene in the vehicle environment. The image can be, for example, a central perspective image. Also, the image can represent a bird's eye view. The image may represent an image generated by an optical sensor of the vehicle oriented in a direction of travel of the vehicle. The image can accordingly be a digital video image. In accordance with the position of the camera, the image can depict a section of the vehicle surroundings lying in front of the vehicle in the direction of travel, in particular a road section in front of the vehicle in the direction of travel and infrastructure areas adjoining it laterally, and objects located in the cutout.

The objects may be, for example, road users such. B. vehicles, pedestrians or cyclists, a drivable area of the scene shown - a so-called open space -, the lanes of the road, traffic infrastructure, lateral boundaries such. As guardrails and posts, green strips, etc., vegetation, buildings, the sky, etc. act. In the context in question, an object can be understood to be a real object of the real vehicle environment or a virtual object representing a real object in the digital image of the vehicle environment. A position of an object may correspond to a position of the pixels depicting the object in the digital image. The spatial relationship between the first object and the second object may, for example, designate a distance between the objects, a distance of at least one object from the vehicle, a movement direction and / or speed of at least one of the objects, or a partial concealment of one of the objects by the other , The relationship information may be determined using a number and size of pixels of the image representing a relation between the first and second objects in the image, such as a distance of the objects.

For an advantageous tracking of at least one object in the vehicle environment, the steps of the method can be carried out repeatedly.

In a preferred embodiment, the concept presented here describes an interface of a video sensor which is optimally adapted to the characteristics of the sensor and which directly uses the physical properties of a video sensor. As a result, performance-reducing transformations of the sensor information - for example, from a frontal view of a situation in a grid representation (network representation) or bird's eye view - are avoided or supplemented by further direct video information. The information derived directly from the physical properties of the sensor can supplement the interfaces, for example in the bird's-eye view projection. This procedure increases the amount of information that can be used by the video sensor. This goes so far that in the future the number of additional sensors for implementing driver assistance functions can be greatly reduced. This results in great advantages in the market for sensors that make the fullest possible use of the potential of the respective sensors. The concept presented here also makes it possible to increase the range and / or availability of the recorded objects within the driving situation.

According to an embodiment, in the step of using, the relation information may be used for a lateral steering function, for example a lane keeping function, and / or for a longitudinal control of the vehicle. The longitudinal control includes, for example, the consequences of a vehicle with a. Thus, the lane keeping function can be realized particularly robust.

It is also favorable if the method has a step of associating the first object and / or the second object with an object class. Accordingly, in the step of using, the object class may be further used to interpret the vehicle environment. According to this embodiment of the method, the vehicle environment can be advantageously interpreted with little computational effort.

According to another embodiment, the method may include a step of forming concealment information. The concealment information may represent occlusion of the first object by the second object in the image. Accordingly, in the step of using, the concealment information may be further used to interpret the vehicle environment. The concealment information may readily assist in determining a spatial relationship of the first object relative to the vehicle.

For example, in the step of forming a relation, further relationship information may be further formed using another first position of another first object in the image and another second position of another second object in the image. In this case, the further relation information can represent a spatial relation between the further first object and the further second object in the vehicle environment. Accordingly, in the step of using further, the further relation information may be used to interpret the vehicle environment. The further relation information can readily be used for a verification of the relationship information.

According to a particular embodiment, the step of using may comprise a step of generating a relational value between the relational information and the further relational information. Thus, the vehicle environment can be interpreted using the relational value.

This embodiment of the method allows for advantageous propagation of the vehicle environment interpretation to the distance.

For example, in the step of using the relational value for a following journey behind another vehicle may be used.

It is also favorable if the method has a step of determining a bearing of one of the objects or of another object with respect to the optical sensor. Accordingly, in the step of using, the bearing may also be used to interpret the vehicle environment. Thus, the polar detection principle of the optical sensor can be used in its direct use advantageous for the interpretation of the vehicle environment.

The approach presented here also creates a device that is designed to perform the steps of a variant of a method presented here in appropriate facilities to drive or implement. Also by this embodiment of the invention in the form of a device, the object underlying the invention can be solved quickly and efficiently.

For this purpose, the device may comprise at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading sensor signals from the sensor or for outputting data or control signals to the sensor Actuator and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The arithmetic unit may be, for example, a signal processor, a microcontroller, an ASSP, an SOC or the like, wherein the memory unit may be a flash memory, an EPROM or a magnetic memory unit. Data can also be stored in a cloud decentralized from the vehicle. The communication interface can be designed to read or output data wirelessly and / or by line, wherein a communication interface that can read or output line-bound data, for example, electrically or optically read this data from a corresponding data transmission line or output to a corresponding data transmission line.

In the present case, a device can be understood as meaning an electrical device which processes sensor signals and outputs control and / or data signals in dependence thereon. The device may have an interface, which may be formed in hardware and / or software. In the case of a hardware-based embodiment, the interfaces can be part of a so-called system ASIC and / or system ASSP, for example, which contains a wide variety of functions of the device. However, it is also possible that the interfaces are their own integrated circuits or at least partially consist of discrete components. In a software training, the interfaces may be software modules that are present, for example, on a microcontroller in addition to other software modules.

In an advantageous embodiment, the device performs a formation of relationship information using positions of objects in a digital image of a vehicle environment of a vehicle. For this purpose, the device can access video signals of a video sensor installed in the vehicle.

Furthermore, a vehicle with an optical sensor, a driver assistance system and a device as described above is presented. The device is electrically conductively connected to the optical sensor and / or the driver assistance system.

The driver assistance system may provide a lane keeping assist or functions such as autonomous driving or automatic braking of the vehicle.

Also of advantage is a computer program product or computer program with program code which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and / or controlling the steps of the method according to one of the embodiments described above is used, especially when the program product or program is executed on a computer or a device.

Embodiments of the approach presented here are shown in the drawings and explained in more detail in the following description. It shows:

1 a vehicle with a driver assistance system according to an embodiment;

2 a block diagram of an apparatus for interpreting a vehicle environment of a vehicle according to an embodiment;

3 a flowchart of a method for interpreting a vehicle environment of a vehicle according to an embodiment;

4 a schematic representation of a central perspective image of a freeway scenario for forming a relationship information according to an embodiment;

5 a schematic representation of a central perspective image of an urban Scene for forming a relationship information according to an embodiment; and

6 a schematic representation of another central perspective image of an urban scene for a formation of a relationship information according to an embodiment.

In the following description of favorable embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and similar acting, wherein a repeated description of these elements is omitted.

1 shows a vehicle in a plan view 100 , At the in 1 The embodiment shown is the vehicle 100 around a car. It may be at the vehicle 100 also to another road-bound vehicle such. As a truck or a motorcycle act.

The exemplary vehicle 100 has a plurality of video cameras in different positions in the vehicle 100 are installed. A first video camera 102 is behind a windshield of the vehicle 100 installed and on one in one direction of travel 104 of the vehicle 100 in front of the vehicle 100 lying section of a vehicle environment 106 of the vehicle 100 directed. A second video camera 108 is in a right side mirror of the vehicle 100 installed and on one in the direction of travel 104 laterally to the right of the vehicle 100 lying section of the vehicle environment 106 of the vehicle 100 directed. A third video camera 110 is in a left side mirror of the vehicle 100 installed and on one in the direction of travel 104 laterally to the left of the vehicle 100 lying section of the vehicle environment 106 of the vehicle 100 directed. A fourth video camera 112 is in a rear area of the vehicle 100 installed and on one in the direction of travel 104 behind the vehicle 100 lying section of the vehicle environment 106 of the vehicle 100 directed.

The vehicle 100 also has a driver assistance system 114 on, for example, via a CAN bus of the vehicle 100 with the video cameras 102 . 108 . 110 . 112 electrically conductive is connected. Optical sensors or video sensors of video cameras 102 . 108 . 110 . 112 generate video signals of the vehicle environment visible to them 106 in the associated image processing facilities of the video cameras 102 . 108 . 110 . 112 to digital images of the detected vehicle environment 106 are processed.

About downstream and / or in the video cameras 102 . 108 . 110 . 112 contained functions are this image information via suitable interfaces for the realization of driver assistance functions to the driver assistance system 114 provided.

The video cameras 102 . 108 . 110 . 112 capture as shown in 1 a situation in which the vehicle is 100 located through the surface of their respective image sensors and recognize therein for the driver assistance function relevant objects or other information. Exemplary are in the representation in 1 a first object 115 and a second object 116 in the vehicle environment 106 shown in the in 1 shown exemplary driving situation in the detection range of the first video camera 102 lie.

For example, an optical sensor detects 118 the first video camera 102 in the driving situation of the vehicle 100 one in the direction of travel 104 in front of the vehicle 100 lying road section and the two objects 115 . 116 and provides a corresponding image signal. The image processing device of the first video camera 102 generates a central perspective digital image of the detected environment using the image signal 106 in which infrastructure elements as well as the objects 115 . 116 the captured section of the vehicle environment 106 are shown flat. The central perspective view is chosen here merely by way of example. For example, a bird's-eye view is also possible.

According to the concept proposed here, the vehicle is 100 with at least one device 120 to interpret the vehicle environment 106 of the vehicle 100 fitted. The device 120 is electrically conductive with the first video camera 102 connected and trained to using positions of virtual images of the objects 115 . 116 in one of the video camera 102 generated central perspective image of the vehicle environment 106 to form at least one relationship information that is a spatial relation between the real objects 115 . 116 in the vehicle environment 106 represents, and use this relationship information to the vehicle environment 106 to interpret.

At the in 1 the embodiment shown is the device 120 Part of the first video camera 102 , For example in the form of an interface or an interface of the video camera 102 that optimally match the characteristics of the sensor 118 adapted, and which uses the physical properties of a video sensor directly.

According to one embodiment, the rest are in the vehicle 100 installed video cameras 108 . 110 . 112 with its own device 120 fitted.

In an alternative variant, the device 120 from the video camera 102 removed in the vehicle 100 installed and, for example via the CAN bus electrically conductive with the video camera 102 coupled. For example, the device 120 also part of the driver assistance system 114 be and be trained to digital images of all available cameras 102 . 108 . 110 . 112 central to relation information for the interpretation of the vehicle environment 106 to process. In addition to the digital images can also descriptions of the vehicle environment of the respective cameras 102 . 108 . 110 . 112 to the central unit 114 be transmitted.

Task of the device 120 is the provision of relative object descriptions and relationships based on a constellation of the vehicle environment 106 captured objects 115 . 116 in the central perspective image of the flat image sensor of the video camera 102 based.

A driver assistance camera system designed in accordance with the concept proposed here has high performance combined with low system complexity and low system costs.

According to the concept presented, driver assistance functions of the driver assistance system 114 based on the relative descriptions of the video sensor 118 Implement an improved control strategy that enhances the quality of each function. By way of example, a (relative) collision time can be governed more robustly than absolute coordinates that are determined indirectly from the video material. This leads to advantageous behavioral patterns for a video-based driver assistance system 114 from. For example, for a following trip on a lane, the relative distance to a preceding vehicle is regulated relative to the distance to the lane course. For a ride on a free surface is an orientation at relative distances to lateral boundaries such as crash barriers.

In support of the scene interpretation can also be information in the vehicle 100 deposited digital card can be used to advantage. If the road shape does not change on a current stretch of road, as is typical for a run on freeways without ascents and descents, the situation can be relatively reliably assumed to be uniform and constant. Here should be the near and far range of the camera 102 have great similarities in the infrastructure of the road. The relative descriptions of the scene content may be advantageously used by driver assistance functions optimized for this interface format.

According to one embodiment, in addition to or as an alternative to at least some of the video cameras 102 . 108 . 110 . 112 Radar sensors in the vehicle 100 be installed. These detect the backscatter signal of objects 115 . 116 and measure the distance of objects 115 . 116 , There are other sensors such as Lidar, ultrasound, etc., conceivable in this composite.

The concept of relative object descriptions presented here for use in novel video sensor interfaces for driver assistance systems presented here can also be implemented in combination with driver assistance, in which the information from the different sensors is fused in a driver assistance system for optimizing the functional behavior. Before the merger, the information is converted into a uniform, fusible representation. Very often, Cartesian, absolute representations of the situation in which the vehicle is located, are used. This form of representation is referred to as a so-called grid representation (network representation). This unified modeling of the scene implies that the sensors, as a result of being converted into the fusion-capable format, can generally no longer provide their data and information in their native coordinate system.

Be the information of the video sensor 118 Transformed into a grid representation, the frontal viewing direction from which the scene was originally taken is rotated by almost 90 ° and the scene viewed from above. Originally contained in the image of the scene relationships between objects 115 . 116 get lost. Functions based on this "classic" grid representation use, for example, the video sensor 118 only suboptimal. This can be compensated for by the addition of the perspective information based on the relationship information according to the approach proposed here.

With the concept proposed here, the sensor setup of the vehicle 100 be advantageously simplified. Thus, a reduction to only one radar sensor and a video sensor can be realized. It is also conceivable that even complex driving functions are realized on a single main sensor; Here is above all the video sensor 118 because it has the widest range and can provide the most complete information about a driving situation.

2 shows a block diagram of an embodiment of the device 120 for interpreting a vehicle environment of a vehicle. The contraption 120 has an educational institution 200 and a use device 202 on.

Via a suitable interface-for example to an image sensor of a video camera of a vehicle-a central perspective image produced by the image sensor is produced 204 a vehicle environment of the vehicle to the educational institution 200 provided. The educational institution 200 is configured to obtain a relation information using a first position of a first object in the image and a second position of a second object in the image 206 to form and to the use device 202 provide.

The relation information 206 representing a spatial relation between the first object and the second object in the vehicle environment becomes the use device 202 processed to interpret the vehicle environment with regard to possibly required driver assistance functions.

According to an embodiment of the device 120 will be in the educational institution 200 further using a further relationship information using a further first position of a further first object in the image and a further second position of a further second object in the image 208 is formed, which represents a spatial relation between the further first object and the further second object in the vehicle environment. The further information about the relationship 208 is also in the use device 202 processed to further interpret the vehicle environment in terms of possibly required driver assistance functions.

A result of the in-use device 202 Carried out interpretation of the vehicle environment is by means of a result information 210 in suitably prepared form, for example, provided to a driver assistance system of the vehicle, which controls, for example, a lane keeping function of the vehicle using this.

According to an embodiment of the device 120 is the educational institution 200 further configured to associate the objects shown in the digital image of one or more classes of objects. Accordingly, in the use device 202 the vehicle environment is further interpreted using the classification of the objects.

Alternatively or additionally, the educational institution 200 Furthermore, in the case of an object being obscured by another object in the central perspective image, it can also be designed to conceal concealment information 212 to form and to the use device 202 provide. Accordingly, in the use device 202 the vehicle environment further using the concealment information 212 interpreted.

The number of relations between objects is not on, for example, the relations 206 . 208 . 212 . 214 limited. It can be any number of relations of types 206 . 208 . 212 . 214 give or other meaningful relations that are not explicitly mentioned here.

Due to the polar measuring principle of the optical sensor, according to a further exemplary embodiment, it is appropriate in the educational institution 200 a bearing 214 determine at least one of the detected by the sensor and shown in the image objects and to the use device 202 provide. This uses the bearing 214 for an interpretation of the vehicle environment, in particular with regard to a path of the vehicle optionally crossing moving objects such. B. pedestrians or cyclists.

3 shows a flowchart of an embodiment of a method 300 for interpreting a vehicle environment of a vehicle. The procedure 300 can of the in the 1 and 2 shown device for interpreting a vehicle environment.

In a step of providing 302 a central perspective image of the vehicle environment is provided which was generated by an optical sensor of the vehicle oriented in a direction of travel of the vehicle. In a step of making 304 At least one relationship information representative of a spatial relationship between objects in the vehicle environment is formed using positions of objects in the image. In a step of using 306. For example, the relationship information is used in a driver assistance device or a computing unit of a camera of the vehicle associated with the optical sensor in order to interpret the vehicle environment.

For a tracking or tracking function of moving objects in the vehicle environment, the steps 302 . 304 . 306. be executed repeatedly.

When using 306. or interpreting the information formed 304 Further information may emerge as educated information 304 also in the procedure 300 be used.

4 shows for an illustrative explanation of the concept presented here the relative object descriptions as a basis for Driver assistance functions schematically an exemplary central perspective digital image 400 a vehicle environment. Shown here is a typical highway scenario as seen and imaged by an image sensor of a video camera mounted behind the windshield of an exemplary vehicle - the vehicle itself. The own vehicle is thought before in 4 shown schematic perspective image 400 and is in 4 through a line 402 representing a position of a center longitudinal axis of the own vehicle with respect to that in the digital image 400 shown driving scene shows.

In the central perspective image 400 in 4 For example, a highway scene is schematically shown as a driving situation. The scene consists of an open space 404 , on the lane by markings 406 Marked are. Lateral limits one through the open space 404 formed lane are by the side markings 408 bounded. Next to the open space 404 are guardrails 410 established. In front of your own vehicle drives a truck 412 , The truck 412 drives a lane 414 the roadway 404 who is also driven by his own vehicle. The marks 406 , the page markers 408 , the guardrails 410 , the truck 412 and the lane 414 form objects in the image 400 ,

A new relative description of the in the digital image 400 shown scene succeeds as follows. Of the own vehicle representing center axis 402 becomes a lateral distance 416 to the right lateral lane boundary 408 measured. The truck drives ahead 412 with a lateral distance 418 to the right lateral lane boundary 408 , The lateral distance is measured 418 from a central axis of the truck 412 representing another line 420 to the lane boundary 408 ,

According to the concept proposed here, in the central perspective image 400 represented relations between objects 406 . 408 . 410 . 412 . 414 an interpretation of the vehicle environment of the own vehicle based. Relations describe relations between the objects 406 . 408 . 410 . 412 . 414 of the picture 400 such as a "neighborhood relationship". In 4 this is given by the further vehicle 412 holding your own vehicle at the guardrail 410 along ahead. Such relative relationships are similar to interpreting a situation by humans and offer great potential in understanding a scene or driving situation. Thus, a possible action of an assistance function of the own vehicle may be exactly this vehicle 412 and the guardrail 410 to follow. For example, if the vehicle ahead 412 can not be detected well, the vehicle follows the guardrail 410 ,

According to one embodiment, an on-vehicle device for interpreting a vehicle environment first determines using the lateral distance 416 a relation information and using the lateral distance 418 another relationship information. Using the relation information and the further relation information, the device subsequently generates a relation value which contains an interpretation result of the image 400 shown in the driver assistance system of the own vehicle for a function of a subsequent drive behind the truck 412 can be used.

The own vehicle can thus functionally the preceding vehicle 412 Follow by taking the side clearance 416 to the same mark 408 stops without the preceding vehicle 412 exactly absolute - z. B. by a Cartesian coordinate system - must be measured. When driving, care must be taken only that the own vehicle, for example, when approaching not with the preceding vehicle 412 collided. The requirements for detection of the preceding vehicle 412 can thus be advantageously reduced. Instead, it is observed whether the of the distances 416 . 418 to the common object 408 derived relational value remains unchanged.

Object descriptions used in the approach presented here are more robust than other forms of description because they track structures that are directly visible to a camera, or use direct metrics, such as the. For example, the size of an object 406 . 408 . 410 . 412 . 414 in the picture 400 or the distances of objects 406 . 408 . 410 . 412 . 414 among themselves. In addition to increasing the robustness succeeds in increasing the range of the video sensor.

An alternative reference object for vehicle environment interpretation may be through the side guardrail 410 be given to which the truck 412 a relative distance 422 Has. Even your own vehicle has a corresponding distance 424 to the same guardrail 410 , Another exemplary alternative is a lateral distance 426 from the truck 412 to the mark 406 There may be further relationship information between objects 406 . 408 . 410 . 412 . 414 determined and based on the ambient interpretation.

Relationship information can also be within an object 406 . 408 . 410 . 412 . 414 be determined. So has the middle lane 414 in the vicinity of a first measured width 428 , Assuming that the lane width does not change in the current scene in the far range, then a second measured width 430 in the distance almost identical to the value 428 be. By using this inner relation, an increase in robustness and range is also achieved here. The measured value 430 , for which the camera has determined a metric value in its own vehicle, can also measure other objects in the vicinity of a position of the measured value 430 be used.

At the in 4 shown example of the relative description of a traffic situation in the possible scenario highway shows vividly the approach presented here, that for the interpretation of an image content of an image 400 In a vehicle environment, first the perspective of the camera in the own vehicle is maintained and there a closed and complete description of the situation is generated. A projection onto a bird's-eye view is not initially carried out in order to obtain an added value of information and to avoid loopholes and errors justified by conversion technology. Instead, maximum accuracy and availability of information is obtained directly from the video sensor.

In particular, the polar properties of the measuring principle of the video camera are taken into account. For example, the observation of an object represents 406 . 408 . 410 . 412 . 414 at a certain angle - the bearing - a robust and direct measure. The accuracy and resolution of a measurement of the distance and the object size is higher due to the polarity in the close range of the camera than in the far range. Size changes can also be measured directly from the object appearance and used to calculate, for example, a time-to-contact or TTC value. "Time-to-Contact" describes the time after which the vehicle would collide with a specific observed object.

Are objects 406 . 408 . 410 . 412 . 414 recognized and optionally classified, they can be tracked in the area image signal of the video sensor in the sense of a so-called tracking. Regions with uniform track vectors, so to speak, an optical "flow" of a picture sequence, provide an additional indication of objects in the vehicle environment, in particular moving objects such. B. the truck 412 , dar.

The mark 408 forms in the in 4 shown exemplary image 400 a common relative reference object for the objects 402 and 412 , About the common relative reference object 408 succeeds in linking the near vehicle area with the far area.

In the close range in front of the camera, it is often additionally possible to obtain reliable information for measuring the detected objects 406 . 408 . 410 . 412 . 414 to win. These are the basis for a Cartesian description of the scene through the measurement of lateral (lateral) and frontal (longitudinal) distances and are a prerequisite for a bird's eye view or grid representation of the scene. At close range, a three-dimensional or 3D survey of the scene succeeds. Such surveys can be propagated via the relations suitable in the far field of the scene. These strategies are also used in human vision.

As an example, be a visually captured lane 406 mentioned as an example object whose width will not change much in the course of the near to the far range. Based on this width, for example, there is the possibility of the width of other vehicles such as the truck 412 to estimate or propagate in the distance. In the long-distance area, especially two-dimensional or 2D features in the perspective image 400 evaluated. The device for interpreting a vehicle environment is designed to create a scene interpretation of the current driving situation from the combination of all available image properties and thereby achieves a high degree of completeness.

For example, the image properties available according to the approach proposed herein include relative object relationships of the objects 406 . 408 . 410 . 412 . 414 like neighborhoods and directions of movement, the propagation of object relations in the scene - as with the example of the guardrail 410 Furthermore, specific object descriptions for the near area (3D) and the far area (2D) of the camera, wherein relations can be drawn or propagated between the information from the near and far areas of the scene, and the use of direct video sensor or camera sizes in Reference to neighborhoods, the surface appearance of objects, bearing, TTC, polar property of the video sensor, etc.

5 To illustrate the concept of relative object descriptions presented herein as a basis for driver assistance functions, schematically illustrates an exemplary central perspective digital image 500 a vehicle environment in a typical urban environment. Shown here is a possible urban scenario as seen and imaged by an image sensor of a video camera mounted behind the windshield of an exemplary vehicle of its own. Here too is your own vehicle thought before the in 5 shown schematic perspective image 500 ,

In the in 5 shown example of the relative description of a traffic scene using another embodiment of a central perspective image 500 a vehicle environment occurs a person 502 from a footpath 504 on the lane 414 who is in the roadway 404 located. Between the lane 414 and the footpath 504 there is a curb 506 , Right of the footpath 504 there is a building 508 , The person or the pedestrian 502 occurs behind a parked another vehicle 412 on the street and enters a pedestrian crossing or zebra crossing 510 ,

Also at the in 5 shown urban scenario is used as fully as possible for the interpretation of the vehicle environment, the area-wide detection of situations by the image sensor or camera lmager installed in the own vehicle video camera during image acquisition. The on-vehicle device for interpreting the vehicle environment provides a position of the pedestrian 502 in the digital image 500 in relation to a position of the vehicle 412 to use a distance 512 between the pedestrian 502 and the vehicle 412 to form a relationship information.

The relation information describes here a neighborhood relation between the objects 502 . 412 , In addition, here the device determines concealment information, which is a partial occlusion 513 of the pedestrian 502 through the vehicle 412 represents. From these relations, for example, geometrical boundary conditions of the scene in the image can be obtained 500 derived. This can be used to determine that a longitudinal position 514 the person 502 - or the distance in front of the own vehicle - behind a longitudinal position 516 one end of the vehicle 412 lies. An evaluation of the relation that covers the object 513 describes, supports the metric interpretation of the scene here. A function based on this can thus have a higher performance, eg. B. have a higher detection rate with lower error detection rate.

Another relationship information in the digital image 500 is by a distance 518 the person 502 from the curb 506 certainly. A complementary content describes another relationship information that is at a distance 520 the person 502 based on the roadway center. According to embodiments, these relationship information may be mutually supportive and / or reduce errors in measurements. Further, using the occlusion information, the measurement of the distance 518 be supported.

In the digital image 500 the person moves 502 over the pedestrian crossing or zebra crossing 510 , There is therefore a local association relation 522 , here specifically a "located-on-it" -relation 522 , between the pedestrian 502 and the object 510 , Also using the "find-yourself-up" relation 522 the device can determine a relationship information. This can be used in multiple ways to interpret the vehicle environment. First, a classification of the pedestrian crossing 510 an object hypothesis of the person 502 support and vice versa. On the other hand, the combination of the pedestrian-classified object 502 and the classified as a pedestrian crossing object 510 a (probable) intention of the pedestrian 502 be derived via the pedestrian crossing 510 the roadway 414 to cross. Likewise, the on the occlusion relation 513 based concealment information for environmental interpretation, etc.

The described neighborhood relations thus initially facilitate the interpretation process in the video sensor and subsequently provide important information for driving functions based on the relative information on the external interfaces of the camera. Locally linked areas in the picture 500 - for example, the sidewalk 504 - can also be used to interpret the meaning. In particular, the interpretation can be propagated from the near-end of the video sensor to the far-end. In addition to the described relations, which can be determined at a time or in an image of an image sequence in the scene, the dynamics of object movements can be used for the description by means of relations.

6 schematically shows another exemplary central perspective digital image 600 a vehicle environment in a typical urban environment created by a video camera of a vehicle moving in the vehicle environment. What is shown here is essentially the urban scenario 5 without the parked vehicle.

The representation in 6 illustrates the polar measurement of an object - here the pedestrian - made possible by the concept presented here 502 - using polar coordinates of the pedestrian 502 in the image 600 ,

The device for interpreting the vehicle environment uses direct measures that are determined by the video sensor. This is how a polar distance describes 602 a distance between your own vehicle and the person 502 , This is at a suitable angle under a polar angle 604 with the center line 402 of your own vehicle. Based on these distance and angle values, the device determines a bearing of the pedestrian 502 in relation to the optical sensor in the vehicle and uses this bearing to interpret the vehicle environment.

In one embodiment, the environmental interpretation device further utilizes expansion 606 the person 502 in lateral and / or longitudinal direction as well as an object height 608 the person 502 , Such local dimensions 606 . 608 in the digital image 600 are measured in image pixels and / or metric sizes and / or angular dimensions.

In a tracking or tracking function of the concept presented here, by tracking or tracking objects such as the pedestrian 502 Bearing changes or changes to the object size 608 or object extent 606 be measured. Time-varying bearings indicate that your own vehicle is not with the object 502 colliding, a time-unchanged bearing, however, shows a probable collision with the object 502 at.

Changing bearings between objects can also be used to determine the relative distance or depth estimate of an object. This depth estimate can be used as further support to interpret the scene. Here, the parallax of the various objects can be taken into account.

The basis of the 1 to 6 The presented techniques for interpreting the vehicle environment offer the possibility of a bottom-up analysis of a central perspective digital image 204 . 400 . 500 . 600 presented scene. In particular, distance relations, neighborhood relations, relations between common reference objects, occlusion relations, local association relations (eg, "find themselves-up", "find themselves-under"), become complementary relations for determining suitable relationship information for implementing the concept proposed here and propagating relations (eg between the near and far range of the scene in the digital image 400 . 500 . 600 ) used. Also other relations are conceivable.

The relations are each directly in the picture 204 . 400 . 500 . 600 moderated and / or refer to other suitable coordinate systems. Added to this are motion relations that describe relative changes to and / or between objects of a scene. Exemplary motion relations are a bearing change as opposed to a constant bearing, a change in size, a relative bearing change between objects (parallax), and the tracking of relation changes; For example, neighborhood relations may change and an object behavior derived therefrom. Examples include a departing or approaching vehicle ahead, vehicles that are being overhauled or overhauled, vehicles that are coming in and hiding other road users.

In addition to the bottom-up approach, the analysis concept can advantageously be supplemented by a top-down analysis. From the relationships and / or relations more abstract information regarding a scene can be derived. For example, in the digital image 204 . 400 . 500 . 600 objects close together often in the same context as other objects in the scene. Further, completeness of the interpretation of a scene can be achieved by finding and handling "holes" in the description and / or scene interpretation. In addition, a completion of a scene interpretation and reduction of uninterpreted areas of the scene is conceivable. According to one embodiment, similar movement patterns of objects - for example, vehicles in the crowd, pedestrians in group, etc. - can be recognized and taken into account. A behavior of objects can be detected, or object-connecting relations can be recognized and / or possible intentions of the objects can be determined from them. It is also possible to use a behavior pattern or intent recognition to improve the recognition range and / or availability of object information.

According to a further embodiment, the recognition of a scene structure is also conceivable. For example, findings about a main direction of a road and / or a road course and a proximity to the vanishing point are obtained. It identifies the (essential) constraints of the road surface that can be traveled and identifies laterally structuring elements such as crash barriers, guide posts, a tree avenue, etc.

The sequence of an exemplary scene interpretation can advantageously be structured by relations into two phases. In a first phase, a bottom-up analysis of a scene takes place, for example, data-driven from sensor information. In a second phase, a top-down analysis of the scene is carried out, driven by content and meaning from the perspective of the scene.

The concept of relative object description presented here as a novel video sensor interface for driver assistance systems is suitable for realizing high-performance automatic and / or (partially) automated driving functions. The scenarios that can be improved with the proposed methods therefore play an important role in the roadmap for automatic driving. Against the background of the existing in this context high availability, for example, the longitudinal and transverse guidance of vehicles use of the approach proposed here is advantageous.

If an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.

Claims (12)

Procedure ( 300 ) for interpreting a vehicle environment ( 106 ) of a vehicle ( 100 ), the process ( 300 ) comprises the following steps: providing ( 302 ) of an image ( 204 ; 400 ; 500 ; 600 ) of the vehicle environment ( 106 ); Form ( 304 ) a relation information ( 206 ) using a first position of a first object ( 115 ) in the image ( 204 ; 400 ; 500 ; 600 ) and a second position of a second object ( 116 ) in the image ( 204 ; 400 ; 500 ; 600 ), where the relationship information ( 206 ) a spatial relation between the first object ( 115 ) and the second object ( 116 ) in the vehicle environment ( 106 represents; and Use ( 306. ) of the relation information ( 206 ) to the vehicle environment ( 106 ) to interpret. Procedure ( 300 ) according to claim 1, wherein in the step of using ( 306. ) the relation information ( 206 ) for a lateral steering function and / or longitudinal regulation of the vehicle ( 100 ) is used. Procedure ( 300 ) according to one of the preceding claims, comprising a step of associating the first object ( 115 ) and / or the second object ( 116 ) to an object class, wherein in the step of using ( 306. ) the object class is used to determine the vehicle environment ( 106 ) to interpret. Procedure ( 300 ) according to one of the preceding claims, with a step of forming concealment information ( 212 ), which is a masking of the first object ( 115 ) by the second object ( 116 ) in the image ( 204 ; 400 ; 500 ; 600 ), wherein in the step of using ( 306. ) the concealment information ( 212 ) is used to control the vehicle environment ( 106 ) to interpret. Procedure ( 300 ) according to one of the preceding claims, in which in the step of forming ( 304 ) further a further information ( 208 ) using a further first position of a further first object in the image ( 204 ; 400 ; 500 ; 600 ) and a further second position of a further second object in the image ( 204 ; 400 ; 500 ; 600 ), the further relationship information ( 208 ) a spatial relation between the further first object and the further second object in the vehicle environment ( 106 ), and wherein in the step of using ( 306. ) further the further information ( 208 ) is used to control the vehicle environment ( 106 ) to interpret. Procedure ( 300 ) according to claim 5, wherein the step of using ( 306. ) a step of generating a relational value between the relation information ( 206 ) and the further relationship information ( 208 ) to the vehicle environment ( 106 ) using the relational value. Procedure ( 300 ) according to claim 6, wherein in the step of using ( 306. ) the relational value for a subsequent journey behind another vehicle ( 412 ) is used. Procedure ( 300 ) according to one of the preceding claims, comprising a step of determining a bearing ( 214 ) one of the objects ( 115 . 116 ) or another object in relation to the optical sensor ( 118 ), wherein in the step of using ( 306. ) the bearing ( 214 ) is used to control the vehicle environment ( 106 ) to interpret. Contraption ( 120 ), which is set up to take steps in the procedure ( 300 ) according to one of the preceding claims in corresponding units. Vehicle ( 100 ) with an optical sensor ( 118 ), a driver assistance system ( 114 ) and a device ( 120 ) according to claim 9, wherein the device ( 120 ) electrically conductive with the optical sensor ( 118 ) and / or the driver assistance system ( 114 ) connected is. Computer program adapted to perform the procedure ( 300 ) according to one of the preceding claims. A machine readable storage medium storing the computer program of claim 11.

Images